Azure data factory and secure access

Question

What are the secure options available to prevent ADF and ADLS from cyber threats and other security vulnerabilities besides accessing the services through private link and express routes and using vnet and private endpoints? Please suggest a holistic strategy to secure ADLS, ADF, and other azure services. Thank you.

Answer 1

Hi azure_learner,

Thanks for reaching out to Microsoft Q&A.

Securing ADF and ADLS is crucial to protecting your data and ensuring compliance with security standards. In addition to using private links, express routes, VNets, and private endpoints, there are several other strategies and best practices to consider for a comprehensive security approach.

Here’s a holistic strategy to enhance the security of ADLS, ADF, and other Azure services:

IAM

• use azure ad for authentication and role-based access control (Rbac) to manage who has access to your resources.
• enable managed identities for adf to securely access other azure services without storing credentials in your code.

Data encryption

• Ensure that data in adls is encrypted at rest using azure storage service encryption (SSE) with microsoft managed keys or customer managed keys.
• Use https to encrypt data in transit between clients and azure services.

Network security

• Use service endpoints to secure your azure services to your virtual.
• Try applying NSG(network security groups) to filter network traffic to and from azure resources in an azure virtual network.

Advanced threat protection

• Enable azure defender for storage to detect and respond to potential threats.
• Use azure security center to get a unified view of security across your azure environment and to implement security recommendations.

Data governance and compliance

• Use azure policy to enforce organizational standards and to assess compliance at scale.
• Implement azure blueprints to define a repeatable set of azure resources that adhere to your organization’s standards, patterns, and requirements.

Monitoring and logging

• Use azure monitor to collect, analyze, and act on telemetry data from your azure resources.
• Implement azure log analytics to query and analyze logs from your azure resources.

Backup and disaster recovery

• Use azure backup to protect your data and applications from data loss.
• Implement azure site recovery to ensure business continuity by keeping your applications running during outages.

Compliance certificatiions

• Ensure that your adf and adls implementations comply with industry standards and certifications such as iso 27001, hipaa, and gdpr.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

Answer 2

Hello azure_learner,

Greetings! Welcome to Microsoft Q&A Platform.

Azure Data Factory with a Self-hosted Integration Runtime on a Windows virtual machine in Azure to connect to an on-premises data store via Azure ExpressRoute, Azure Data Factory does create encrypted database connections. According to the security considerations for data movement in Azure Data Factory, the service ensures that all data movement is secure. Specifically, Azure Data Factory, including the Azure Integration Runtime and Self-hosted Integration Runtime, does not store any temporary data, cache data, or logs except for linked service credentials for cloud data stores, which are encrypted using certificates

If the cloud data store supports HTTPS or TLS, all data transfers between data movement services in Data Factory and a cloud data store are via secure channel HTTPS or TLS.

Therefore, if your on-premises data store secures data movement using TLS or HTTPS, then encryption applies to both data in transit and credentials, ensuring the security of your data.

You can see the below Microsoft document.

https://learn.microsoft.com/en-us/azure/data-factory/data-movement-security-considerations

Adding additional information on securing ADLS storage account,

This article will help you on Data Factory supports service principal and MSI authentication for Data Lake Storage Gen2

You can associate a security principal with an access level for files and directories. These associations are captured in an access control list (ACL). Each file and directory in your storage account has an access control list. When a security principal attempts an operation on a file or directory, An ACL check determines whether that security principal (user, group, service principal, or managed identity) has the correct permission level to perform the operation.

For more information: Access control lists (ACLs) in Azure Data Lake Storage Gen2

Lear more on FAQ

Additional information: We can't use access control lists to provide a level of access that is lower than a level granted by an Azure RABC role assignment. For example, if you assign the Storage Blob Data Contributor role to a service principal, then you can't use access control lists to prevent that service principal from writing to a directory. So I suggest you remove the service principal Azure RABC role assignment when you use the ACL to control access.

The above-mentioned operations can be performed using Storage Explorer

Hope this helps! Kindly let us know if the above helps or you need further assistance on this issue.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.