Architecture recommendations - Hybrid Cloud interconnect - security - access ...

Question

Hello there

We want to put in place Azure Databricks services for ML/IA - Model training and processing

The data will be pushed from another Cloud Provider (existing DataPlatform)

I am very interested in having support and recommendation from an Azure Architect and the Best practices to apply here (technical and economical)

• Monitoring the dataset from Azure to train the model
• Best access and security rules to put in place
• Sizing of data flow

Thank you for all the welcome advices !

Kind regards

Pascal

Answer 1

Hi @Grenot Pascal ,

Welcome to Microsoft Q&A forum and thanks for reaching out here.

This is a broader ask and I may not give a concrete answer as there will be several if and so situations, but below information is consolidated from my conversation with other internal experts.

Monitoring datasets:    

• Data Drift
• Use DLT (built in data quality functionality, see if that is enough)
• Build you custom data quality dashboard to monitor drift   
      

Best access and security rules:

• It depends on company policy.
• Generally, Data Scientists are granted read-only access to production data. If Unity Catalog is used, this becomes easier.

 

Sizing of data flow:   
Did not follow entirely, maybe you can elaborate. But below suggestion is based on what I understood

• Cheapest option is to see if you can access source data in place. 
• If you plan on moving source data to Azure, then saving the data in a Lakehouse would be cheaper.

Hope this helps.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.