Architecture recommendations - Hybrid Cloud interconnect - security - access ...

Grenot Pascal 20 Reputation points
2023-05-03T09:35:01.1133333+00:00

Hello there

We want to put in place Azure Databricks services for ML/IA - Model training and processing

User's image

The data will be pushed from another Cloud Provider (existing DataPlatform)

I am very interested in having support and recommendation from an Azure Architect and the Best practices to apply here (technical and economical)

  • Monitoring the dataset from Azure to train the model
  • Best access and security rules to put in place
  • Sizing of data flow

Thank you for all the welcome advices !

Kind regards

Pascal

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,017 questions
{count} votes

Accepted answer
  1. KranthiPakala-MSFT 46,432 Reputation points Microsoft Employee
    2023-05-05T02:39:03.97+00:00

    Hi @Grenot Pascal ,

    Welcome to Microsoft Q&A forum and thanks for reaching out here.

    This is a broader ask and I may not give a concrete answer as there will be several if and so situations, but below information is consolidated from my conversation with other internal experts.

    Monitoring datasets:    

    • Data Drift
    • Use DLT (built in data quality functionality, see if that is enough)
    • Build you custom data quality dashboard to monitor drift   
          

    Best access and security rules:

    • It depends on company policy.
    • Generally, Data Scientists are granted read-only access to production data. If Unity Catalog is used, this becomes easier.

     

    Sizing of data flow:   
    Did not follow entirely, maybe you can elaborate. But below suggestion is based on what I understood

    • Cheapest option is to see if you can access source data in place. 
    • If you plan on moving source data to Azure, then saving the data in a Lakehouse would be cheaper.

    Hope this helps.


    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful