DataDrift in AzureML

Amoozegar, Shahrzad 1

Hi, What is the underlying algorithm for azureml DataDriftDetector class? and what is the mathematical implication of the threshold in datadrift?

1 answer

Ramr-msft 17,736 Reputation points

2022-08-09T03:15:57.817+00:00

@Amoozegar, Shahrzad Thanks for the question. Concept Drift Detection Methods:
Approaches to Drift Detection:

we will pivot to focusing on detecting drift in Data – regardless of what it is, where it came from. Our prereq becomes usage of TimeSeriesDataset, which is a low bar for entry.
datadrift = DataDriftDetector.create(ws, baseline=tsd.timerange(start, end), target=tsd)
Here is the document for Detect data drift (preview) on datasets.
Please sign in to rate this answer.
Amoozegar, Shahrzad 1 Reputation point

2022-08-10T00:47:59.383+00:00

Hi @Ramr-msft ,

Thanks for the response. I understand that the drift detection works on timeseries data, still I need to know how to determine threshold for data drift, and that's why I need to understand what algorithm is used in here and what is the meaning of threshold.

Thanks,
Shahrzad Amoozegar

Ramr-msft 17,736 Reputation points

2022-09-02T03:11:48.14+00:00

@ AmoozegarShahrzad-3484 Thanks for the details. Internal details can't be shared publicly, The threshold is configured in the monitor settings.

Amoozegar, Shahrzad 1 Reputation point

2022-09-02T12:04:52.06+00:00

@Ramr-msft I am trying to set this threshold and dont know how to set it. In other words I dont know if 0.3 is noticeable drift that I should retrain the model or 0.2.

Ramr-msft 17,736 Reputation points

2022-09-06T00:31:55.53+00:00

@Amoozegar, Shahrzad Data drift is what would cause the model to degrade over time. So, a model monitoring solution would need to take that into account. Unfortunately, setting up the data drift modeling and alerting is cumbersome. It can be done with a combination of application insights, log analytics and event grid. Using these tools you can set a threshold whereby if the data drifts X% you will auto-trigger a retraining of the model.

https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/work-with-data/datadrift-tutorial/datadrift-tutorial.ipynb
https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-monitor-datasets?tabs=python
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

DataDrift in AzureML

1 answer

Your answer