Anomaly Detection for High-Frequency Data

Question

Anomaly Detection for High-Frequency Data

Hans Meyer 46

Hi there,

Is my understanding correct, that univariate detection is not very suitable when trying to detect anomalies in high-frequency (millisecond-sampling frequency) due to the limited number of datapoints (8640)? Would the alternative just be to use another variable and try multivariate detection with more variables, or is there also a maximal number of datapoints for the inference API?

Thank you in advance!

Ramr-msft 14,596 Reputation points

2021-07-27T13:31:03.32+00:00

@Hans Meyer Thanks for the question. We have forwarded to the product team to check for limit on number of datapoints.

Please follow the document for metric advisor anomaly detection faq.
Ramr-msft 14,596 Reputation points

2021-07-27T13:31:06.427+00:00

@Hans Meyer Thanks, Multivariate detection API can not be applied to one variable. You have to prepare multiple signals to detect anomalies. For training, you could send at most 1 million points. For inference, each time you could inference 20,000 points.

Multivariate anomaly detection API overview:
This new feature has a different workflow compared with the existing univariate feature. There are two phases to obtain the detection results, the training phase, and the inference phase. In the training phase, you need to provide some historical data to let the model learn past patterns. Then in the inference phase, you can call the inference API to acquire detection results of multivariate time-series in a given range.

Learn more from https://aka.ms/AnomalyDetector and our technical blog Introducing Multivariate Anomaly Detection.
Hans Meyer 46 Reputation points

2021-07-29T06:27:06.917+00:00

This doesn't answer my question. I was simply wondering: is there is a limit in datapoints sent to the training/inference API for the multivariate anomaly detection? Also, can the multivariate detection API be used with only one variable (to avoid the datapoint limitation for univariate detection)?
Ramr-msft 14,596 Reputation points

2021-07-30T08:58:35.523+00:00

@Hans Meyer Thanks for the details. We are investigating on this will update on the same.

Ramr-msft 14,596 Reputation points

2021-07-27T13:31:03.32+00:00

@Hans Meyer Thanks for the question. We have forwarded to the product team to check for limit on number of datapoints.

Please follow the document for metric advisor anomaly detection faq.
Ramr-msft 14,596 Reputation points

2021-07-27T13:31:06.427+00:00

@Hans Meyer Thanks, Multivariate detection API can not be applied to one variable. You have to prepare multiple signals to detect anomalies. For training, you could send at most 1 million points. For inference, each time you could inference 20,000 points.

Multivariate anomaly detection API overview:
This new feature has a different workflow compared with the existing univariate feature. There are two phases to obtain the detection results, the training phase, and the inference phase. In the training phase, you need to provide some historical data to let the model learn past patterns. Then in the inference phase, you can call the inference API to acquire detection results of multivariate time-series in a given range.

Learn more from https://aka.ms/AnomalyDetector and our technical blog Introducing Multivariate Anomaly Detection.
Hans Meyer 46 Reputation points

2021-07-29T06:27:06.917+00:00

This doesn't answer my question. I was simply wondering: is there is a limit in datapoints sent to the training/inference API for the multivariate anomaly detection? Also, can the multivariate detection API be used with only one variable (to avoid the datapoint limitation for univariate detection)?
Ramr-msft 14,596 Reputation points

2021-07-30T08:58:35.523+00:00

@Hans Meyer Thanks for the details. We are investigating on this will update on the same.

Answer 1

@Hans Meyer Thanks, For uni-variate detection, can you try to use length within 8640 to see how the detection performs. Actually to address the length issue, you can try sample data or aggregate data to a higher frequency. If none of the above works, would like to understand how many data points need to used for one-time detection. Multivariate detection API can not be applied to one variable. You have to prepare multiple signals to detect anomalies. For training, you could send at most 1 million points. For inference, each time you could inference 20,000 points.

Anomaly Detection for High-Frequency Data

0 additional answers

Activity