ערוך

שתף באמצעות

Quickstart: Use the Multivariate Anomaly Detector client library

  • מאמר

Important

Starting on the 20th of September, 2023 you won’t be able to create new Anomaly Detector resources. The Anomaly Detector service is being retired on the 1st of October, 2026.

Get started with the Anomaly Detector multivariate client library for C#. Follow these steps to install the package and start using the algorithms provided by the service. The new multivariate anomaly detector APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for C# to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Library reference documentation | Library source code | Package (NuGet)

Prerequisites

  • Azure subscription - Create one for free
  • The current version of .NET Core
  • Once you have your Azure subscription, create an Anomaly Detector resource in the Azure portal to get your key and endpoint. Wait for it to deploy and select the Go to resource button.
    • You will need the key and endpoint from the resource you create to connect your application to the Anomaly Detector API. Paste your key and endpoint into the code below later in the quickstart. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Set up

Create a storage account

Multivariate Anomaly Detector requires your sample file to be stored in Azure Blob Storage.

  1. Create an Azure Storage account.
  2. Go to Access Control(IAM), and select ADD to Add role assignment.
  3. Search role of Storage Blob Data Reader, highlight this account type and then select Next.
  4. Select assign access to Managed identity, and Select Members, then choose the Anomaly Detector resource that you created earlier, then select Review + assign.

This configuration can sometimes be a little confusing, if you have trouble we recommend consulting our multivariate Jupyter Notebook sample, which walks through this process more in-depth.

Download sample data

This quickstart uses one file for sample data sample_data_5_3000.csv. This file can be downloaded from our GitHub sample data

You can also download the sample data by running:

curl "https://github.com/Azure-Samples/AnomalyDetector/blob/master/sampledata/multivariate/sample_data_5_3000.csv" --output sample_data_5_3000_.csv

Upload sample data to Storage Account

  1. Go to your Storage Account, select Containers and create a new container.
  2. Select Upload and upload sample_data_5_3000.csv
  3. Select the data that you uploaded and copy the Blob URL as you need to add it to the code sample in a few steps.

Retrieve key and endpoint

To successfully make a call against the Anomaly Detector service, you need the following values:

Variable name Value
ANOMALY_DETECTOR_ENDPOINT This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Example endpoint: https://YOUR_RESOURCE_NAME.cognitiveservices.azure.com/
ANOMALY_DETECTOR_API_KEY The API key value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Create environment variables

Create and assign persistent environment variables for your key and endpoint.

Important

If you use an API key, store it securely somewhere else, such as in Azure Key Vault. Don't include the API key directly in your code, and never post it publicly.

For more information about AI services security, see Authenticate requests to Azure AI services.

setx ANOMALY_DETECTOR_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE"

setx ANOMALY_DETECTOR_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"

Create a new .NET Core application

In a console window (such as cmd, PowerShell, or Bash), use the dotnet new command to create a new console app with the name anomaly-detector-quickstart-multivariate. This command creates a simple "Hello World" project with a single C# source file: Program.cs.

dotnet new console -n anomaly-detector-quickstart-multivariate

Change your directory to the newly created app folder. You can build the application with:

dotnet build

The build output should contain no warnings or errors.

...
Build succeeded.
 0 Warning(s)
 0 Error(s)
...

Install the client library

Within the application directory, install the Anomaly Detector client library for .NET with the following command:

dotnet add package Azure.AI.AnomalyDetector --prerelease

From the project directory, open the program.cs file and replace with the following code:

using Azure.AI.AnomalyDetector;
using Azure;
using static System.Environment;

internal class Program
{
    private static void Main(string[] args)
    {
        string endpoint = GetEnvironmentVariable("ANOMALY_DETECTOR_ENDPOINT"); 
        string apiKey = GetEnvironmentVariable("ANOMALY_DETECTOR_API_KEY");
        string datasource = "Path-to-sample-file-in-your-storage-account";  // example path:https://docstest001.blob.core.windows.net/test/sample_data_5_3000.csv
        Console.WriteLine(endpoint);
        var endpointUri = new Uri(endpoint);
        var credential = new AzureKeyCredential(apiKey);

        //create client
        AnomalyDetectorClient client = new AnomalyDetectorClient(endpointUri, credential);

        // train
        TimeSpan offset = new TimeSpan(0);
        DateTimeOffset start_time = new DateTimeOffset(2021, 1, 2, 0, 0, 0, offset);
        DateTimeOffset end_time = new DateTimeOffset(2021, 1, 2, 5, 0, 0, offset);
        string model_id = null;
        try
        {
            model_id = TrainModel(client, datasource, start_time, end_time);

            // detect
            end_time = new DateTimeOffset(2021, 1, 2, 1, 0, 0, offset);
            MultivariateDetectionResult result = BatchDetect(client, datasource, model_id, start_time, end_time);
            if (result != null)
            {
                Console.WriteLine(string.Format("Result ID: {0}", result.ResultId.ToString()));
                Console.WriteLine(string.Format("Result summary: {0}", result.Summary.ToString()));
                Console.WriteLine(string.Format("Result length: {0}", result.Results.Count));
                Console.WriteLine(string.Format("Anomalies found: {0}", result.Results.Where(r => r.Value.IsAnomaly).Count()));
            }

            // delete
            DeleteModel(client, model_id);
        }
        catch (Exception e)
        {
            string msg = string.Format("Multivariate error. {0}", e.Message);
            Console.WriteLine(msg);
            throw;
        }

        int GetModelNumber(AnomalyDetectorClient client)
        {
            int model_number = 0;
            foreach (var multivariateModel in client.GetMultivariateModels())
            {
                model_number++;
            }
            return model_number;
        }

        string TrainModel(AnomalyDetectorClient client, string datasource, DateTimeOffset start_time, DateTimeOffset end_time, int max_tryout = 500)
        {
            try
            {
                Console.WriteLine("Training new model...");

                Console.WriteLine(string.Format("{0} available models before training.", GetModelNumber(client)));

                ModelInfo request = new ModelInfo(datasource, start_time, end_time);
                request.SlidingWindow = 200;

                Console.WriteLine("Training new model...(it may take a few minutes)");
                AnomalyDetectionModel response = client.TrainMultivariateModel(request);
                string trained_model_id = response.ModelId;
                Console.WriteLine(string.Format("Training model id is {0}", trained_model_id));

                // Wait until the model is ready. It usually takes several minutes
                ModelStatus? model_status = null;
                int tryout_count = 1;
                response = client.GetMultivariateModel(trained_model_id);
                while (tryout_count < max_tryout & model_status != ModelStatus.Ready & model_status != ModelStatus.Failed)
                {
                    Thread.Sleep(1000);
                    response = client.GetMultivariateModel(trained_model_id);
                    model_status = response.ModelInfo.Status;
                    Console.WriteLine(string.Format("try {0}, model_id: {1}, status: {2}.", tryout_count, trained_model_id, model_status));
                    tryout_count += 1;
                };

                if (model_status == ModelStatus.Ready)
                {
                    Console.WriteLine("Creating model succeeds.");
                    Console.WriteLine(string.Format("{0} available models after training.", GetModelNumber(client)));
                    return trained_model_id;
                }

                if (model_status == ModelStatus.Failed)
                {
                    Console.WriteLine("Creating model failed.");
                    Console.WriteLine("Errors:");
                    try
                    {
                        Console.WriteLine(string.Format("Error code: {0}, Message: {1}", response.ModelInfo.Errors[0].Code.ToString(), response.ModelInfo.Errors[0].Message.ToString()));
                    }
                    catch (Exception e)
                    {
                        Console.WriteLine(string.Format("Get error message fail: {0}", e.Message));
                    }
                }
                return null;
            }
            catch (Exception e)
            {
                Console.WriteLine(string.Format("Train error. {0}", e.Message));
                throw;
            }
        }

        MultivariateDetectionResult BatchDetect(AnomalyDetectorClient client, string datasource, string model_id, DateTimeOffset start_time, DateTimeOffset end_time, int max_tryout = 500)
        {
            try
            {
                Console.WriteLine("Start batch detect...");
                MultivariateBatchDetectionOptions request = new MultivariateBatchDetectionOptions(datasource, 10, start_time, end_time);

                Console.WriteLine("Start batch detection, this might take a few minutes...");
                MultivariateDetectionResult response = client.DetectMultivariateBatchAnomaly(model_id, request);
                string result_id = response.ResultId;
                Console.WriteLine(string.Format("result id is: {0}", result_id));

                // get detection result
                MultivariateDetectionResult resultResponse = client.GetMultivariateBatchDetectionResult(result_id);
                MultivariateBatchDetectionStatus result_status = resultResponse.Summary.Status;
                int tryout_count = 0;
                while (tryout_count < max_tryout & result_status != MultivariateBatchDetectionStatus.Ready & result_status != MultivariateBatchDetectionStatus.Failed)
                {
                    Thread.Sleep(1000);
                    resultResponse = client.GetMultivariateBatchDetectionResult(result_id);
                    result_status = resultResponse.Summary.Status;
                    Console.WriteLine(string.Format("try: {0}, result id: {1} Detection status is {2}", tryout_count, result_id, result_status.ToString()));
                    Console.Out.Flush();
                }

                if (result_status == MultivariateBatchDetectionStatus.Failed)
                {
                    Console.WriteLine("Detection failed.");
                    Console.WriteLine("Errors:");
                    try
                    {
                        Console.WriteLine(string.Format("Error code: {}. Message: {}", resultResponse.Summary.Errors[0].Code.ToString(), resultResponse.Summary.Errors[0].Message.ToString()));
                    }
                    catch (Exception e)
                    {
                        Console.WriteLine(string.Format("Get error message fail: {0}", e.Message));
                    }
                    return null;
                }
                return resultResponse;
            }
            catch (Exception e)
            {
                Console.WriteLine(string.Format("Detection error. {0}", e.Message));
                throw;
            }
        }

        void DeleteModel(AnomalyDetectorClient client, string model_id)
        {
            client.DeleteMultivariateModel(model_id);
            int model_number = GetModelNumber(client);
            Console.WriteLine(string.Format("{0} available models after deletion.", model_number));
        }
 
    }
}

Run the application

Run the application with the dotnet run command from your application directory.

dotnet run

Clean up resources

If you want to clean up and remove an Azure AI services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with the resource group.

Next steps

Get started with the Anomaly Detector multivariate client library for JavaScript. Follow these steps to install the package and start using the algorithms provided by the service. The new multivariate anomaly detector APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for JavaScript to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much, and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Library reference documentation | Library source code | Package (npm) | Sample code

Prerequisites

  • Azure subscription - Create one for free
  • The current version of Node.js
  • Once you have your Azure subscription, create an Anomaly Detector resource in the Azure portal to get your key and endpoint. Wait for it to deploy and select the Go to resource button.
    • You'll need the key and endpoint from the resource you create to connect your application to the Anomaly Detector API. You'll paste your key and endpoint into the code below later in the quickstart. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Setting up

Create a new Node.js application

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it.

mkdir myapp && cd myapp

Run the npm init command to create a node application with a package.json file.

npm init

Create a file named index.js and import the following libraries: `

'use strict'

const fs = require('fs');
const parse = require("csv-parse/lib/sync");
const { AnomalyDetectorClient } = require('@azure/ai-anomaly-detector');
const { AzureKeyCredential } = require('@azure/core-auth');

Create variables your resource's Azure endpoint and key. Create another variable for the example data file.

Note

You will always have the option of using one of two keys. This is to allow secure key rotation. For the purposes of this quickstart use the first key.

const apiKey = "YOUR_API_KEY";
const endpoint = "YOUR_ENDPOINT";
const data_source = "YOUR_SAMPLE_ZIP_FILE_LOCATED_IN_AZURE_BLOB_STORAGE_WITH_SAS";

Important

Remember to remove the key from your code when you're done, and never post it publicly. For production, use a secure way of storing and accessing your credentials like Azure Key Vault. See the Azure AI services security article for more information.

To use the Anomaly Detector multivariate APIs, you need to first train your own models. Training data is a set of multiple time series that meet the following requirements:

Each time series should be a CSV file with two (and only two) columns, "timestamp" and "value" (all in lowercase) as the header row. The "timestamp" values should conform to ISO 8601; the "value" could be integers or decimals with any number of decimal places. For example:

timestamp value
2019-04-01T00:00:00Z 5
2019-04-01T00:01:00Z 3.6
2019-04-01T00:02:00Z 4
... ...

Each CSV file should be named after a different variable that will be used for model training. For example, "temperature.csv" and "humidity.csv". All the CSV files should be zipped into one zip file without any subfolders. The zip file can have whatever name you want. The zip file should be uploaded to Azure Blob storage. Once you generate the blob SAS (Shared access signatures) URL for the zip file, it can be used for training. Refer to this document for how to generate SAS URLs from Azure Blob Storage.

Install the client library

Install the ms-rest-azure and azure-ai-anomalydetector NPM packages. The csv-parse library is also used in this quickstart:

npm install @azure/ai-anomaly-detector csv-parse

Your app's package.json file is updated with the dependencies.

Code examples

These code snippets show you how to do the following with the Anomaly Detector client library for Node.js:

Authenticate the client

Instantiate a AnomalyDetectorClient object with your endpoint and credentials.

const client = new AnomalyDetectorClient(endpoint, new AzureKeyCredential(apiKey));

Train a model

Construct a model result

First we need to construct a model request. Make sure that start and end time align with your data source.

const Modelrequest = {
  source: data_source,
  startTime: new Date(2021,0,1,0,0,0),
  endTime: new Date(2021,0,2,12,0,0),
  slidingWindow:200
};

Train a new model

You pass your model request to the Anomaly Detector client trainMultivariateModel method.

console.log("Training a new model...")
const train_response = await client.trainMultivariateModel(Modelrequest)
const model_id = train_response.location?.split("/").pop() ?? ""
console.log("New model ID: " + model_id)

To check if training of your model is complete you can track the model's status:

let model_response = await client.getMultivariateModel(model_id);
let model_status = model_response.modelInfo.status;

while (model_status != 'READY' && model_status != 'FAILED'){
  await sleep(10000).then(() => {});
  model_response = await client.getMultivariateModel(model_id);
  model_status = model_response.modelInfo.status;
}

if (model_status == 'FAILED') {
  console.log("Training failed.\nErrors:");
  for (let error of model_response.modelInfo?.errors ?? []) {
    console.log("Error code: " + error.code + ". Message: " + error.message);
  }
}

console.log("TRAINING FINISHED.");

Detect anomalies

Use the detectAnomaly and getDectectionResult functions to determine if there are any anomalies within your datasource.

console.log("Start detecting...");
const detect_request = {
  source: data_source,
  startTime: new Date(2021,0,2,12,0,0),
  endTime: new Date(2021,0,3,0,0,0)
};
const result_header = await client.detectAnomaly(model_id, detect_request);
const result_id = result_header.location?.split("/").pop() ?? "";
let result = await client.getDetectionResult(result_id);
let result_status = result.summary.status;

while (result_status != 'READY' && result_status != 'FAILED'){
  await sleep(2000).then(() => {});
  result = await client.getDetectionResult(result_id);
  result_status = result.summary.status;
}

if (result_status == 'FAILED') {
  console.log("Detection failed.\nErrors:");
  for (let error of result.summary.errors ?? []) {
    console.log("Error code: " + error.code + ". Message: " + error.message)
  }
}
console.log("Result status: " + result_status);
console.log("Result Id: " + result.resultId);

Export model

Note

The export command is intended to be used to allow running Anomaly Detector multivariate models in a containerized environment. This is not currently not supported for multivariate, but support will be added in the future.

To export your trained model use the exportModel function.

const export_result = await client.exportModel(model_id)
const model_path = "model.zip"
const destination = fs.createWriteStream(model_path)
export_result.readableStreamBody?.pipe(destination)
console.log("New model has been exported to "+model_path+".")

Delete model

To delete an existing model that is available to the current resource use the deleteMultivariateModel function.

client.deleteMultivariateModel(model_id)
console.log("New model has been deleted.")

Run the application

Before running the application it can be helpful to check your code against the full sample code

Run the application with the node command on your quickstart file.

node index.js

Clean up resources

If you want to clean up and remove an Azure AI services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with the resource group.

Next steps

Library reference documentation |Library source code | Package (PyPi) |Find the sample code on GitHub

Get started with the Anomaly Detector multivariate client library for Python. Follow these steps to install the package, and start using the algorithms provided by the service. The new multivariate anomaly detector APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for Python to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much, and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Prerequisites

  • An Azure subscription - Create one for free
  • Python 3.x
  • Once you have your Azure subscription, create an Anomaly Detector resource in the Azure portal to get your key and endpoint. Wait for it to deploy and select the Go to resource button. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Set up

Install the client library. You can install the client library with:

pip install --upgrade azure.ai.anomalydetector

Create a storage account

Multivariate Anomaly Detector requires your sample file to be stored in Azure Blob Storage.

  1. Create an Azure Storage account.
  2. Go to Access Control(IAM), and select ADD to Add role assignment.
  3. Search role of Storage Blob Data Reader, highlight this account type and then select Next.
  4. Select assign access to Managed identity, and Select Members, then choose the Anomaly Detector resource that you created earlier, then select Review + assign.

This configuration can sometimes be a little confusing, if you have trouble we recommend consulting our multivariate Jupyter Notebook sample, which walks through this process more in-depth.

Download sample data

This quickstart uses one file for sample data sample_data_5_3000.csv. This file can be downloaded from our GitHub sample data

You can also download the sample data by running:

curl "https://github.com/Azure-Samples/AnomalyDetector/blob/master/sampledata/multivariate/sample_data_5_3000.csv" --output sample_data_5_3000_.csv

Upload sample data to Storage Account

  1. Go to your Storage Account, select Containers and create a new container.
  2. Select Upload and upload sample_data_5_3000.csv
  3. Select the data that you uploaded and copy the Blob URL as you need to add it to the code sample in a few steps.

Retrieve key and endpoint

To successfully make a call against the Anomaly Detector service, you need the following values:

Variable name Value
ANOMALY_DETECTOR_ENDPOINT This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Example endpoint: https://YOUR_RESOURCE_NAME.cognitiveservices.azure.com/
ANOMALY_DETECTOR_API_KEY The API key value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Create environment variables

Create and assign persistent environment variables for your key and endpoint.

Important

If you use an API key, store it securely somewhere else, such as in Azure Key Vault. Don't include the API key directly in your code, and never post it publicly.

For more information about AI services security, see Authenticate requests to Azure AI services.

setx ANOMALY_DETECTOR_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE"

setx ANOMALY_DETECTOR_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"

Create a new Python application

  1. Create a new Python file called sample_multivariate_detect.py. Then open it up in your preferred editor or IDE.

  2. Replace the contents of sample_multivariate_detect.py with the following code. You need to modify the paths for the variables blob_url.

import time
from datetime import datetime, timezone
from azure.ai.anomalydetector import AnomalyDetectorClient
from azure.core.credentials import AzureKeyCredential
from azure.ai.anomalydetector.models import *
import os

SUBSCRIPTION_KEY =  os.environ['ANOMALY_DETECTOR_API_KEY']
ANOMALY_DETECTOR_ENDPOINT = os.environ['ANOMALY_DETECTOR_ENDPOINT']

ad_client = AnomalyDetectorClient(ANOMALY_DETECTOR_ENDPOINT, AzureKeyCredential(SUBSCRIPTION_KEY))

time_format = "%Y-%m-%dT%H:%M:%SZ"
blob_url = "Path-to-sample-file-in-your-storage-account"  # example path: https://docstest001.blob.core.windows.net/test/sample_data_5_3000.csv

train_body = ModelInfo(
    data_source=blob_url,
    start_time=datetime.strptime("2021-01-02T00:00:00Z", time_format),
    end_time=datetime.strptime("2021-01-02T05:00:00Z", time_format),
    data_schema="OneTable",
    display_name="sample",
    sliding_window=200,
    align_policy=AlignPolicy(
        align_mode=AlignMode.OUTER,
        fill_n_a_method=FillNAMethod.LINEAR,
        padding_value=0,
    ),
)

batch_inference_body = MultivariateBatchDetectionOptions(
       data_source=blob_url,
       top_contributor_count=10,
       start_time=datetime.strptime("2021-01-02T00:00:00Z", time_format),
       end_time=datetime.strptime("2021-01-02T05:00:00Z", time_format),
   )


print("Training new model...(it may take a few minutes)")
model = ad_client.train_multivariate_model(train_body)
model_id = model.model_id
print("Training model id is {}".format(model_id))

## Wait until the model is ready. It usually takes several minutes
model_status = None
model = None

while model_status != ModelStatus.READY and model_status != ModelStatus.FAILED:
    model = ad_client.get_multivariate_model(model_id)
    print(model)
    model_status = model.model_info.status
    print("Model is {}".format(model_status))
    time.sleep(30)
if model_status == ModelStatus.READY:
    print("Done.\n--------------------")
    # Return the latest model id

# Detect anomaly in the same data source (but a different interval)
result = ad_client.detect_multivariate_batch_anomaly(model_id, batch_inference_body)
result_id = result.result_id

# Get results (may need a few seconds)
anomaly_results = ad_client.get_multivariate_batch_detection_result(result_id)
print("Get detection result...(it may take a few seconds)")

while anomaly_results.summary.status != MultivariateBatchDetectionStatus.READY and anomaly_results.summary.status != MultivariateBatchDetectionStatus.FAILED:
    anomaly_results = ad_client.get_multivariate_batch_detection_result(result_id)
    print("Detection is {}".format(anomaly_results.summary.status))
    time.sleep(5)
    
   
print("Result ID:\t", anomaly_results.result_id)
print("Result status:\t", anomaly_results.summary.status)
print("Result length:\t", len(anomaly_results.results))

# See detailed inference result
for r in anomaly_results.results:
    print(
        "timestamp: {}, is_anomaly: {:<5}, anomaly score: {:.4f}, severity: {:.4f}, contributor count: {:<4d}".format(
            r.timestamp,
            r.value.is_anomaly,
            r.value.score,
            r.value.severity,
            len(r.value.interpretation) if r.value.is_anomaly else 0,
        )
    )
    if r.value.interpretation:
        for contributor in r.value.interpretation:
            print(
                "\tcontributor variable: {:<10}, contributor score: {:.4f}".format(
                    contributor.variable, contributor.contribution_score
                )
            )

Run the application

Run the application with the python command on your quickstart file.

python sample_multivariate_detect.py

Output

10 available models before training.
Training new model...(it may take a few minutes)
Training model id is 3a695878-a88f-11ed-a16c-b290e72010e0
{'modelId': '3a695878-a88f-11ed-a16c-b290e72010e0', 'createdTime': '2023-02-09T15:34:23Z', 'lastUpdatedTime': '2023-02-09T15:34:23Z', 'modelInfo': {'dataSource': 'https://docstest001.blob.core.windows.net/test/sample_data_5_3000 (1).csv', 'dataSchema': 'OneTable', 'startTime': '2021-01-02T00:00:00Z', 'endTime': '2021-01-02T05:00:00Z', 'displayName': 'sample', 'slidingWindow': 200, 'alignPolicy': {'alignMode': 'Outer', 'fillNAMethod': 'Linear', 'paddingValue': 0.0}, 'status': 'CREATED', 'errors': [], 'diagnosticsInfo': {'modelState': {'epochIds': [], 'trainLosses': [], 'validationLosses': [], 'latenciesInSeconds': []}, 'variableStates': []}}}
Model is CREATED
{'modelId': '3a695878-a88f-11ed-a16c-b290e72010e0', 'createdTime': '2023-02-09T15:34:23Z', 'lastUpdatedTime': '2023-02-09T15:34:55Z', 'modelInfo': {'dataSource': 'https://docstest001.blob.core.windows.net/test/sample_data_5_3000 (1).csv', 'dataSchema': 'OneTable', 'startTime': '2021-01-02T00:00:00Z', 'endTime': '2021-01-02T05:00:00Z', 'displayName': 'sample', 'slidingWindow': 200, 'alignPolicy': {'alignMode': 'Outer', 'fillNAMethod': 'Linear', 'paddingValue': 0.0}, 'status': 'READY', 'errors': [], 'diagnosticsInfo': {'modelState': {'epochIds': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100], 'trainLosses': [1.0493712276220322, 0.5454281121492386, 0.42524269968271255, 0.38019897043704987, 0.3472398854792118, 0.34301353991031647, 0.3219067454338074, 0.3108387663960457, 0.30357857793569565, 0.29986055195331573], 'validationLosses': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'latenciesInSeconds': [0.3412797451019287, 0.25798678398132324, 0.2556419372558594, 0.3165152072906494, 0.2748451232910156, 0.26111531257629395, 0.2571413516998291, 0.257282018661499, 0.2549862861633301, 0.25806593894958496]}, 'variableStates': [{'variable': 'series_0', 'filledNARatio': 0.0, 'effectiveCount': 301, 'firstTimestamp': '2021-01-02T00:00:00Z', 'lastTimestamp': '2021-01-02T05:00:00Z'}, {'variable': 'series_1', 'filledNARatio': 0.0, 'effectiveCount': 301, 'firstTimestamp': '2021-01-02T00:00:00Z', 'lastTimestamp': '2021-01-02T05:00:00Z'}, {'variable': 'series_2', 'filledNARatio': 0.0, 'effectiveCount': 301, 'firstTimestamp': '2021-01-02T00:00:00Z', 'lastTimestamp': '2021-01-02T05:00:00Z'}, {'variable': 'series_3', 'filledNARatio': 0.0, 'effectiveCount': 301, 'firstTimestamp': '2021-01-02T00:00:00Z', 'lastTimestamp': '2021-01-02T05:00:00Z'}, {'variable': 'series_4', 'filledNARatio': 0.0, 'effectiveCount': 301, 'firstTimestamp': '2021-01-02T00:00:00Z', 'lastTimestamp': '2021-01-02T05:00:00Z'}]}}}
Model is READY
Done.
--------------------
10 available models after training.
Get detection result...(it may take a few seconds)
Detection is CREATED
Detection is READY
Result ID:	 70a6cdf8-a88f-11ed-a461-928899e62c38
Result status:	 READY
Result length:	 301
timestamp: 2021-01-02 00:00:00+00:00, is_anomaly: 0    , anomaly score: 0.1770, severity: 0.0000, contributor count: 0   
timestamp: 2021-01-02 00:01:00+00:00, is_anomaly: 0    , anomaly score: 0.3446, severity: 0.0000, contributor count: 0   
timestamp: 2021-01-02 00:02:00+00:00, is_anomaly: 0    , anomaly score: 0.2397, severity: 0.0000, contributor count: 0   
timestamp: 2021-01-02 00:03:00+00:00, is_anomaly: 0    , anomaly score: 0.1270, severity: 0.0000, contributor count: 0   
timestamp: 2021-01-02 00:04:00+00:00, is_anomaly: 0    , anomaly score: 0.3321, severity: 0.0000, contributor count: 0   
timestamp: 2021-01-02 00:05:00+00:00, is_anomaly: 0    , anomaly score: 0.4053, severity: 0.0000, contributor count: 0   
timestamp: 2021-01-02 00:06:00+00:00, is_anomaly: 0    , anomaly score: 0.4371, severity: 0.0000, contributor count: 0   
timestamp: 2021-01-02 00:07:00+00:00, is_anomaly: 1    , anomaly score: 0.6615, severity: 0.3850, contributor count: 5   
	contributor variable: series_3  , contributor score: 0.2939
	contributor variable: series_1  , contributor score: 0.2834
	contributor variable: series_4  , contributor score: 0.2329
	contributor variable: series_0  , contributor score: 0.1543
	contributor variable: series_2  , contributor score: 0.0354

The output results have been truncated for brevity.

Clean up resources

If you want to clean up and remove an Anomaly Detector resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it. You also may want to consider deleting the environment variables you created if you no longer intend to use them.

Get started with the Anomaly Detector multivariate client library for Java. Follow these steps to install the package start using the algorithms provided by the service. The new multivariate anomaly detector APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for Java to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much, and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Library reference documentation | Library source code | Package (Maven) | Sample code

Prerequisites

  • Azure subscription - Create one for free
  • The current version of the Java Development Kit(JDK)
  • The Gradle build tool, or another dependency manager.
  • Once you have your Azure subscription, create an Anomaly Detector resource in the Azure portal to get your key and endpoint. Wait for it to deploy and select the Go to resource button.
    • You'll need the key and endpoint from the resource you create to connect your application to the Anomaly Detector API. You'll paste your key and endpoint into the code below later in the quickstart. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Setting up

Create a new Gradle project

This quickstart uses the Gradle dependency manager. You can find more client library information on the Maven Central Repository.

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it.

mkdir myapp && cd myapp

Run the gradle init command from your working directory. This command creates essential build files for Gradle, including build.gradle.kts which is used at runtime to create and configure your application.

gradle init --type basic

When prompted to choose a DSL, select Kotlin.

Install the client library

Locate build.gradle.kts and open it with your preferred IDE or text editor. Then copy in this build configuration. Be sure to include the project dependencies.

dependencies {
    compile("com.azure:azure-ai-anomalydetector")
}

Create a Java file

Create a folder for your sample app. From your working directory, run the following command:

mkdir -p src/main/java

Navigate to the new folder and create a file called MetricsAdvisorQuickstarts.java. Open it in your preferred editor or IDE and add the following import statements:

package com.azure.ai.anomalydetector;

import com.azure.ai.anomalydetector.models.*;
import com.azure.core.credential.AzureKeyCredential;
import com.azure.core.http.*;
import com.azure.core.http.policy.*;
import com.azure.core.http.rest.PagedIterable;
import com.azure.core.http.rest.PagedResponse;
import com.azure.core.http.rest.Response;
import com.azure.core.http.rest.StreamResponse;
import com.azure.core.util.Context;
import reactor.core.publisher.Flux;

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.*;
import java.time.format.DateTimeFormatter;
import java.util.Iterator;
import java.util.List;
import java.util.UUID;
import java.util.stream.Collectors;

Create variables your resource's Azure endpoint and key. Create another variable for the example data file.

Note

You will always have the option of using one of two keys. This is to allow secure key rotation. For the purposes of this quickstart use the first key.

String key = "YOUR_API_KEY";
String endpoint = "YOUR_ENDPOINT";

Important

Remember to remove the key from your code when you're done, and never post it publicly. For production, use a secure way of storing and accessing your credentials like Azure Key Vault. See the Azure AI services security article for more information.

To use the Anomaly Detector multivariate APIs, you need to first train your own models. Training data is a set of multiple time series that meet the following requirements:

Each time series should be a CSV file with two (and only two) columns, "timestamp" and "value" (all in lowercase) as the header row. The "timestamp" values should conform to ISO 8601; the "value" could be integers or decimals with any number of decimal places. For example:

timestamp value
2019-04-01T00:00:00Z 5
2019-04-01T00:01:00Z 3.6
2019-04-01T00:02:00Z 4
... ...

Each CSV file should be named after a different variable that will be used for model training. For example, "temperature.csv" and "humidity.csv". All the CSV files should be zipped into one zip file without any subfolders. The zip file can have whatever name you want. The zip file should be uploaded to Azure Blob storage. Once you generate the blob SAS (Shared access signatures) URL for the zip file, it can be used for training. Refer to this document for how to generate SAS URLs from Azure Blob Storage.

Code examples

These code snippets show you how to do the following with the Anomaly Detector client library for Node.js:

Authenticate the client

Instantiate a anomalyDetectorClient object with your endpoint and credentials.

HttpHeaders headers = new HttpHeaders()
    .put("Accept", ContentType.APPLICATION_JSON);

HttpPipelinePolicy authPolicy = new AzureKeyCredentialPolicy("Ocp-Apim-Subscription-Key",
 new AzureKeyCredential(key));
AddHeadersPolicy addHeadersPolicy = new AddHeadersPolicy(headers);

HttpPipeline httpPipeline = new HttpPipelineBuilder().httpClient(HttpClient.createDefault())
    .policies(authPolicy, addHeadersPolicy).build();
// Instantiate a client that will be used to call the service.
HttpLogOptions httpLogOptions = new HttpLogOptions();
httpLogOptions.setLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS);

AnomalyDetectorClient anomalyDetectorClient = new AnomalyDetectorClientBuilder()
    .pipeline(httpPipeline)
    .endpoint(endpoint)
    .httpLogOptions(httpLogOptions)
    .buildClient();

Train a model

Construct a model result and train model

First we need to construct a model request. Make sure that start and end time align with your data source.

To use the Anomaly Detector multivariate APIs, we need to train our own model before using detection. Data used for training is a batch of time series, each time series should be in a CSV file with only two columns, "timestamp" and "value"(the column names should be exactly the same). Each CSV file should be named after each variable for the time series. All of the time series should be zipped into one zip file and be uploaded to Azure Blob storage, and there's no requirement for the zip file name. Alternatively, an extra meta.json file can be included in the zip file if you wish the name of the variable to be different from the .zip file name. Once we generate blob SAS (Shared access signatures) URL, we can use the url to the zip file for training.

Path path = Paths.get("test-data.csv");
List<String> requestData = Files.readAllLines(path);
List<TimeSeriesPoint> series = requestData.stream()
    .map(line -> line.trim())
    .filter(line -> line.length() > 0)
    .map(line -> line.split(",", 2))
    .filter(splits -> splits.length == 2)
    .map(splits -> {
        TimeSeriesPoint timeSeriesPoint = new TimeSeriesPoint();
        timeSeriesPoint.setTimestamp(OffsetDateTime.parse(splits[0]));
        timeSeriesPoint.setValue(Float.parseFloat(splits[1]));
        return timeSeriesPoint;
    })
    .collect(Collectors.toList());

Integer window = 28;
AlignMode alignMode = AlignMode.OUTER;
FillNAMethod fillNAMethod = FillNAMethod.LINEAR;
Integer paddingValue = 0;
AlignPolicy alignPolicy = new AlignPolicy()
                                .setAlignMode(alignMode)
                                .setFillNAMethod(fillNAMethod)
                                .setPaddingValue(paddingValue);
String source = "YOUR_SAMPLE_ZIP_FILE_LOCATED_IN_AZURE_BLOB_STORAGE_WITH_SAS";
OffsetDateTime startTime = OffsetDateTime.of(2021, 1, 2, 0, 0, 0, 0, ZoneOffset.UTC);
OffsetDateTime endTime = OffsetDateTime.of(2021, 1, 3, 0, 0, 0, 0, ZoneOffset.UTC);
String displayName = "Devops-MultiAD";

ModelInfo request = new ModelInfo()
                        .setSlidingWindow(window)
                        .setAlignPolicy(alignPolicy)
                        .setSource(source)
                        .setStartTime(startTime)
                        .setEndTime(endTime)
                        .setDisplayName(displayName);
TrainMultivariateModelResponse trainMultivariateModelResponse = anomalyDetectorClient.trainMultivariateModelWithResponse(request, Context.NONE);
String header = trainMultivariateModelResponse.getDeserializedHeaders().getLocation();
String[] substring = header.split("/");
UUID modelId = UUID.fromString(substring[substring.length - 1]);
System.out.println(modelId);

//Check model status until the model is ready
Response<Model> trainResponse;
while (true) {
    trainResponse = anomalyDetectorClient.getMultivariateModelWithResponse(modelId, Context.NONE);
    ModelStatus modelStatus = trainResponse.getValue().getModelInfo().getStatus();
    if (modelStatus == ModelStatus.READY || modelStatus == ModelStatus.FAILED) {
        break;
    }
    TimeUnit.SECONDS.sleep(10);
}

if (trainResponse.getValue().getModelInfo().getStatus() != ModelStatus.READY){
    System.out.println("Training failed.");
    List<ErrorResponse> errorMessages = trainResponse.getValue().getModelInfo().getErrors();
    for (ErrorResponse errorMessage : errorMessages) {
        System.out.println("Error code:  " + errorMessage.getCode());
        System.out.println("Error message:  " + errorMessage.getMessage());
    }
}

Detect anomalies

DetectionRequest detectionRequest = new DetectionRequest().setSource(source).setStartTime(startTime).setEndTime(endTime);
DetectAnomalyResponse detectAnomalyResponse = anomalyDetectorClient.detectAnomalyWithResponse(modelId, detectionRequest, Context.NONE);
String location = detectAnomalyResponse.getDeserializedHeaders().getLocation();
String[] substring = location.split("/");
UUID resultId = UUID.fromString(substring[substring.length - 1]);

DetectionResult detectionResult;
while (true) {
    detectionResult = anomalyDetectorClient.getDetectionResult(resultId);
    DetectionStatus detectionStatus = detectionResult.getSummary().getStatus();;
    if (detectionStatus == DetectionStatus.READY || detectionStatus == DetectionStatus.FAILED) {
        break;
    }
    TimeUnit.SECONDS.sleep(10);
}

if (detectionResult.getSummary().getStatus() != DetectionStatus.READY){
    System.out.println("Inference failed");
    List<ErrorResponse> detectErrorMessages = detectionResult.getSummary().getErrors();
    for (ErrorResponse errorMessage : detectErrorMessages) {
        System.out.println("Error code:  " + errorMessage.getCode());
        System.out.println("Error message:  " + errorMessage.getMessage());
    }
}

Export model

Note

The export command is intended to be used to allow running Anomaly Detector multivariate models in a containerized environment. This is not currently not supported for multivariate, but support will be added in the future.

To export your trained model use the exportModelWithResponse.

StreamResponse response_export = anomalyDetectorClient.exportModelWithResponse(model_id, Context.NONE);
Flux<ByteBuffer> value = response_export.getValue();
FileOutputStream bw = new FileOutputStream("result.zip");
value.subscribe(s -> write(bw, s), (e) -> close(bw), () -> close(bw));

Delete model

To delete an existing model that is available to the current resource use the deleteMultivariateModelWithResponse function.

Response<Void> deleteMultivariateModelWithResponse = anomalyDetectorClient.deleteMultivariateModelWithResponse(model_id, Context.NONE);

Run the application

You can build the app with:

gradle build

Run the application

Before running it can be helpful to check your code against the full sample code.

Run the application with the run goal:

gradle run

Clean up resources

If you want to clean up and remove an Azure AI services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with the resource group.

Next steps