Quickstart: Use the Multivariate Anomaly Detector client library

Get started with the Anomaly Detector multivariate client library for C#. Follow these steps to install the package and start using the algorithms provided by the service. The new multivariate anomaly detection APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for C# to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Library reference documentation | Library source code | Package (NuGet)

Prerequisites

  • Azure subscription - Create one for free
  • The current version of .NET Core
  • Once you have your Azure subscription, create an Anomaly Detector resource in the Azure portal to get your key and endpoint. Wait for it to deploy and select the Go to resource button.
    • You will need the key and endpoint from the resource you create to connect your application to the Anomaly Detector API. Paste your key and endpoint into the code below later in the quickstart. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Setting up

Create a new .NET Core application

In a console window (such as cmd, PowerShell, or Bash), use the dotnet new command to create a new console app with the name anomaly-detector-quickstart-multivariate. This command creates a simple "Hello World" project with a single C# source file: Program.cs.

dotnet new console -n anomaly-detector-quickstart-multivariate

Change your directory to the newly created app folder. You can build the application with:

dotnet build

The build output should contain no warnings or errors.

...
Build succeeded.
 0 Warning(s)
 0 Error(s)
...

Install the client library

Within the application directory, install the Anomaly Detector client library for .NET with the following command:

dotnet add package Azure.AI.AnomalyDetector --version 3.0.0-preview.3

From the project directory, open the program.cs file and add the following using directives:

using System;
using System.Collections.Generic;
using System.Drawing.Text;
using System.IO;
using System.Linq;
using System.Linq.Expressions;
using System.Net.NetworkInformation;
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
using Azure.AI.AnomalyDetector.Models;
using Azure.Core.TestFramework;
using Microsoft.Identity.Client;
using NUnit.Framework;

In the application's main() method, create variables for your resource's Azure endpoint, your API key, and a custom datasource.

Note

You will always have the option of using one of two keys. This is to allow secure key rotation. For the purposes of this quickstart use the first key.

string endpoint = "YOUR_API_KEY";
string apiKey =  "YOUR_ENDPOINT";
string datasource = "YOUR_SAMPLE_ZIP_FILE_LOCATED_IN_AZURE_BLOB_STORAGE_WITH_SAS";

Important

Remember to remove the key from your code when you're done, and never post it publicly. For production, use a secure way of storing and accessing your credentials like Azure Key Vault. See the Cognitive Services security article for more information.

To use the Anomaly Detector multivariate APIs, you need to first train your own models. Training data is a set of multiple time series that meet the following requirements:

Each time series should be a CSV file with two (and only two) columns, "timestamp" and "value" (all in lowercase) as the header row. The "timestamp" values should conform to ISO 8601; the "value" could be integers or decimals with any number of decimal places. For example:

timestamp value
2019-04-01T00:00:00Z 5
2019-04-01T00:01:00Z 3.6
2019-04-01T00:02:00Z 4
... ...

Each CSV file should be named after a different variable that will be used for model training. For example, "temperature.csv" and "humidity.csv". All the CSV files should be zipped into one zip file without any subfolders. The zip file can have whatever name you want. The zip file should be uploaded to Azure Blob storage. Once you generate the blob SAS (Shared access signatures) URL for the zip file, it can be used for training. Refer to this document for how to generate SAS URLs from Azure Blob Storage.

Code examples

These code snippets show you how to do the following with the Anomaly Detector multivariate client library for .NET:

Authenticate the client

Instantiate an Anomaly Detector client with your endpoint and key.

var endpointUri = new Uri(endpoint);
var credential = new AzureKeyCredential(apiKey)

AnomalyDetectorClient client = new AnomalyDetectorClient(endpointUri, credential);

Train the model

Create a new private async task as below to handle training your model. You will use TrainMultivariateModel to train the model and GetMultivariateModelAysnc to check when training is complete.

private async Task<Guid?> trainAsync(AnomalyDetectorClient client, string datasource, DateTimeOffset start_time, DateTimeOffset end_time)
{
    try
    {
        Console.WriteLine("Training new model...");

        int model_number = await getModelNumberAsync(client, false).ConfigureAwait(false);
        Console.WriteLine(String.Format("{0} available models before training.", model_number));

        ModelInfo data_feed = new ModelInfo(datasource, start_time, end_time);
        Response response_header = client.TrainMultivariateModel(data_feed);
        response_header.Headers.TryGetValue("Location", out string trained_model_id_path);
        Guid trained_model_id = Guid.Parse(trained_model_id_path.Split('/').LastOrDefault());
        Console.WriteLine(trained_model_id);

        // Wait until the model is ready. It usually takes several minutes
        Response<Model> get_response = await client.GetMultivariateModelAsync(trained_model_id).ConfigureAwait(false);
        while (get_response.Value.ModelInfo.Status != ModelStatus.Ready & get_response.Value.ModelInfo.Status != ModelStatus.Failed)
        {
            System.Threading.Thread.Sleep(10000);
            get_response = await client.GetMultivariateModelAsync(trained_model_id).ConfigureAwait(false);
            Console.WriteLine(String.Format("model_id: {0}, createdTime: {1}, lastUpdateTime: {2}, status: {3}.", get_response.Value.ModelId, get_response.Value.CreatedTime, get_response.Value.LastUpdatedTime, get_response.Value.ModelInfo.Status));
        }

        if (get_response.Value.ModelInfo.Status != ModelStatus.Ready)
        {
            Console.WriteLine(String.Format("Trainig failed."));
            IReadOnlyList<ErrorResponse> errors = get_response.Value.ModelInfo.Errors;
            foreach (ErrorResponse error in errors)
            {
                Console.WriteLine(String.Format("Error code: {0}.", error.Code));
                Console.WriteLine(String.Format("Error message: {0}.", error.Message));
            }
            throw new Exception("Training failed.");
        }

        model_number = await getModelNumberAsync(client).ConfigureAwait(false);
        Console.WriteLine(String.Format("{0} available models after training.", model_number));
        return trained_model_id;
    }
    catch (Exception e)
    {
        Console.WriteLine(String.Format("Train error. {0}", e.Message));
        throw new Exception(e.Message);
    }
}

Detect anomalies

To detect anomalies using your newly trained model, create a private async Task named detectAsync. You will create a new DetectionRequest and pass that as a parameter to DetectAnomalyAsync.

private async Task<DetectionResult> detectAsync(AnomalyDetectorClient client, string datasource, Guid model_id,DateTimeOffset start_time, DateTimeOffset end_time)
{
    try
    {
        Console.WriteLine("Start detect...");
        Response<Model> get_response = await client.GetMultivariateModelAsync(model_id).ConfigureAwait(false);

        DetectionRequest detectionRequest = new DetectionRequest(datasource, start_time, end_time);
        Response result_response = await client.DetectAnomalyAsync(model_id, detectionRequest).ConfigureAwait(false);
        var ok = result_response.Headers.TryGetValue("Location", out string result_id_path);
        Guid result_id = Guid.Parse(result_id_path.Split('/').LastOrDefault());
        // get detection result
        Response<DetectionResult> result = await client.GetDetectionResultAsync(result_id).ConfigureAwait(false);
        while (result.Value.Summary.Status != DetectionStatus.Ready & result.Value.Summary.Status != DetectionStatus.Failed)
        {
            System.Threading.Thread.Sleep(2000);
            result = await client.GetDetectionResultAsync(result_id).ConfigureAwait(false);
        }

        if (result.Value.Summary.Status != DetectionStatus.Ready)
        {
            Console.WriteLine(String.Format("Inference failed."));
            IReadOnlyList<ErrorResponse> errors = result.Value.Summary.Errors;
            foreach (ErrorResponse error in errors)
            {
                Console.WriteLine(String.Format("Error code: {0}.", error.Code));
                Console.WriteLine(String.Format("Error message: {0}.", error.Message));
            }
            return null;
        }

        return result.Value;
    }
    catch (Exception e)
    {
        Console.WriteLine(String.Format("Detection error. {0}", e.Message));
        throw new Exception(e.Message);
    }
}

Export model

Note

The export command is intended to be used to allow running Anomaly Detector multivariate models in a containerized environment. This is not currently not supported for multivariate, but support will be added in the future.

To export the model you trained previously, create a private async Task named exportAysnc. You will use ExportModelAsync and pass the model ID of the model you wish to export.

private async Task exportAsync(AnomalyDetectorClient client, Guid model_id, string model_path = "model.zip")
{
    try
    {
        Stream model = await client.ExportModelAsync(model_id).ConfigureAwait(false);
        if (model != null)
        {
            var fileStream = File.Create(model_path);
            model.Seek(0, SeekOrigin.Begin);
            model.CopyTo(fileStream);
            fileStream.Close();
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(String.Format("Export error. {0}", e.Message));
        throw new Exception(e.Message);
    }
}

Delete model

To delete a model that you have created previously use DeleteMultivariateModelAsync and pass the model ID of the model you wish to delete. To retrieve a model ID you can us getModelNumberAsync:

private async Task deleteAsync(AnomalyDetectorClient client, Guid model_id)
{
    await client.DeleteMultivariateModelAsync(model_id).ConfigureAwait(false);
    int model_number = await getModelNumberAsync(client).ConfigureAwait(false);
    Console.WriteLine(String.Format("{0} available models after deletion.", model_number));
}
private async Task<int> getModelNumberAsync(AnomalyDetectorClient client, bool delete = false)
{
    int count = 0;
    AsyncPageable<ModelSnapshot> model_list = client.ListMultivariateModelAsync(0, 10000);
    await foreach (ModelSnapshot x in model_list)
    {
        count += 1;
        Console.WriteLine(String.Format("model_id: {0}, createdTime: {1}, lastUpdateTime: {2}.", x.ModelId, x.CreatedTime, x.LastUpdatedTime));
        if (delete & count < 4)
        {
            await client.DeleteMultivariateModelAsync(x.ModelId).ConfigureAwait(false);
        }
    }
    return count;
}

Main method

Now that you have all the component parts, you need to add additional code to your main method to call your newly created tasks.


{
    //read endpoint and apiKey
     string endpoint = "YOUR_API_KEY";
    string apiKey =  "YOUR_ENDPOINT";
    string datasource = "YOUR_SAMPLE_ZIP_FILE_LOCATED_IN_AZURE_BLOB_STORAGE_WITH_SAS";
    Console.WriteLine(endpoint);
    var endpointUri = new Uri(endpoint);
    var credential = new AzureKeyCredential(apiKey);

    //create client
    AnomalyDetectorClient client = new AnomalyDetectorClient(endpointUri, credential);

    // train
    TimeSpan offset = new TimeSpan(0);
    DateTimeOffset start_time = new DateTimeOffset(2021, 1, 1, 0, 0, 0, offset);
    DateTimeOffset end_time = new DateTimeOffset(2021, 1, 2, 12, 0, 0, offset);
    Guid? model_id_raw = null;
    try
    {
        model_id_raw = await trainAsync(client, datasource, start_time, end_time).ConfigureAwait(false);
        Console.WriteLine(model_id_raw);
        Guid model_id = model_id_raw.GetValueOrDefault();

        // detect
        start_time = end_time;
        end_time = new DateTimeOffset(2021, 1, 3, 0, 0, 0, offset);
        DetectionResult result = await detectAsync(client, datasource, model_id, start_time, end_time).ConfigureAwait(false);
        if (result != null)
        {
            Console.WriteLine(String.Format("Result ID: {0}", result.ResultId));
            Console.WriteLine(String.Format("Result summary: {0}", result.Summary));
            Console.WriteLine(String.Format("Result length: {0}", result.Results.Count));
        }

        // export model
        await exportAsync(client, model_id).ConfigureAwait(false);

        // delete
        await deleteAsync(client, model_id).ConfigureAwait(false);
    }
    catch (Exception e)
    {
        String msg = String.Format("Multivariate error. {0}", e.Message);
        if (model_id_raw != null)
        {
            await deleteAsync(client, model_id_raw.GetValueOrDefault()).ConfigureAwait(false);
        }
        Console.WriteLine(msg);
        throw new Exception(msg);
    }
}

Run the application

Run the application with the dotnet run command from your application directory.

dotnet run

Clean up resources

If you want to clean up and remove a Cognitive Services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with the resource group.

Next steps

Get started with the Anomaly Detector multivariate client library for JavaScript. Follow these steps to install the package and start using the algorithms provided by the service. The new multivariate anomaly detection APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for JavaScript to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Library reference documentation | Library source code | Package (npm) | Sample code

Prerequisites

  • Azure subscription - Create one for free
  • The current version of Node.js
  • Once you have your Azure subscription, create an Anomaly Detector resource in the Azure portal to get your key and endpoint. Wait for it to deploy and click the Go to resource button.
    • You will need the key and endpoint from the resource you create to connect your application to the Anomaly Detector API. You'll paste your key and endpoint into the code below later in the quickstart. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Setting up

Create a new Node.js application

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it.

mkdir myapp && cd myapp

Run the npm init command to create a node application with a package.json file.

npm init

Create a file named index.js and import the following libraries: `

'use strict'

const fs = require('fs');
const parse = require("csv-parse/lib/sync");
const { AnomalyDetectorClient } = require('@azure/ai-anomaly-detector');
const { AzureKeyCredential } = require('@azure/core-auth');

Create variables your resource's Azure endpoint and key. Create another variable for the example data file.

Note

You will always have the option of using one of two keys. This is to allow secure key rotation. For the purposes of this quickstart use the first key.

const apiKey = "YOUR_API_KEY";
const endpoint = "YOUR_ENDPOINT";
const data_source = "YOUR_SAMPLE_ZIP_FILE_LOCATED_IN_AZURE_BLOB_STORAGE_WITH_SAS";

Important

Remember to remove the key from your code when you're done, and never post it publicly. For production, use a secure way of storing and accessing your credentials like Azure Key Vault. See the Cognitive Services security article for more information.

To use the Anomaly Detector multivariate APIs, you need to first train your own models. Training data is a set of multiple time series that meet the following requirements:

Each time series should be a CSV file with two (and only two) columns, "timestamp" and "value" (all in lowercase) as the header row. The "timestamp" values should conform to ISO 8601; the "value" could be integers or decimals with any number of decimal places. For example:

timestamp value
2019-04-01T00:00:00Z 5
2019-04-01T00:01:00Z 3.6
2019-04-01T00:02:00Z 4
... ...

Each CSV file should be named after a different variable that will be used for model training. For example, "temperature.csv" and "humidity.csv". All the CSV files should be zipped into one zip file without any subfolders. The zip file can have whatever name you want. The zip file should be uploaded to Azure Blob storage. Once you generate the blob SAS (Shared access signatures) URL for the zip file, it can be used for training. Refer to this document for how to generate SAS URLs from Azure Blob Storage.

Install the client library

Install the ms-rest-azure and azure-ai-anomalydetector NPM packages. The csv-parse library is also used in this quickstart:

npm install @azure/ai-anomaly-detector csv-parse

Your app's package.json file will be updated with the dependencies.

Code examples

These code snippets show you how to do the following with the Anomaly Detector client library for Node.js:

Authenticate the client

Instantiate a AnomalyDetectorClient object with your endpoint and credentials.

const client = new AnomalyDetectorClient(endpoint, new AzureKeyCredential(apiKey));

Train a model

Construct a model result

First we need to construct a model request. Make sure that start and end time align with your data source.

const Modelrequest = {
  source: data_source,
  startTime: new Date(2021,0,1,0,0,0),
  endTime: new Date(2021,0,2,12,0,0),
  slidingWindow:200
};

Train a new model

You will need to pass your model request to the Anomaly Detector client trainMultivariateModel method.

console.log("Training a new model...")
const train_response = await client.trainMultivariateModel(Modelrequest)
const model_id = train_response.location?.split("/").pop() ?? ""
console.log("New model ID: " + model_id)

To check if training of your model is complete you can track the model's status:

let model_response = await client.getMultivariateModel(model_id);
let model_status = model_response.modelInfo.status;

while (model_status != 'READY' && model_status != 'FAILED'){
  await sleep(10000).then(() => {});
  model_response = await client.getMultivariateModel(model_id);
  model_status = model_response.modelInfo.status;
}

if (model_status == 'FAILED') {
  console.log("Training failed.\nErrors:");
  for (let error of model_response.modelInfo?.errors ?? []) {
    console.log("Error code: " + error.code + ". Message: " + error.message);
  }
}

console.log("TRAINING FINISHED.");

Detect anomalies

Use the detectAnomaly and getDectectionResult functions to determine if there are any anomalies within your datasource.

console.log("Start detecting...");
const detect_request = {
  source: data_source,
  startTime: new Date(2021,0,2,12,0,0),
  endTime: new Date(2021,0,3,0,0,0)
};
const result_header = await client.detectAnomaly(model_id, detect_request);
const result_id = result_header.location?.split("/").pop() ?? "";
let result = await client.getDetectionResult(result_id);
let result_status = result.summary.status;

while (result_status != 'READY' && result_status != 'FAILED'){
  await sleep(2000).then(() => {});
  result = await client.getDetectionResult(result_id);
  result_status = result.summary.status;
}

if (result_status == 'FAILED') {
  console.log("Detection failed.\nErrors:");
  for (let error of result.summary.errors ?? []) {
    console.log("Error code: " + error.code + ". Message: " + error.message)
  }
}
console.log("Result status: " + result_status);
console.log("Result Id: " + result.resultId);

Export model

Note

The export command is intended to be used to allow running Anomaly Detector multivariate models in a containerized environment. This is not currently not supported for multivariate, but support will be added in the future.

To export your trained model use the exportModel function.

const export_result = await client.exportModel(model_id)
const model_path = "model.zip"
const destination = fs.createWriteStream(model_path)
export_result.readableStreamBody?.pipe(destination)
console.log("New model has been exported to "+model_path+".")

Delete model

To delete an existing model that is available to the current resource use the deleteMultivariateModel function.

client.deleteMultivariateModel(model_id)
console.log("New model has been deleted.")

Run the application

Before running the application it can be helpful to check your code against the full sample code

Run the application with the node command on your quickstart file.

node index.js

Clean up resources

If you want to clean up and remove a Cognitive Services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with the resource group.

Next steps

Library reference documentation |Library source code | Package (PyPi) |Find the sample code on GitHub

Get started with the Anomaly Detector multivariate client library for Python. Follow these steps to install the package, and start using the algorithms provided by the service. The new multivariate anomaly detection APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for Python to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much, and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Prerequisites

Set up

Install the client library. You can install the client library with:

pip install --upgrade azure.ai.anomalydetector

If you don't already have it installed, you will also need to install the pandas library:

pip install pandas

Download sample data

This quickstart uses the sample_data_5_3000.zip file that can be downloaded from our GitHub sample data

You can also download the sample data by running:

curl "https://github.com/Azure-Samples/AnomalyDetector/blob/master/sampledata/multivariate/multivariate_sample_data_5_3000.zip" --output sample_data_5_3000_.zip

Generate SAS URL

Multivariate Anomaly Detector requires your sample file to be stored as a .zip file in Azure Blob Storage.

  1. Create an Azure Storage account.

  2. From within your storage account, create a new storage container with the Public access level set to private.

  3. Open your container and select upload. Upload the sample_data_5_3000.zip file from the previous step. Screenshot of the storage upload user experience.

  4. Select the ... to open the context menu next to your newly uploaded zip file and select Generate SAS. Screenshot of the Blob storage context menu with Generate SAS highlighted.

  5. Select Generate SAS Token and URL.

  6. You will need to copy the SAS URL into the ANOMALY_DETECTOR_DATA_SOURCE environment variable in the next section.

    Note

    The steps above are the bare minimum to generate a SAS URL. For a more in-depth article on the process, we recommend consulting this Form Recognizer article.

Retrieve key and endpoint

To successfully make a call against the Anomaly Detector service, you'll need the following values:

Variable name Value
ANOMALY_DETECTOR_ENDPOINT This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Example endpoint: https://YOUR_RESOURCE_NAME.cognitiveservices.azure.com/
ANOMALY_DETECTOR_API_KEY The API key value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.
ANOMALY_DETECTOR_DATA_SOURCE This quickstart uses the sample_data_5-3000.zip file that can be downloaded from our GitHub sample data. This file will then need to be added to Azure Blob Storage and made accessible via a SAS URL.

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Create environment variables

Create and assign persistent environment variables for your key and endpoint.

setx ANOMALY_DETECTOR_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE"
setx ANOMALY_DETECTOR_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"
setx ANOMALY_DETECTOR_DATA_SOURCE "REPLACE_WITH_YOUR_SAS_URL_TO_THE_SAMPLE_ZIP_FILE"

Create a new Python application

  1. Create a new Python file called quickstart.py. Then open it up in your preferred editor or IDE.

  2. Replace the contents of quickstart.py with the following code. If you're using the environment variables from the earlier steps in the quickstart no changes to the code will be needed:

import os
import time
from datetime import datetime, timezone

from azure.ai.anomalydetector import AnomalyDetectorClient
from azure.ai.anomalydetector.models import DetectionRequest, ModelInfo, LastDetectionRequest
from azure.ai.anomalydetector.models import ModelStatus, DetectionStatus
from azure.core.credentials import AzureKeyCredential
from azure.core.exceptions import HttpResponseError

SUBSCRIPTION_KEY = os.environ["ANOMALY_DETECTOR_API_KEY"]
ANOMALY_DETECTOR_ENDPOINT = os.environ["ANOMALY_DETECTOR_ENDPOINT"]
DATA_SOURCE = os.environ["ANOMALY_DETECTOR_DATA_SOURCE"]

ad_client = AnomalyDetectorClient(AzureKeyCredential(SUBSCRIPTION_KEY), ANOMALY_DETECTOR_ENDPOINT)
model_list = list(ad_client.list_multivariate_model(skip=0, top=10000))
print("{:d} available models before training.".format(len(model_list)))

print("Training new model...(it may take a few minutes)")
data_feed = ModelInfo(start_time=datetime(2021, 1, 1, 0, 0, 0, tzinfo=timezone.utc), end_time=datetime(2021, 1, 2, 12, 0, 0, tzinfo=timezone.utc), source=DATA_SOURCE)
response_header = \
        ad_client.train_multivariate_model(data_feed, cls=lambda *args: [args[i] for i in range(len(args))])[-1]
trained_model_id = response_header['Location'].split("/")[-1]

model_status = None

while model_status != ModelStatus.READY and model_status != ModelStatus.FAILED:
    model_info = ad_client.get_multivariate_model(trained_model_id).model_info
    model_status = model_info.status
    time.sleep(30)
    print ("MODEL STATUS: " + model_status)

if model_status == ModelStatus.READY:
            new_model_list = list(ad_client.list_multivariate_model(skip=0, top=10000))
            print("Model training complete.\n--------------------")
            print("{:d} available models after training.".format(len(new_model_list)))
            print("New Model ID " + trained_model_id)

detection_req = DetectionRequest(source=DATA_SOURCE, start_time=datetime(2021, 1, 2, 12, 0, 0, tzinfo=timezone.utc), end_time=datetime(2021, 1, 3, 0, 0, 0, tzinfo=timezone.utc))
response_header = ad_client.detect_anomaly(trained_model_id, detection_req, cls=lambda *args: [args[i] for i in range(len(args))])[-1]
result_id = response_header['Location'].split("/")[-1]

# Get results (may need a few seconds)
r = ad_client.get_detection_result(result_id)
print("Get detection result...(it may take a few seconds)")

while r.summary.status != DetectionStatus.READY and r.summary.status != DetectionStatus.FAILED:
    r = ad_client.get_detection_result(result_id)
    time.sleep(1)

print("Result ID:\t", r.result_id)
print("Result status:\t", r.summary.status)
print("Result length:\t", len(r.results))
print("\nAnomaly details:")
for i in r.results:
        if i.value.is_anomaly:
            print("timestamp: {}, is_anomaly: {:<5}, anomaly score: {:.4f}, severity: {:.4f}, contributor count: {:<4d}".format(i.timestamp, str(i.value.is_anomaly), i.value.score, i.value.severity, len(i.value.interpretation) if i.value.is_anomaly else 0))
            if i.value.interpretation is not None:
                for interp in i.value.interpretation:
                    print("\tcorrelation changes: {:<10}, contribution score: {:.4f}".format(interp.variable, interp.contribution_score))

Run the application

Run the application with the python command on your quickstart file.

python quickstart.py

Output

0 available models before training.
Training new model...(it may take a few minutes)
MODEL STATUS: CREATED
MODEL STATUS: RUNNING
MODEL STATUS: RUNNING
MODEL STATUS: RUNNING
MODEL STATUS: RUNNING
MODEL STATUS: READY
Model training complete.
--------------------
1 available models after training.
New Model ID GUID
Get detection result...(it may take a few seconds)
Result ID: GUID
Result status:     READY
Result length:     721
timestamp: 2021-01-02 12:06:00+00:00, is_anomaly: True , anomaly score: 0.5633, severity: 0.3278, contributor count: 5
    correlation changes: series_2  , contribution score: 0.2950
    correlation changes: series_3  , contribution score: 0.2281
    correlation changes: series_1  , contribution score: 0.2148
    correlation changes: series_4  , contribution score: 0.1927
    correlation changes: series_0  , contribution score: 0.0694
timestamp: 2021-01-02 12:27:00+00:00, is_anomaly: True , anomaly score: 0.4873, severity: 0.2836, contributor count: 5
    correlation changes: series_2  , contribution score: 0.4787
    correlation changes: series_4  , contribution score: 0.2131
    correlation changes: series_1  , contribution score: 0.1528
    correlation changes: series_3  , contribution score: 0.1338
    correlation changes: series_0  , contribution score: 0.0215
timestamp: 2021-01-02 13:08:00+00:00, is_anomaly: True , anomaly score: 0.5176, severity: 0.3012, contributor count: 5
    correlation changes: series_1  , contribution score: 0.4417
    correlation changes: series_4  , contribution score: 0.1921
    correlation changes: series_3  , contribution score: 0.1730
    correlation changes: series_0  , contribution score: 0.1591
    correlation changes: series_2  , contribution score: 0.0341
timestamp: 2021-01-02 13:19:00+00:00, is_anomaly: True , anomaly score: 0.6038, severity: 0.3514, contributor count: 5
    correlation changes: series_0  , contribution score: 0.3545
    correlation changes: series_3  , contribution score: 0.3002
    correlation changes: series_2  , contribution score: 0.2700
    correlation changes: series_4  , contribution score: 0.0608
    correlation changes: series_1  , contribution score: 0.0144
timestamp: 2021-01-02 13:22:00+00:00, is_anomaly: True , anomaly score: 0.5010, severity: 0.2915, contributor count: 5

We also have an in-depth Jupyter Notebook to help you get started.

Clean up resources

If you want to clean up and remove an Anomaly Detector resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it. You also may want to consider deleting the environment variables you created if you no longer intend to use them.

Get started with the Anomaly Detector multivariate client library for Java. Follow these steps to install the package start using the algorithms provided by the service. The new multivariate anomaly detection APIs enable developers by easily integrating advanced AI for detecting anomalies from groups of metrics, without the need for machine learning knowledge or labeled data. Dependencies and inter-correlations between different signals are automatically counted as key factors. This helps you to proactively protect your complex systems from failures.

Use the Anomaly Detector multivariate client library for Java to:

  • Detect system level anomalies from a group of time series.
  • When any individual time series won't tell you much and you have to look at all signals to detect a problem.
  • Predicative maintenance of expensive physical assets with tens to hundreds of different types of sensors measuring various aspects of system health.

Library reference documentation | Library source code | Package (Maven) | Sample code

Prerequisites

  • Azure subscription - Create one for free
  • The current version of the Java Development Kit(JDK)
  • The Gradle build tool, or another dependency manager.
  • Once you have your Azure subscription, create an Anomaly Detector resource in the Azure portal to get your key and endpoint. Wait for it to deploy and click the Go to resource button.
    • You will need the key and endpoint from the resource you create to connect your application to the Anomaly Detector API. You'll paste your key and endpoint into the code below later in the quickstart. You can use the free pricing tier (F0) to try the service, and upgrade later to a paid tier for production.

Setting up

Create a new Gradle project

This quickstart uses the Gradle dependency manager. You can find more client library information on the Maven Central Repository.

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it.

mkdir myapp && cd myapp

Run the gradle init command from your working directory. This command will create essential build files for Gradle, including build.gradle.kts which is used at runtime to create and configure your application.

gradle init --type basic

When prompted to choose a DSL, select Kotlin.

Install the client library

Locate build.gradle.kts and open it with your preferred IDE or text editor. Then copy in this build configuration. Be sure to include the project dependencies.

dependencies {
    compile("com.azure:azure-ai-anomalydetector")
}

Create a Java file

Create a folder for your sample app. From your working directory, run the following command:

mkdir -p src/main/java

Navigate to the new folder and create a file called MetricsAdvisorQuickstarts.java. Open it in your preferred editor or IDE and add the following import statements:

package com.azure.ai.anomalydetector;

import com.azure.ai.anomalydetector.models.*;
import com.azure.core.credential.AzureKeyCredential;
import com.azure.core.http.*;
import com.azure.core.http.policy.*;
import com.azure.core.http.rest.PagedIterable;
import com.azure.core.http.rest.PagedResponse;
import com.azure.core.http.rest.Response;
import com.azure.core.http.rest.StreamResponse;
import com.azure.core.util.Context;
import reactor.core.publisher.Flux;

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.time.*;
import java.time.format.DateTimeFormatter;
import java.util.Iterator;
import java.util.List;
import java.util.UUID;
import java.util.stream.Collectors;

Create variables your resource's Azure endpoint and key. Create another variable for the example data file.

Note

You will always have the option of using one of two keys. This is to allow secure key rotation. For the purposes of this quickstart use the first key.

String key = "YOUR_API_KEY";
String endpoint = "YOUR_ENDPOINT";

Important

Remember to remove the key from your code when you're done, and never post it publicly. For production, use a secure way of storing and accessing your credentials like Azure Key Vault. See the Cognitive Services security article for more information.

To use the Anomaly Detector multivariate APIs, you need to first train your own models. Training data is a set of multiple time series that meet the following requirements:

Each time series should be a CSV file with two (and only two) columns, "timestamp" and "value" (all in lowercase) as the header row. The "timestamp" values should conform to ISO 8601; the "value" could be integers or decimals with any number of decimal places. For example:

timestamp value
2019-04-01T00:00:00Z 5
2019-04-01T00:01:00Z 3.6
2019-04-01T00:02:00Z 4
... ...

Each CSV file should be named after a different variable that will be used for model training. For example, "temperature.csv" and "humidity.csv". All the CSV files should be zipped into one zip file without any subfolders. The zip file can have whatever name you want. The zip file should be uploaded to Azure Blob storage. Once you generate the blob SAS (Shared access signatures) URL for the zip file, it can be used for training. Refer to this document for how to generate SAS URLs from Azure Blob Storage.

Code examples

These code snippets show you how to do the following with the Anomaly Detector client library for Node.js:

Authenticate the client

Instantiate a anomalyDetectorClient object with your endpoint and credentials.

HttpHeaders headers = new HttpHeaders()
    .put("Accept", ContentType.APPLICATION_JSON);

HttpPipelinePolicy authPolicy = new AzureKeyCredentialPolicy(key,
    new AzureKeyCredential(key));
AddHeadersPolicy addHeadersPolicy = new AddHeadersPolicy(headers);

HttpPipeline httpPipeline = new HttpPipelineBuilder().httpClient(HttpClient.createDefault())
    .policies(authPolicy, addHeadersPolicy).build();
// Instantiate a client that will be used to call the service.
HttpLogOptions httpLogOptions = new HttpLogOptions();
httpLogOptions.setLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS);

AnomalyDetectorClient anomalyDetectorClient = new AnomalyDetectorClientBuilder()
    .pipeline(httpPipeline)
    .endpoint(endpoint)
    .httpLogOptions(httpLogOptions)
    .buildClient();

Train a model

Construct a model result and train model

First we need to construct a model request. Make sure that start and end time align with your data source.

To use the Anomaly Detector multivariate APIs, we need to train our own model before using detection. Data used for training is a batch of time series, each time series should be in a CSV file with only two columns, "timestamp" and "value"(the column names should be exactly the same). Each CSV file should be named after each variable for the time series. All of the time series should be zipped into one zip file and be uploaded to Azure Blob storage, and there is no requirement for the zip file name. Alternatively, an extra meta.json file can be included in the zip file if you wish the name of the variable to be different from the .zip file name. Once we generate blob SAS (Shared access signatures) URL, we can use the url to the zip file for training.

Path path = Paths.get("test-data.csv");
List<String> requestData = Files.readAllLines(path);
List<TimeSeriesPoint> series = requestData.stream()
    .map(line -> line.trim())
    .filter(line -> line.length() > 0)
    .map(line -> line.split(",", 2))
    .filter(splits -> splits.length == 2)
    .map(splits -> {
        TimeSeriesPoint timeSeriesPoint = new TimeSeriesPoint();
        timeSeriesPoint.setTimestamp(OffsetDateTime.parse(splits[0]));
        timeSeriesPoint.setValue(Float.parseFloat(splits[1]));
        return timeSeriesPoint;
    })
    .collect(Collectors.toList());

Integer window = 28;
AlignMode alignMode = AlignMode.OUTER;
FillNAMethod fillNAMethod = FillNAMethod.LINEAR;
Integer paddingValue = 0;
AlignPolicy alignPolicy = new AlignPolicy()
                                .setAlignMode(alignMode)
                                .setFillNAMethod(fillNAMethod)
                                .setPaddingValue(paddingValue);
String source = "YOUR_SAMPLE_ZIP_FILE_LOCATED_IN_AZURE_BLOB_STORAGE_WITH_SAS";
OffsetDateTime startTime = OffsetDateTime.of(2021, 1, 2, 0, 0, 0, 0, ZoneOffset.UTC);
OffsetDateTime endTime = OffsetDateTime.of(2021, 1, 3, 0, 0, 0, 0, ZoneOffset.UTC);
String displayName = "Devops-MultiAD";

ModelInfo request = new ModelInfo()
                        .setSlidingWindow(window)
                        .setAlignPolicy(alignPolicy)
                        .setSource(source)
                        .setStartTime(startTime)
                        .setEndTime(endTime)
                        .setDisplayName(displayName);
TrainMultivariateModelResponse trainMultivariateModelResponse = anomalyDetectorClient.trainMultivariateModelWithResponse(request, Context.NONE);
String header = trainMultivariateModelResponse.getDeserializedHeaders().getLocation();
String[] substring = header.split("/");
UUID modelId = UUID.fromString(substring[substring.length - 1]);
System.out.println(modelId);

//Check model status until the model is ready
Response<Model> trainResponse;
while (true) {
    trainResponse = anomalyDetectorClient.getMultivariateModelWithResponse(modelId, Context.NONE);
    ModelStatus modelStatus = trainResponse.getValue().getModelInfo().getStatus();
    if (modelStatus == ModelStatus.READY || modelStatus == ModelStatus.FAILED) {
        break;
    }
    TimeUnit.SECONDS.sleep(10);
}

if (trainResponse.getValue().getModelInfo().getStatus() != ModelStatus.READY){
    System.out.println("Training failed.");
    List<ErrorResponse> errorMessages = trainResponse.getValue().getModelInfo().getErrors();
    for (ErrorResponse errorMessage : errorMessages) {
        System.out.println("Error code:  " + errorMessage.getCode());
        System.out.println("Error message:  " + errorMessage.getMessage());
    }
}

Detect anomalies

DetectionRequest detectionRequest = new DetectionRequest().setSource(source).setStartTime(startTime).setEndTime(endTime);
DetectAnomalyResponse detectAnomalyResponse = anomalyDetectorClient.detectAnomalyWithResponse(modelId, detectionRequest, Context.NONE);
String location = detectAnomalyResponse.getDeserializedHeaders().getLocation();
String[] substring = location.split("/");
UUID resultId = UUID.fromString(substring[substring.length - 1]);

DetectionResult detectionResult;
while (true) {
    detectionResult = anomalyDetectorClient.getDetectionResult(resultId);
    DetectionStatus detectionStatus = detectionResult.getSummary().getStatus();;
    if (detectionStatus == DetectionStatus.READY || detectionStatus == DetectionStatus.FAILED) {
        break;
    }
    TimeUnit.SECONDS.sleep(10);
}

if (detectionResult.getSummary().getStatus() != DetectionStatus.READY){
    System.out.println("Inference failed");
    List<ErrorResponse> detectErrorMessages = detectionResult.getSummary().getErrors();
    for (ErrorResponse errorMessage : detectErrorMessages) {
        System.out.println("Error code:  " + errorMessage.getCode());
        System.out.println("Error message:  " + errorMessage.getMessage());
    }
}

Export model

Note

The export command is intended to be used to allow running Anomaly Detector multivariate models in a containerized environment. This is not currently not supported for multivariate, but support will be added in the future.

To export your trained model use the exportModelWithResponse.

StreamResponse response_export = anomalyDetectorClient.exportModelWithResponse(model_id, Context.NONE);
Flux<ByteBuffer> value = response_export.getValue();
FileOutputStream bw = new FileOutputStream("result.zip");
value.subscribe(s -> write(bw, s), (e) -> close(bw), () -> close(bw));

Delete model

To delete an existing model that is available to the current resource use the deleteMultivariateModelWithResponse function.

Response<Void> deleteMultivariateModelWithResponse = anomalyDetectorClient.deleteMultivariateModelWithResponse(model_id, Context.NONE);

Run the application

You can build the app with:

gradle build

Run the application

Before running it can be helpful to check your code against the full sample code.

Run the application with the run goal:

gradle run

Clean up resources

If you want to clean up and remove a Cognitive Services subscription, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with the resource group.

Next steps