Video Analyzer FAQ

What system variables can I use in the pipeline topology definition?

What is the privacy policy for Video Analyzer?

Video Analyzer is covered by the Microsoft Privacy Statement. The privacy statement explains the personal data Microsoft processes, how Microsoft processes it, and for what purposes Microsoft processes it. To learn more about privacy, visit the Microsoft Trust Center.

Can I deploy the edge module to a Windows 10 device?

Yes. For more information, see Linux containers on Windows 10.

I am using a virtual machine as simulated edge device. Can I resize the VM config after deployment?

Yes, you can scale a VM up or down to a different size after edge modules such as Video Analyzer have been deployed to it. You may need to restart the IoT Edge runtime on your virtual machine. For a Linux VM, use command sudo systemctl restart iotedge. If live pipelines had been running, they should resume after the restart completes. You would see interruptions in the output from the pipeline (example: inference events, video recording) during the restart.

How can I add sample video files to the RTSP camera simulator module on my IoT Edge device?

If you have used a quickstart or tutorial to install the camera simulator, then to add a new video file:

• Check that the file is in a supported format and uses the H.264 video codec
• Download the file to /home/localedgeuser/samples/input/<videofile.mkv> on your IoT Edge device, as shown in this section

Sample media files available for testing can be found at Media datasets.

Do I need to use a special SDK on my device to send in a video stream?

No, Video Analyzer supports capturing media by using RTSP (Real-Time Streaming Protocol) for video streaming, which is supported on most IP cameras.

Can I push media to Video Analyzer by using protocols other than RTSP?

No, Video Analyzer supports only RTSP for capturing video from IP cameras. Any camera that supports RTSP streaming over TCP/HTTP should work.

Can I reset or update the RTSP source URL in a live pipeline?

Yes, when the live pipeline is in inactive state.

Is an RTSP simulator available to use during testing and development?

Yes, an RTSP simulator edge module is available for use in the quickstarts and tutorials to support the learning process. This module is provided as best-effort and might not always be available. We recommend strongly that you not use the simulator for more than a few hours. You should invest in testing with your actual RTSP source before you plan a production deployment.

I have multiple AI models wrapped in a Docker container. How should I use them with Video Analyzer?

Solutions vary depending on the communication protocol that's used by the inferencing server to communicate with Video Analyzer. The following sections describe how each protocol works.

Use the HTTP protocol:

• Single container (module named as avaextension):

  In your inferencing server, you can use a single port but different endpoints for different AI models. For example, for a Python sample you can use different routes per model as shown here:

  @app.route('/score/face_detection', methods=['POST']) 
  … 
  Your code specific to face detection model
  
  @app.route('/score/vehicle_detection', methods=['POST']) 
  … 
  Your code specific to vehicle detection model 
  … 
  

  And then in your Video Analyzer deployment, when you activate live pipelines set the inference server URL for each one as shown here:

  1st live pipeline: inference server URL=http://avaextension:44000/score/face_detection
  2nd live pipeline: inference server URL=http://avaextension:44000/score/vehicle_detection

  Note

  Alternatively, you can expose your AI models on different ports and call them when you activate live pipelines.

• Multiple containers:

  Each container is deployed with a different name. In the quickstarts and tutorials, we showed you how to deploy an extension named avaextension. Now you can develop two different containers, each with the same HTTP interface, which means they have the same /score endpoint. Deploy these two containers with different names, and ensure that both are listening on different ports.

  For example, one container named avaextension1 is listening for the port 44000, and a second container named avaextension2 is listening for the port 44001.

  In your Video Analyzer topology, you instantiate two live pipelines with different inference URLs, as shown here:

  1st live pipeline: inference server URL = http://avaextension1:44000/score
  2nd live pipeline: inference server URL = http://avaextension2:44001/score

Use the gRPC protocol:

• The gRPC extension node has a property extensionConfiguration, an optional string that can be used as a part of the gRPC contract. When you have multiple AI models packaged in a single inference server, you don't need to expose a node for every AI model. Instead, for a live pipeline, you, as the extension provider, can define how to select the different AI models by using the extensionConfiguration property. During execution Video Analyzer passes this string to your inferencing server, which can use it to invoke the desired AI model.

I'm building a gRPC server around an AI model, and I want to be able to support its use by multiple cameras or live pipelines. How should I build my server?

First, be sure that your server can either handle more than one request at a time or work in parallel threads.

For example, a default number of parallel channels has been set in the following Azure Video Analyzer gRPC sample:

server = grpc.server(futures.ThreadPoolExecutor(max_workers=3)) 

In the preceding gRPC server instantiation, the server can open only three channels at a time per camera, or per live pipeline. Don't try to connect more than three instances to the server. If you do try to open more than three channels, requests will remain pending until an existing channel drops.

The preceding gRPC server implementation is used in our Python samples. As a developer, you can implement your own server or use the preceding default implementation to increase the worker number, which you set to the number of cameras to use for video feeds.

To set up and use multiple cameras, you can instantiate multiple live pipelines, each pointing to the same or a different inference server (for example, the server mentioned in the preceding paragraph).

I want to be able to receive multiple frames before I make an inferencing decision. How can I enable that?

Our current default samples work in a stateless mode. They don't keep the state of the previous calls, or the ID of the caller. This means that multiple live pipelines might call the same inference server, but the server can't distinguish who is calling or the state per caller.

Use the HTTP protocol:

To keep the state, each caller, or live pipeline, calls the inferencing server by using the HTTP query parameter that's unique to caller. For example, the inference server URL addresses for each live pipeline are shown here:

1st live pipeline: http://avaextension:44000/score?id=1
2nd live pipeline: http://avaextension:44000/score?id=2

…

On the server side, the id helps identify the caller. If id=1, then the server can keep the state separately for that live pipeline. It can then keep the received video frames in a buffer. For example, use an array, or a dictionary with a DateTime key, and the value is the frame. You can then define the server to process (infer) after x number of frames are received.

Use the gRPC protocol:

With a gRPC extension, each session is for a single camera feed, so there's no need to provide an identifier. With the extensionConfiguration property, you can store the video frames in a buffer and define the server to process (infer) after x number of frames are received.

Do all ProcessMediaStreams on a particular container run the same AI model?

No. Start or stop calls from the end user in a live pipeline constitute a session, or perhaps there's a camera disconnect or reconnect. The goal is to persist one session if the camera is streaming video.

• Two cameras sending video for processing (to two separate live pipelines) creates two sessions.
• One camera going to a live pipeline that has two gRPC extension nodes creates two sessions.

Each session is a full duplex connection between Video Analyzer and the gRPC server, and each session can have a different model.

Video Analyzer also adds support for multiple gRPC extensions for a single camera in a live pipeline. You can use these gRPC extensions to carry out AI processing sequentially, in parallel, or as a combination of both.

What is the maximum number of simultaneous ProcessMediaStreams?

Video Analyzer applies no limits to this number.

How can I decide whether my inferencing server should use CPU or GPU or any other hardware accelerator?

Your decision depends on the complexity of the developed AI model and how you want to use the CPU and hardware accelerators. As you're developing the AI model, you can specify what resources the model should use and what actions it should perform.

How do I view the bounding boxes generated by my inference server?

You can record the inference results along with the media in your video resource. You can use a widget to play back the video with an overlay of the inference data.

How will I know what the mandatory fields for the media stream descriptor are?

Any field that you don't supply a value to is given a default value, as specified by gRPC.

Video Analyzer uses the proto3 version of the protocol buffer language. All the protocol buffer data that's used by Video Analyzer contracts is available in the protocol buffer files.

How can I ensure that I'm using the latest protocol buffer files?

You can obtain the latest protocol buffer files on the contract files site. Whenever we update the contract files, they'll be in this location. There's no immediate plan to update the protocol files, so look for the package name at the top of the files to know the version. It should read:

microsoft.azure.media.live_video_analytics.extensibility.grpc.v1

Any updates to these files will increment the "v-value" at the end of the name.

What gRPC features are available for me to use with Video Analyzer? Which features are mandatory and which are optional?

You can use any server-side gRPC features, provided that the Protocol Buffers (Protobuf) contract is fulfilled.

Can I monitor the pipeline on the edge by using Azure Event Grid?

Yes. You can consume Prometheus metrics and publish them to your event grid.

Can I use Azure Monitor to view the health, metrics, and performance of my pipelines in the cloud or on the edge?

Yes, we support this approach. To learn more, see Azure Monitor Metrics overview.

Are there any tools to make it easier to monitor the Video Analyzer edge module?

Visual Studio Code supports the Azure IoT Tools extension, with which you can easily monitor the Video Analyzer edge module endpoints. You can use this tool to quickly start monitoring your IoT hub built-in endpoint for "events" and view the inference messages that are routed from the edge device to the cloud.

In addition, you can use this extension to edit the module twin for the Video Analyzer edge module to modify the pipeline settings.

For more information, see the monitoring and logging article.

How is Video Analyzer billed?

For billing details, see Video Analyzer pricing.

Next steps

Quickstart: Get started with Azure Video Analyzer