How to setup real time inferencing for robotic application?

Question

How to setup real time inferencing for robotic application?

Derick John 20

I'm new to Azure and would like to know my best option for an application where I need to send real-time video data to a model for inferencing and then get the predicted actions. I already have the setup for model training and the model registered in ML studio.
I've a robot client and since it doesn't have a GPU on itself, it will be using Azure resources to do actions given the live video feed. I've looked into creating online endpoints which it says is for real time inferencing, but in the docs the data that is passed to do inferencing are all json files. I've also looked into AKS, but it feels like things are same here as well.

I would appreciate any help and I'm also open to receive tech support from Azure team as well.

Prashanth Veeragoni 4,930 Reputation points Microsoft External Staff Moderator

2025-06-26T10:13:05.6033333+00:00
Hi Derick John,

I Understand that you are building a robotic system where the robot captures live video and sends it to a machine learning model for real-time inferencing. Since the robot itself lacks a GPU, all model inference must happen in the cloud — specifically, using Azure resources.

Azure Machine Learning Studio supports this setup via Online Endpoints, which are designed for real-time inferencing workloads. Although most documentation examples show JSON inputs (typically for tabular or text data), image or video frame data can also be used, with appropriate formatting and handling.

Best Solution:

Azure ML + Online Endpoint + Preprocessing:

Azure ML Online Endpoints can be used for real-time inference, but the catch is:

• The endpoint expects JSON data (as you've seen).

• But you can preprocess the video on the robot, convert it into frames (images), and send it frame-by-frame (base64 encoded or binary) to the endpoint.

• On the Azure side, you decode the image, run the inference, and send back action predictions.

Endpoint Input Format Options:

Option A: Base64 Image

• Robot encodes image frame as base64 and sends in JSON.

• Example payload:

{ "image": "<base64-encoded string>" }

Option B: Multipart Binary

• If latency is a concern and payloads are large, you can enable multipart/form-data for image upload (slightly advanced, custom handling in scoring script).

Sample Scoring Script (Azure ML Endpoint):

# score.py import base64 import cv2 import numpy as np from io import BytesIO from azureml.core.model import Model def init(): global model # Load your model (e.g., ONNX, PyTorch, etc.) model_path = Model.get_model_path('your_model_name') model = load_model(model_path) # Implement your model loading logic def run(raw_data): try: data = json.loads(raw_data) img_data = base64.b64decode(data['image']) np_arr = np.frombuffer(img_data, np.uint8) image = cv2.imdecode(np_arr, cv2.IMREAD_COLOR) # Preprocess and predict prediction = model_infer(image) # Return the action return {"action": prediction} except Exception as e: return {"error": str(e)}

Robot Client (Python Example):

import cv2 import requests import base64 import json cap = cv2.VideoCapture(0) while True: ret, frame = cap.read() _, buffer = cv2.imencode('.jpg', frame) img_str = base64.b64encode(buffer).decode() payload = json.dumps({"image": img_str}) headers = {"Content-Type": "application/json"} response = requests.post("<your_online_endpoint_url>", headers=headers, data=payload) print("Predicted action:", response.json())

Alternative with AKS

If latency, autoscaling, or GPU use becomes critical:

• Deploy your model as a real-time service on AKS via Azure ML.

• AKS gives better control over containers, scaling, and lower latency than managed endpoints.

• Same architecture, but with more flexibility and complexity.

Hope this helps, do let me know if you have any further queries.

Thank you
Derick John 20 Reputation points

2025-06-26T17:38:44.6766667+00:00

Thanks @Prashanth Veeragoni I'll look into this.
If I face any issues with setting up the AKS, how can I contact you or Azure support?

Derick John 20

Hi @Prashanth Veeragoni I've currently set up my robot client code as:


while True:
        # Capture observation
        observation = robot.capture_observation()

        # Convert to JSON-serializable format
        input_data = {}
        for name, tensor in observation.items():
            if "image" in name:
                tensor = tensor.type(torch.float32) / 255
                tensor = tensor.permute(2, 0, 1).contiguous()
            input_data[name] = tensor.unsqueeze(0).cpu().numpy().tolist()

        # Send request to Azure ML endpoint
        headers = {
            "Content-Type": "application/json",
        }

        response = requests.post(endpoint_url, headers=headers, data=json.dumps(input_data))

        # Handle response
        if response.status_code == 200:
            result = response.json()
            action_tensor = torch.tensor(result["action"], dtype=torch.float32)
            robot.send_action(action_tensor)
        else:
            print("Error:", response.status_code, response.text)

And the run function in my scoring script as:

def run(raw_data):
    try:
        # Parse input JSON
        if isinstance(raw_data, str):
            raw_data = json.loads(raw_data)

        # Convert input to tensors
        observation = {}
        for key, value in raw_data.items():
            tensor = torch.tensor(value, dtype=torch.float32)
            observation[key] = tensor.to(device)

        # Run inference
        with torch.no_grad():
            action = policy.select_action(observation).squeeze(0).cpu().numpy()

        return {"action": action.tolist()}

    except Exception as e:
        return {"error": str(e)}

I'm a bit confused about how I can enable multipart/form-data in the scoring script. I guess, in my case, using multipart/form-data would be suitable for real-time inference.
Also, it would be the same if I went with AKS endpoint deployment, right?

Your answer

Derick John 20 Reputation points

2025-06-26T17:38:44.6766667+00:00

Thanks @Prashanth Veeragoni I'll look into this.
If I face any issues with setting up the AKS, how can I contact you or Azure support?
Derick John 20 Reputation points

2025-06-27T09:05:42.38+00:00

Hi @Prashanth Veeragoni I've currently set up my robot client code as:

while True: # Capture observation observation = robot.capture_observation() # Convert to JSON-serializable format input_data = {} for name, tensor in observation.items(): if "image" in name: tensor = tensor.type(torch.float32) / 255 tensor = tensor.permute(2, 0, 1).contiguous() input_data[name] = tensor.unsqueeze(0).cpu().numpy().tolist() # Send request to Azure ML endpoint headers = { "Content-Type": "application/json", } response = requests.post(endpoint_url, headers=headers, data=json.dumps(input_data)) # Handle response if response.status_code == 200: result = response.json() action_tensor = torch.tensor(result["action"], dtype=torch.float32) robot.send_action(action_tensor) else: print("Error:", response.status_code, response.text)

And the run function in my scoring script as:

def run(raw_data): try: # Parse input JSON if isinstance(raw_data, str): raw_data = json.loads(raw_data) # Convert input to tensors observation = {} for key, value in raw_data.items(): tensor = torch.tensor(value, dtype=torch.float32) observation[key] = tensor.to(device) # Run inference with torch.no_grad(): action = policy.select_action(observation).squeeze(0).cpu().numpy() return {"action": action.tolist()} except Exception as e: return {"error": str(e)}

I'm a bit confused about how I can enable multipart/form-data in the scoring script. I guess, in my case, using multipart/form-data would be suitable for real-time inference.
Also, it would be the same if I went with AKS endpoint deployment, right?

Share via

How to setup real time inferencing for robotic application?

Your answer