ONNX

ONNX Runtime GenAI lets a .NET Agent Framework application run a compatible model locally. Use it for offline development, on-device inference, or deployments where model execution must stay on the host.

Warning

Treat model or embedding output, and model-generated tool calls when supported, as untrusted application input. Validate them before code performs side effects or renders content. If you configure a custom endpoint, verify that you trust its URL and operator.

Note

The current ONNX client doesn't support function calling. Function tools passed to the agent are ignored.

Prerequisites

  • .NET 8 or later.
  • A model exported for ONNX Runtime GenAI.
  • Sufficient local memory and a compatible execution provider for the selected model.

Install the packages

dotnet add package Microsoft.ML.OnnxRuntimeGenAI
dotnet add package Microsoft.Agents.AI --prerelease

Configuration

ONNX_MODEL_PATH="<path-to-onnx-runtime-genai-model-directory>"

Create an ONNX-backed agent

Download a model exported for ONNX Runtime GenAI and point ONNX_MODEL_PATH to the model directory.

The model files, execution provider, quantization, and available memory determine hardware compatibility and performance. Review the model license before redistributing it.

Tools

The current ONNX client doesn't support function calling or provider-hosted tools.

Next steps