Voice Live SDK – Interim Responses Accepted but Never Emitted for Azure AI Foundry Agent

Question

Voice Live SDK – Interim Responses Accepted but Never Emitted for Azure AI Foundry Agent

Hi Microsoft team,

I'm investigating an issue with Voice Live interim (filler) responses when connecting to an Azure AI Foundry Agent via the .NET Voice Live SDK.

Scenario

I have an Azure AI Foundry Agent with Voice-First Experience enabled. The agent produces interim responses correctly in the Foundry portal playground.

However, when connecting to the same agent using the Voice Live SDK and configuring interim responses through AgentSessionConfig / SessionTarget.FromAgent(...), the service never emits any interim response events, despite appearing to accept and store the configuration.

Investigation

1. JSON serialization bug (fixed)

The original implementation serialized the LlmInterimResponseConfig object using:

BinaryData.FromObjectAsJson(...)

This uses reflection-based JSON serialization and does not respect the SDK's custom wire-format serializer.

As a result, the payload contained:

• PascalCase property names
• Empty Triggers array
• Missing type discriminator

The service could not correctly interpret the payload.

Fix implemented:

ModelReaderWriter.Write(...)

from System.ClientModel.Primitives.

The resulting payload now matches the documented wire format exactly, including:

{
"type": "llm_interim_response",
"triggers": ["tool_start"],
"latency_threshold_ms": 3000
}

and other documented snake_case properties.

Using reflection and payload inspection, I confirmed the serialized output matches the Microsoft documentation.

---

2. API version ruled out

I verified the SDK exposes:

• V2025_10_01
• V2026_01_01_PREVIEW

Documentation indicates interim responses require:

2026-01-01-preview

The client is explicitly configured to use this preview API version.

Result:

• No behavioral change.

---

3. Server confirms configuration is accepted

To verify the service receives the configuration, I added logging for the session.updated event and inspected the returned session state.

The server echoes back the full interim-response configuration, including default values populated by the service (for example max_completion_tokens: 50).

This indicates:

✅ Request reaches the service

✅ Request is successfully parsed

✅ Configuration is stored by the session

Example observation:

{
"interim_response": {
...
"max_completion_tokens": 50
}
}

The presence of service-generated defaults strongly suggests the server has accepted and normalized the configuration.

---

4. Full event-stream audit

To determine whether interim responses were being emitted under a different event type, I implemented a catch-all logger that records every incoming event and raw payload from the server.

Testing included:

• Multiple conversational turns
• Deliberate pauses
• Tool-calling scenarios
• A live web-search tool invocation (which should satisfy the tool_start trigger use case)

Results:

• Every server event was captured and classified
• No interim/filler response event was ever emitted

The only previously unrecognized events were:

response.web_search_call.*

These appear to be unrelated SDK gaps around tool-call event models and are not connected to interim-response functionality.

Across all test runs, zero interim response events were observed.

---

Conclusion

The evidence suggests:

1. The client now sends a schema-correct request.
2. The Voice Live service accepts and stores the configuration.
3. The session reflects the configured interim-response settings.
4. The service never actually emits or executes interim responses when connected through this integration path.

The same Azure AI Foundry Agent does generate interim responses in the Foundry portal playground.

This suggests there may be a functional gap between:

• Foundry Portal / Hosted Agent experience
• Voice Live SDK + SessionTarget.FromAgent(...) / AgentSessionConfig

or some limitation specific to agents with Voice-First Experience enabled when accessed through the Agent Service integration route.

At this point I can no longer identify any client-side issues that would prevent interim responses from working.

Questions

Are interim responses currently supported when using:

• SessionTarget.FromAgent(...)
  • AgentSessionConfig
    • Azure AI Foundry Agent Service integration
    Is there a known limitation or preview gap that prevents interim responses from being emitted through this connection path? Is the Foundry portal playground using a different backend integration that supports interim responses while the Agent Service path does not? Can Microsoft confirm whether this scenario should currently work end-to-end?

Any guidance would be appreciated. I can provide SDK version details, event logs, serialized payload samples, and a minimal reproducible example if needed.

Thanks.

Answer 1

Your trace shows the service accepts the session update but no bridging audio is produced through SessionTarget.FromAgent. Current Microsoft documentation explicitly supports interim responses in Foundry Agent mode, so this is not a documented agent-mode limitation. The current .NET quickstart uses Azure.AI.VoiceLive 1.1.0-beta.3, starts the session with AgentSessionConfig, then sends a VoiceLiveSessionOptions update whose InterimResponse is built from LlmInterimResponseConfig.Align the test exactly with that sample: upgrade the prerelease package, configure both InterimResponseTrigger.Tool and Latency, set a low latency threshold, use BinaryData.FromObjectAsJson(interimConfig), and call ConfigureSessionAsync only after StartSessionAsync completes. Do not send tool_start; the documented trigger value is tool. Test latency without a tool, then test model mode to isolate the Agent Service path. If agent mode alone still emits nothing, collect the session ID, UTC time, region, SDK/API versions, and raw event stream and open Microsoft Support; the public docs do not identify a portal-only backend.