Custom Semantic Kernel Agent Stuck in GroupChatOrchestration After Yield Return

Environment:

• Language: C#
• Library: Microsoft Semantic Kernel
• Using Azure OpenAI models
• .NET 8

Problem: A custom agent FastRagAgent has been implemented to call a SemanticSearchAgent.RagSearch function. The agent operates correctly on its own, but when integrated into a GroupChatOrchestration, the process hangs after the yield return statement, preventing the manager from proceeding to the next agent.

Code Snippet:

Implementation of the FastRagAgent class that inherits from Agent:

public override async IAsyncEnumerable<AgentResponseItem<ChatMessageContent>> InvokeAsync(
    ICollection<ChatMessageContent> messages,
    AgentThread? thread = null,
    AgentInvokeOptions? options = null,
    [EnumeratorCancellation] CancellationToken cancellationToken = default)
{
    var lastUserMessage = messages?.LastOrDefault(m => m.Role == AuthorRole.User)?.Content ?? string.Empty;
    if (string.IsNullOrEmpty(lastUserMessage)) yield break;

    var result = await _semanticSearchAgent.RagSearch(lastUserMessage);
    var message = new ChatMessageContent
    {
        AuthorName = "SemanticSearchAgent",
        Role = AuthorRole.Assistant,
        Items = new ChatMessageContentItemCollection
        {
            new TextContent { Text = result }
        },
        Metadata = new Dictionary<string, object>
        {
            ["FinishReason"] = "Stop",
            ["CreatedAt"] = DateTime.UtcNow,
            ["Id"] = Guid.NewGuid().ToString()
        }
    };

    // Reached here but stuck afterward (the chat manager should call the next agent, but it does not)
    yield return new AgentResponseItem<ChatMessageContent>(message, thread);
    yield break;
}

GroupChat:

ChatHistory history = [];
ValueTask responseCallback(ChatMessageContent message)
{
    history.Add(message);
    return ValueTask.CompletedTask;
}
var customManager = new CustomGroupChatManager { MaximumInvocationCount = 5 };
GroupChatOrchestration orchestration = new GroupChatOrchestration(
    customManager,
    _fastRagAgent,
    other agents...)
{
    ResponseCallback = responseCallback,
};

InProcessRuntime runtime = new InProcessRuntime();
await runtime.StartAsync();

The setup works when agents are wrapped with chatCompletionAgent, but this increases latency. What can be done to address this issue?