OpenAI Realtime API transcription with GPT-4o-transcribe, WS connection is forcefully closed with code 1006.

Context:

I am building a voice-to-voice conversational AI service, which obviously must include a STT service. For this, I have selected to deploy GPT-4o-transcribe, with the OpenAI Realtime API (hosted by Azure, Azure OAI service).
I have followed the documentation given by OpenAI at:
https://platform.openai.com/docs/api-reference/realtime
(I understand that there is a difference between OpenAI and Azure OAI, regarding architecture etc., but the documentation seems to be correct regarding Azure OAI service as well).

I can start the connection and get transcriptions. Everything is working great, but, at random points in time, the connection is closed, with code 1006. No reason is given. And no previous events show any sign of failures. As mentioned, this happens randomly, thus it being very hard to replicate the issue.

Technical implementation:

I run my own VAD (Voice Activity Detection) model, so I do not rely on the VAD provided by the Realtime service. Moreover, audio is continuously sent from the client, to my server, as small chunks. Here is the current flow:

At startup, the connection to the Realtime service is created, with:
Target URL:

wss://${AZURE_RESOURCE}/openai/v1/realtime?deployment=${AZURE_DEPLOYMENT}&intent=transcription;

Headers:

{
  headers: {"api-key": KEY}
}

And session.update:

{
  type: `session.update`,
       session: {
       type: `transcription`,
       include: [
          `item.input_audio_transcription.logprobs`
       ],
       audio: {
             input: {
                  format: {
                         rate: 24000, 
                         type: `audio/pcm`
                   },
                   noise_reduction: {
                      type: `near_field`
                   },
                   turn_detection: null,
                   transcription: {
                       language: `en`,
                       model: AZURE_DEPLOYMENT,
                       prompt: PROMPT
                    }

              }
        }
   }
}

Then, the flow is as follows:

1. VAD emits that user is speaking;
2. This triggers a message to the Realtime Service:

{ type: `input_audio_buffer.clear` }

3. Every chunk of audio is sent as PCM 24k, base64 encoded, to the Realtime Service as:

{
   type: input_audio_buffer.append,
   audio: BASE_64_PCM_24
}

4. At some point, the VAD emits that user has stopped speaking, which triggers this message to the Realtime service:

{ type: `input_audio_buffer.commit` }

5. A result is obtained, through the Realtime service message:

conversation.item.input_audio_transcription.completed

Then N seconds passes, until VAD emits that the user is speaking, and the process repeats from 1).
Moreover, other messages from the Realtime service are handled, like errors, etc.

Problem:

Occasionally, the connection is closed, with code 1006. That is, the connection is not closed by the client, but on the server side (Realtime Service server):

Sometimes this happens while transcription is running (that is, while step 3 is running, and audio is being comitted).
Sometimes this happens when the server is in waiting position (before step 1 has fired).
The problem occurs between 5-15 minutes of opening the connection.

Notes:

I have implemented a reconnection functionality, after 25min has passed from opening the connection. This is due to the 30min hard limit, which the connection can live. So this is not the problem.

I run multiple workers of the type described above, one for each current client. Between 0-10 workers exist in the server at any given time. Perhaps multiple concurrent connections are not allowed?
As mentioned, the Realtime service does not emit any error messages before closing the connection with 1006. I am watching these types of error messages:

error

conversation.item.input_audio_transcription.failed

Perhaps there are more to check for.

Any ideas to why the connection is closed in such a manner?
Thanks!