I have a netTcpBinding service that is called from another service and it works more or less. I can stream audio from one service to the other but at some point the CPU drops from about 26% to 0% and the service does not receive any more audio.
The way it is setup, is that the first service is getting audio streamed via UDP from a client. The service, upon receiving the audio, converts it to another format and send it to the other service using netTcpBinding.
So what I see it that, when increase the audio flow, to the first service, it goes as expected op till about 15 Mbps. the CPU follows and goes op steadily. At some point, a unknown limit is hit and the service stops sending anymore audio to the other service.
These are my observations:
- From performance counters, I see that the second that the CPU drops, the counter called "Pool Nonpaged Bytes" sky rockets.
- If I drop the audio instead of converting and sending it to the second service, there is no drop in CPU and no problems, other than the system is broken :D.
- CPU and Memory is not stressed at all.
I am not sure if this is caused by the fact that audio is propagated from the receiving thread all the way to the receiving thread on the second service in one go. What I mean is, I don't have a queue where audio is put upon receiving it and then another thread fetching and sending it to the other service. The service that receives the converted audio has this behavior
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
i have the following config setting:
<serviceThrottling maxConcurrentSessions="4000" maxConcurrentCalls="4000" />
The number of connections between the two services is 1 so i'm not hitting any maximum TCP port limit.
this is the configured binding that I have:
<binding name="RSRStreamingServiceBinding" maxReceivedMessageSize="2147483647" maxBufferSize="2147483647" maxBufferPoolSize="2147483647" receiveTimeout="00:10:00">
<reliableSession inactivityTimeout="00:10:00" />
<readerQuotas maxArrayLength="2147483647" maxStringContentLength="6291456" />
I don't understand how the CPU usage of the service can drop to 0 in a split second. I think that the reason that the "Pool Nonpaged Bytes" goes op is because the first service is trying to send audio but can't for some reason. I guess that it is because the second service is refusing to receive anymore, but i don't understand why.
I have looked at a lot of performance counter and could not find any that shed some light on the matter.
Can anybody help me understand this problem a little bit better and maybe point me to the next point of investigation.