API Limits and Performance of Lanugage Service

Prakhar Birla 1

I'm trying to use the Converational Language Understanding (CLU) of the Cognitive Language Service in an high-performance use case where I'm trying to make say 50 calls per second (TPS) to this service.

The documentation says the limit is 1000 TPM for the Prediction Service. Which would be ~16 TPS, much lower that what I need to achieve. Earlier LUIS would allow the deployment of multiple prediction resources paired with one Authoring resource for scaling further. Also, LUIS allowed containerized prediction deployment and would get ~40 TPS with a 1-core 4GB RAM machine.

Now I'm getting ~200 ms avg with ~16 TPS with the CLU service and the consumer in the same region. How can I scale this setup?

Ramr-msft 17,611 Reputation points

2022-12-29T13:31:38.903+00:00

@Prakhar Birla Thanks for the question. We would recommend using the higher tier: CLU offers different pricing tiers with different limits and performance characteristics. You can try using a higher tier of the service to see if it can handle the higher throughput you need.
Prakhar Birla 1 Reputation point

2022-12-29T14:42:11.847+00:00

Thanks @Ramr-msft for getting back so quickly. I see only these two options:

Is there a premium tier as well, that I don't have access to? I'm assuming "S" implies Standard.
Ramr-msft 17,611 Reputation points

2022-12-30T03:32:51.617+00:00

@Prakhar Birla Thanks for the details. I am checking internally on this will update on the same.
Prakhar Birla 1 Reputation point

2023-01-02T08:26:13.193+00:00

Hey @Ramr-msft , looking forward to your response.