How to build a load balancer for Cognitive Services

Ning Zhu 66

We have set up two Cognitive Service Instances and we want to know whether that is possible to set up a load balancer for the two instances, so developer no need to switch between them?

romungi-MSFT 41,961 Reputation points Microsoft Employee

2023-02-06T10:59:40.1633333+00:00

Ning Zhu If you are using a cognitive service resource endpoints directly in your application and looking to balance the outgoing requests to different resources then I don't think there is a ready made offering from cognitive services to redirect traffic to these endpoints. This is mostly managed on the client side and in some cases, customers are using containers for cognitive services and scaling the services with AKS or ACI.
Ning Zhu 66 Reputation points

2023-02-09T01:28:35.4566667+00:00

There are several layer 7 load balancer such as Front Door, Application Gateway, nothing can be used here?
romungi-MSFT 41,961 Reputation points Microsoft Employee

2023-02-09T13:25:38.94+00:00

I do not have much experience with load balancing applications but for scenarios where you use authentication with the subscription keys as headers for a cognitive resource, a load balancer might still have to route the request to the endpoint that is used in the original request since authentication will fail if the request is routed to the second cognitive service resource.