how to fix read tcp i/o timeout on Azure cosmos db Cassandra API

Question

Hi Support,

I am using Cassandra API of Azure cosmos db to persist traces from Jaeger collector deployed on AKS and frequently I see tcp read i/o timeouts. Could you please let me know if anything to be corrected from database side?

I asked similar question on Jaeger forum and awaiting for the response. https://github.com/orgs/jaegertracing/discussions/4935

failed to Exec query '[query statement="\n\t\tINSERT\n\t\tINTO tag_index(trace_id, span_id, service_name, start_time, tag_key, tag_value)\n\t\tVALUES (?, ?, ?, ?, ?, ?)" values=[4f7fa873e287659bc8c31e1f0c67682e -3980304526964463570 demo-app.gef2 1700123674433403 grpc.status_code 0] consistency=LOCAL_ONE]': read tcp 192.168.138.99:47498->10.114.58.111:10350: i/o timeout

Regards,

Jay

Thank you,

Jay

Answer 1

Hi Jayarami Annapureddy •,

Welcome to Microsoft Q&A forum.

As I understand, you are getting IO timeout error in Cassandra API of Cosmos DB.

If this is related to payload, then exception during a write request is because of payload maybe too large. Currently, there is a limit of 2MB per row.

Server-side retry (SSR) is most beneficial when there is a sudden spike for a short duration of less than 1 minute where in throttling errors can be avoided. If the work load has increased and would stay constantly above the specified RU, then SSR will not help much. The suggestion is to increase the RU appropriately.

After SSR is enabled, the client app should increase read timeout beyond the server retry 60-second setting. We recommend 90 seconds to be on the safer side.

Code Sample Driver3

SocketOptions socketOptions = new SocketOptions() .setReadTimeoutMillis(90000); Code Sample Driver4

ProgrammaticDriverConfigLoaderBuilder configBuilder = DriverConfigLoader.programmaticBuilder() .withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofSeconds(90));

If above does not help, I would recommend you to please file a support ticket for deeper investigation and in case if you don't have a support plan, do let us know here so that we can check on other options to unblock you.

Thanks