How to optimize amount of data sent via LogsIngestionClient.upload operation

Question

Hi,
I am using logs ingestion client in python to upload data.

My usecase is to read messages off of aws sqs and build payloads that can be sent via LogsIngestionClient client.

I built a simple timer trigger function app that reads aws sqs for new notifications, parses them for link and uploads that data using LogsIngestionClient.

Here is a snippet of the code,

    async def _post_data(self, dce_endpoint, dcr_id, stream_name, credential, data):                            
        client = LogsIngestionClient(endpoint=dce_endpoint, credential=credential, logging_enable=False)
        async with client:
            try:
                await client.upload(rule_id=dcr_id, stream_name=stream_name, logs=data)
            except HttpResponseError as e:
                print(f"Upload failed: {e}")   

    async def _flush(self, data: list):
        if data:
            #data = self._split_big_request(data)
            credential = DefaultAzureCredential()
            await self._post_data(self.dce_endpoint, self.dcr_id, self.stream_name, credential, data)

What I am noticing is, although there are lots of messages in the queue, the function app is very slow in consuming the messages and number of log rows that are being processed per min are also low.

This was obtained from metrics, even though, log ingestion byte per min is increasing, the log rows received is still capped at 500, which is strange.
I am using asyncio/await/async to send data.

How can I resolve this bottleneck and have ingestion of data improved?

Answer 1

To optimize the amount of data sent via LogsIngestionClient.upload operation, you can consider tuning the transfer options of the Azure Storage client library for Python. This can be done by configuring the client library transfer options based on the environment your app runs in.

Some considerations for tuning data transfer options include setting the chunk size for uploads and downloads, using parallelism for uploads and downloads, and using compression for uploads.

In your specific case, you may want to experiment with adjusting the chunk size and parallelism settings to see if it improves the performance of your LogsIngestionClient.upload operation. Additionally, you may want to consider using compression for uploads to reduce the amount of data being sent over the network.

---

References:

• Performance tuning for uploads and downloads with Python

Answer 2

Hello @Ashwin Venkatesha

Based on the code snippet you provided, it seems like you are using asyncio to send data to Azure Monitor Logs, is this correct?

Iy's not clear how you are reading messages from AWS SQS. It's possible that the bottleneck is not in the LogsIngestionClient, but rather in the code that reads messages from SQS.

Here are some general tips to improve the performance of your Azure Functions app:

1. Avoid long running functions: Large, long-running functions can cause unexpected timeout issues. Refactor large functions into smaller function sets that work together and return responses fast. Durable Function Orchestrator would be ideal here.
2. Make sure background tasks complete: When your function starts any tasks, callbacks, threads, processes, or tasks, they must complete before your function code returns. Because Functions doesn't track these background threads, site shutdown can occur regardless of background thread status, which can cause unintended behavior in your functions.
3. Cross function communication: If you are not using Durable Functions or Logic Apps to integrate with multiple functions, it's best to use storage queues for cross-function communication. The main reason is that storage queues are cheaper and much easier to provision than other storage options.
4. Use multiple language worker processes: By default, every Functions host instance has a single language worker process. You can increase the number of worker processes per host (up to 10) by using the FUNCTIONS_WORKER_PROCESS_COUNT application setting. Azure Functions then tries to evenly distribute simultaneous function invocations across these workers.
5. Use asyncio compatible libraries: We strongly encourage our customers to make use of asyncio compatible libraries, such as aiohttp and pyzmq. Following these recommendations increases your function's throughput compared to those libraries when implemented synchronously.

The above steps should help you achieve your solution. I hope this helps.