Bulk records to download from azure data explorer

Question

Hi,

I want to download/exporttoCSV logs/records from azure data explorer for past 3 months for one particular table for firewall.
The criteria is pretty wide because of which the number of records for just 1 day is around 1Million for the required criteria.
I need it for 3 months .

I have used "set truncationmaxsize" and "set truncationmaxrecords" also tried with set notruncation, increased connection time to 60 minutes, also kept time range for a maximum of 3 days at a time. Its completing run for 60 minutes and then failing .

Please let me know how can I download such bulk logs/record from adx.

Thanks in Advance.

Answer 1

Hi @Soumya Banerjee ,

Thank you for posting query in Microsoft Q&A Platform.

I will suggest to leverage Azure data factory for your requirement.

Azure Data Factory is Azure's cloud ETL service for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring and single-pane-of-glass monitoring and management.

We can utilize Azure data explorer connector in ADF, to copy data to or from Azure Data Explorer. Click here to know more about same.

Hope this will help. Please let us know if any further queries.

--------------

Please consider hitting Accept Answer button. Accepted answers helps community as well.