I have to ingest 380 million data from azure sql to azure data explorer through azure data factory, can anyone tell me what settings do we make in ADF(DIU's) such that all the data would be ingested at one go consistently. Network Bandwith is 120 Mbps Thank you

Hi @Matam, Savitha , Thank you for posting query in Microsoft Q&A Platform. Azure data factory and Azure Data explorer documentations, have complete details of integrating ADF and Data explorer. Kindly check below shared links for same. Integrate Azure Data Explorer with Azure Data Factory Copy data to or from Azure Data Explorer using Azure Data Factory or Synapse Analytics Hope this helps. Please let us know if any further queries. ----------- Please consider hitting Accept Answer button. Accepted answers help community as well.

ingesting data to azure data exlorer through azure data factory

Accepted answer

ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee

2022-08-18T11:19:57.447+00:00

Hi @Matam, Savitha ,

Thank you for posting query in Microsoft Q&A Platform.

Azure data factory and Azure Data explorer documentations, have complete details of integrating ADF and Data explorer. Kindly check below shared links for same.

Integrate Azure Data Explorer with Azure Data Factory
Copy data to or from Azure Data Explorer using Azure Data Factory or Synapse Analytics

Hope this helps. Please let us know if any further queries.

-----------

Please consider hitting Accept Answer button. Accepted answers help community as well.
Please sign in to rate this answer.
Matam, Savitha 26 Reputation points

2022-08-19T12:45:45.833+00:00

Hi @ShaikMaheer-MSFT ,

Thanks for the quick reply.

I increased DIU to 32 but still the copy activity is taking the default setting (DIU, 4) when tried to ingest 80million data to ADX, it took 7hrs, 40 mins time to ingest.

Also,

I have used the BulkcopyDBfromSql to ADX template , but i am getting the below error when tried to Debug

" Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Incorrect syntax near the keyword 'SELECT'.
Incorrect syntax near the keyword 'as'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Incorrect syntax near the keyword 'SELECT' "

Please help me with the best option , if i am missing any

Thanks
Sign in to comment

ingesting data to azure data exlorer through azure data factory

0 additional answers