question

MatamSavitha-7051 avatar image
0 Votes"
MatamSavitha-7051 asked MatamSavitha-7051 commented

ingesting data to azure data exlorer through azure data factory

I have to ingest 380 million data from azure sql to azure data explorer through azure data factory,
can anyone tell me what settings do we make in ADF(DIU's) such that all the data would be ingested at one go consistently. Network Bandwith is 120 Mbps
Thank you

azure-data-factoryazure-sql-databaseazure-data-explorer
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered MatamSavitha-7051 commented

Hi @MatamSavitha-7051 ,

Thank you for posting query in Microsoft Q&A Platform.

Azure data factory and Azure Data explorer documentations, have complete details of integrating ADF and Data explorer. Kindly check below shared links for same.

Integrate Azure Data Explorer with Azure Data Factory
Copy data to or from Azure Data Explorer using Azure Data Factory or Synapse Analytics

Hope this helps. Please let us know if any further queries.


Please consider hitting Accept Answer button. Accepted answers help community as well.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ShaikMaheer-MSFT ,

Thanks for the quick reply.

I increased DIU to 32 but still the copy activity is taking the default setting (DIU, 4) when tried to ingest 80million data to ADX, it took 7hrs, 40 mins time to ingest.

Also,

I have used the BulkcopyDBfromSql to ADX template , but i am getting the below error when tried to Debug

" Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Incorrect syntax near the keyword 'SELECT'.
Incorrect syntax near the keyword 'as'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Incorrect syntax near the keyword 'SELECT' "

Please help me with the best option , if i am missing any

Thanks

0 Votes 0 ·