How to copy files from SSMS using SHIR to DataLAke Gen 2 in parquet format?

nidhi staning 0 Reputation points
2023-11-23T07:28:55.55+00:00

I am trying to copy csv files from on-prem using SSMS. I created linked services and set up the SHIR to get the data from on-prem. I created the source as SQL SERVER and then I created sink as Data Lake storage Gen 2 and format as Parquet.

When I run the pipeline, I am getting this error. But when I make the sink format as csv, the pipeline runs. Please anybody can explain and help me with this? Thanks

Error details

Copy address table

Error codeUnknown

Failure typeUser configuration issue

DetailsErrorCode=Unknown,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An unknown error occurred.,Source=Microsoft.DataTransfer.Common,''Type=JNI.JavaExceptionCheckException,Message=Exception of type 'JNI.JavaExceptionCheckException' was thrown.,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'

Activity ID3a46d433-a6b4-4d02-a17b-14bdfb79e614User's image

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,425 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,161 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Smaran Thoomu 12,615 Reputation points Microsoft Vendor
    2023-11-23T12:12:14.4266667+00:00

    Hi @nidhi staning

    Welcome to Microsoft Q&A and thanks for reaching out.

    The error message you received indicates that there was an issue with the pipeline when trying to copy data from SQL Server to Data Lake Storage Gen2 in Parquet format. However, the pipeline runs successfully when the sink format is set to CSV.

    One possible reason for this issue could be that the Parquet format is not supported by the SHIR (Self-hosted Integration Runtime) version you are using. Please check the version of SHIR you are using and ensure that it supports Parquet format.

    Another possible reason could be that the Parquet format is not compatible with the data you are trying to copy. Please ensure that the data you are trying to copy is compatible with the Parquet format.

    To troubleshoot this issue, you can check the following:

    1. Check the SHIR version you are using and ensure that it supports Parquet format.
    2. Check the compatibility of the data you are trying to copy with the Parquet format.
    3. Check the pipeline settings and ensure that they are configured correctly.

    You can also refer: How to copy data to and from Azure Data Lake Storage Gen2 using Azure Data Factory

    I hope this helps! Let me know if you have any further questions.