ADF Copy from parquet file type to Synapse Failing

Parthivelu 1 Reputation point
2021-01-14T01:03:34.377+00:00

Hello Team,
I am getting an error when copying parquet type file to Azure synapse using polybase option. I am getting the below error.

Code": 21001, "Message": "ErrorCode=ParquetInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=File is not a valid parquet file.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,'", "EventType": 0,

When i tried for the same parquet file using BulkInsert option, it worked.

Any Ideas?
Thank You

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,443 questions
0 comments No comments
{count} votes

4 answers

Sort by: Most helpful
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-01-15T19:32:00.41+00:00

    Hello @Parthivelu ,

    Thanks for question and also for using this new forum .
    I am not sure if you have gone through this link , there are some limitation of copy activity when handling paraquet file .

    https://learn.microsoft.com/en-us/azure/data-factory/format-parquet#data-type-support . May be you have this issue .

    "Parquet complex data types (e.g. MAP, LIST, STRUCT) are currently supported only in Data Flows, not in Copy Activity"

    Thanks Himanshu

    Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    2 people found this answer helpful.

  2. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-01-19T23:30:43.58+00:00

    Hello @Parthivelu ,
    Thanks for question and also for using this new forum .

    I did tried to do the same using the wildcard option and it worked for me , please do try this out .
    I did had 3 paraquet files and by doing this I was able to insert into the SQL table .

    58371-2021-01-19-15-14-54-calculator.png

    58361-2021-01-19-15-19-01-analytics-moveandtransform-ver.png

    58342-2021-01-19-15-29-44-sqlquery1sql-analyticssharedda.png

    Thanks Himanshu

    Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    1 person found this answer helpful.

  3. lekya goriparti 6 Reputation points
    2022-05-30T07:17:25.09+00:00

    Hi @KUMAR Naveen

    I am facing same issue. Could you pls let me know how did you solve it.

    Thanks in advance

    1 person found this answer helpful.
    0 comments No comments

  4. KUMAR Naveen 1 Reputation point
    2021-06-16T08:27:42.947+00:00

    @HimanshuSinha-msft
    Hi , But with polybase selected option, if we write "*.parquet" , then we got error that polybase only support . or * wildcard path.
    How we can resolve this issue ? I am still getting same issues and not using any complex types . All columns are string types and writing the parquet file through databricks.

    0 comments No comments