ADF Copy from parquet file type to Synapse Failing

Question

Hello Team,
I am getting an error when copying parquet type file to Azure synapse using polybase option. I am getting the below error.

Code": 21001, "Message": "ErrorCode=ParquetInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=File is not a valid parquet file.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,'", "EventType": 0,

When i tried for the same parquet file using BulkInsert option, it worked.

Any Ideas?
Thank You

Answer 1

Hello @Parthivelu ,

Thanks for question and also for using this new forum .
I am not sure if you have gone through this link , there are some limitation of copy activity when handling paraquet file .

https://learn.microsoft.com/en-us/azure/data-factory/format-parquet#data-type-support . May be you have this issue .

"Parquet complex data types (e.g. MAP, LIST, STRUCT) are currently supported only in Data Flows, not in Copy Activity"

Thanks Himanshu

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 2

Hello @Parthivelu ,
Thanks for question and also for using this new forum .

I did tried to do the same using the wildcard option and it worked for me , please do try this out .
I did had 3 paraquet files and by doing this I was able to insert into the SQL table .

Thanks Himanshu

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 3

Hi @KUMAR Naveen

I am facing same issue. Could you pls let me know how did you solve it.

Thanks in advance

Answer 4

@HimanshuSinha-msft
Hi , But with polybase selected option, if we write "*.parquet" , then we got error that polybase only support . or * wildcard path.
How we can resolve this issue ? I am still getting same issues and not using any complex types . All columns are string types and writing the parquet file through databricks.