question

VP-0129 avatar image
0 Votes"
VP-0129 asked lekyagoriparti-3699 published

ADF Copy from parquet file type to Synapse Failing

Hello Team,
I am getting an error when copying parquet type file to Azure synapse using polybase option. I am getting the below error.

Code": 21001, "Message": "ErrorCode=ParquetInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=File is not a valid parquet file.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,'", "EventType": 0,

When i tried for the same parquet file using BulkInsert option, it worked.

Any Ideas?
Thank You

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
1 Vote"
HimanshuSinha-MSFT answered VP-0129 commented

Hello @VP-0129 ,

Thanks for question and also for using this new forum .
I am not sure if you have gone through this link , there are some limitation of copy activity when handling paraquet file .

https://docs.microsoft.com/en-us/azure/data-factory/format-parquet#data-type-support . May be you have this issue .

"Parquet complex data types (e.g. MAP, LIST, STRUCT) are currently supported only in Data Flows, not in Copy Activity"


Thanks Himanshu

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks Himanshu.

I don’t have any of those complex data types but source is parquet partition datasets and I am trying to load all partitions. As an example
/containername/foldername/tablename/date=2020-01-01/files.parquet
/containername/foldername/tablename/date=2020-02-01/files.parquet
/containername/foldername/tablename/date=2020-03-01/files.parquet

Do you have any documentation that shows how to map the above datasets in copy tool to copy all partitions.

Thanks
VP

0 Votes 0 ·
HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered VP-0129 commented

Hello @VP-0129 ,
Thanks for question and also for using this new forum .

I did tried to do the same using the wildcard option and it worked for me , please do try this out .
I did had 3 paraquet files and by doing this I was able to insert into the SQL table .

58371-2021-01-19-15-14-54-calculator.png

58361-2021-01-19-15-19-01-analytics-moveandtransform-ver.png

58342-2021-01-19-15-29-44-sqlquery1sql-analyticssharedda.png


Thanks Himanshu

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @VP-0129 ,
We have not heard back from you on this and was just following up if you have any further queries .
Thanks
Himanshu

0 Votes 0 ·

Hello @VP-0129 ,
We have not heard back from you on this yet and was just following up if you have any further queries .
Thanks
Himanshu

0 Votes 0 ·
VP-0129 avatar image VP-0129 HimanshuSinha-MSFT ·

Thanks Himanshu. The answer helped me, no further queries on this.

Thanks
VP

0 Votes 0 ·
KUMARNaveen-1775 avatar image
0 Votes"
KUMARNaveen-1775 answered KUMARNaveen-1775 edited

@HimanshuSinha-MSFT
Hi , But with polybase selected option, if we write ".parquet" , then we got error that polybase only support . or wildcard path.
How we can resolve this issue ? I am still getting same issues and not using any complex types . All columns are string types and writing the parquet file through databricks.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

lekyagoriparti-3699 avatar image
0 Votes"
lekyagoriparti-3699 answered lekyagoriparti-3699 published

Hi @KUMARNaveen-1775

I am facing same issue. Could you pls let me know how did you solve it.

Thanks in advance

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.