How to check a parquet file version in synapse?

Question

How to check a parquet file version in synapse?

Zhu, Yueli YZ [NC] 280

Hi,

I tried to copy a parquet file to csv format in gen2 by using copy activity, but failed. It shows the following error:

{ "errorCode": "2200", "message": "ErrorCode=ParquetDateTimeExceedLimit,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The Ticks value '-621357696000000000' for the datetime column must be between valid datetime ticks range -621355968000000000 and 2534022144000000000.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,'", "failureType": "UserError", "target": "Parquet to csv", "details": [] }

When I used another parquet file with ParquetV2 to copy parquet into csv file, it throws another error:

"errorCode": "2200", "message": "ErrorCode=InvalidTimestampValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The column with index 57 contains invalid TIMESTAMP value.,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,'", "failureType": "UserError", "target": "Parquet2 to csv", "details": []

So, how can I solve this issue? And also how can I check the parquet format version for the parquet file?

Thanks,

Chandra Boorla 15,460 Reputation points Microsoft External Staff Moderator

2024-11-04T01:29:34.16+00:00

Hi @Zhu, Yueli YZ [NC]

Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer accepted by question author

0 additional answers

Your answer

Chandra Boorla 15,460 Reputation points Microsoft External Staff Moderator

2024-11-04T01:29:34.16+00:00

Hi @Zhu, Yueli YZ [NC]

Following up to see if the below answer was helpful. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Chandra Boorla 15,460 Microsoft External Staff Moderator

Hi @Zhu, Yueli YZ [NC]

Greetings & Welcome to Microsoft Q&A forum! Thanks for posting your query!

The errors you are encountering with Parquet to CSV conversion in Azure Synapse Analytics often arise due to invalid or out-of-range datetime and timestamp values in the Parquet files.

User's image

For additional information, please refer: https://learn.microsoft.com/en-us/azure/data-factory/connector-troubleshoot-orc#error-code-orcdatetimeexceedlimit

Please refer to the below thread link which discussed on the similar issue and that might help you.

https://learn.microsoft.com/en-us/answers/questions/2045695/the-ticks-value-for-the-datetime-column-must-be-be

I hope this information helps. Please do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Zhu, Yueli YZ [NC] 280 Reputation points

2024-10-31T20:06:58.46+00:00

Thanks for your answers. We did try the method. But we need to choose which parquet version is needed for the copy activity. We do need to know the version of the parquet files. I find there are parquet-format 2 and parquet-mr1. So in synapse, how can we check the parquet file version?
Chandra Boorla 15,460 Reputation points Microsoft External Staff Moderator

2024-11-01T12:26:45.62+00:00
Hi @Zhu, Yueli YZ [NC]

But we need to choose which parquet version is needed for the copy activity.

Azure Synapse Analytics supports both Parquet-MR 1.0 and Parquet format 2.0. However, it's recommended to use Parquet format 2.0 for optimal performance and compatibility with modern data processing tools.

Here are some key advantages of using Parquet format 2.0:

Parquet format 2.0 offers better compression and faster read/write performance compared to Parquet-MR 1.0.

It supports more complex data structures, including nested types and arrays, which can be crucial for modern data analysis.

Parquet format 2.0 is widely supported by various data processing tools and frameworks, ensuring seamless integration with your data pipelines.

When configuring your copy activity in Azure Synapse Analytics, ensure that the source and sink data stores are compatible with the Parquet format you're using. If you are unsure about the Parquet version of your source data, you can use tools like PyArrow or Parquet tools to inspect the file metadata.

For more details, please refer: https://arrow.apache.org/docs/python/generated/pyarrow.parquet.read_metadata.html

https://medium.com/@tzhaonj/efficient-data-handling-with-pyarrow-and-parquet-in-python-7b47163497f8#:~:text=Read%20and%20Access%20Metadata%20from%20a%20Parquet%20File%3A,%23%20Read%20the%20Parquet%20file%20parquet_file%20%3D%20pq.ParquetFile%28%27data_with_metadata.parquet%27%29

I hope this information helps. Please do let us know if you have any further queries.

Thank you.
Chandra Boorla 15,460 Reputation points Microsoft External Staff Moderator

2024-11-04T12:50:51.2966667+00:00

Hi @Zhu, Yueli YZ [NC]

Glad that your query got resolved. Please continue using MS Q&A platform for further queries.

Thank you.

Share via

How to check a parquet file version in synapse?

0 additional answers

Your answer