Azure Synapse Copy Error

Question

Azure Synapse Copy Error

Anonymous

We have Synapse ETL pipelines that just began failing this morning on a copy step in all of our environments and for all of our pipelines. The step worked previously and just began failing today. The copy step that is failing is just copying a parquet file from one location in a Gen2 Data Lake to another location (Landing folder versus Archive folder). This Copy step is failing whenever there is a decimal column in the parquet file.

Failure looks like this:

ErrorCode=ParquetJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.UnsupportedOperationException:Unsupported PrimitiveTypeName 8 total entry:2 com.microsoft.datatransfer.bridge.parquet.ParquetWriterBuilderBridge.getPrimitiveTypeName(ParquetWriterBuilderBridge.java:235) com.microsoft.datatransfer.bridge.parquet.ParquetWriterBuilderBridge.addDecimalColumn(ParquetWriterBuilderBridge.java:96) .,Source=Microsoft.DataTransfer.Richfile.ParquetTransferPlugin,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'

QuantumCache 20,681 Reputation points Moderator

2023-08-16T20:52:45.66+00:00

Hello @Kevin Nowell Welcome to QnA forum,

The error message suggests that the error is related to a decimal column in the Parquet file.

Could you please check that the decimal column is correctly formatted and does not contain any invalid data?

Did you try any other method, such as ADF pipeline instead of Synapse pipeline?

Can we see any dummy source data to repro on our sde?
Anonymous

2023-08-16T21:03:27.16+00:00

Thank you for your reply Satish.

The error is not limited to any one column of any one table. We have multiple Synapse pipelines pulling from multiple sources and they all are experiencing the same error all of a sudden this morning when they were working before.

I can read the data in using the copy statement just fine. In fact an earlier step is reading the new data in from the data lake landing folder and saving to a SQL table and that part is working perfectly fine. It is only the step that is using the datalake as a source and as the sink, just copying from one location to another, that we are experiencing the error.

I have not tried using an ADF pipeline instead because I would even if that work I would like to fix this pipeline. I have however downloaded the parquet files and read them in and there does not seem to be any error in the data that I can see or any changes in the column names or anything like that.
QuantumCache 20,681 Reputation points Moderator

2023-08-16T21:31:37.4966667+00:00

Hello @Kevin Nowell, Thanks for the quick response.

Looks like we need to investigate all the pipeline logs.

Could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.

Your Email should have below information.

Thread URL: Link to this thread.

Your Email Subject: Attn Satish Boddu

Subscription ID: your Azure Subscription ID in which the issue is occurring.

Please let me know once you have done the same.
Anonymous

2023-08-16T23:35:05.87+00:00

Thank you Satish. I have sent you the email.
QuantumCache 20,681 Reputation points Moderator

2023-08-17T00:29:07.0533333+00:00

Got it , Thanks for sending the email.
Abhyankar, Aayush (External) 20 Reputation points

2023-08-17T10:22:33.3433333+00:00

Same issue for me. I am running multiple copy pipelines for transfer of parquet files from azure blob storage to SFTP. It was running well until 16th Aug,2023. Suddenly all the started to fail with same above error
QuantumCache 20,681 Reputation points Moderator

2023-08-17T15:35:28.5166667+00:00

Hello @Abhyankar, Aayush (External) Thanks for reaching out on this forum!

Do you have a support plan to reach out to the support team for immediate help? Let me know if you need any help with that?

We are working on this issue offline and we will keep you posted on this thread.
Anonymous

2023-08-17T16:25:39.0966667+00:00

Hello, we are also having this exact same issue with our load at 3AM this morning. Nothing has changed and the pipelines have been running for months with no issues. As of early this morning we can no longer run the load due to this exact issue (parquet file being copied from one location in gen2 datalake to another location in same datalake). Thank you.
QuantumCache 20,681 Reputation points Moderator

2023-08-17T16:30:08.6633333+00:00

Hello @HelenMoniakis-1088 Sorry for the inconvenience caused in this matter.

We have escalated this internally.
Dinesh Rangasamy 5 Reputation points

2023-08-17T16:55:25.57+00:00

Hi Satish, Same issue has been faced in pipelines copying Parquet between ADLS gen2 paths which was working fine before. But also for only for copying the Parquets with Decimal columns.
Anonymous

2023-08-17T17:22:42.9833333+00:00

Hello, I have confirmed that this is happening only to the parquet files that include decimal columns. All other parquet files are being copied successfully from 1 location to another in the same datalake.
Akshaya Mandrekar 0 Reputation points

2023-08-18T03:42:04.48+00:00

We are also having the same issue since yesterday
Torkkeli Julius 5 Reputation points

2023-08-18T11:20:51.0933333+00:00

Hello, do we have any update on this issue? Our solution is suffering from the same error code. In our case the storage is Gen1.
Steven Dodd 15 Reputation points

2023-08-18T11:51:39.67+00:00

We are affected by this issue also in a number of our key pipelines with the same symptoms as reported in other comments (copy activity, parquet, decimal). Nothing has been changed on our side, it started occurring yesterday. Any updates or ETAs on resolution would be great to know!
QuantumCache 20,681 Reputation points Moderator

2023-08-18T18:42:13.1933333+00:00

Hello All,

Steven Dodd ,Torkkeli Julius,Akshaya Mandrekar,HelenMoniakis-1088,Dinesh Rangasamy,Abhyankar, Aayush (External)

We really appreciate your patience in this matter.

Sorry for the inconvenience caused.

We are working with Product team on addressing this issue as soon as possible. Please stay tuned!
Abhyankar, Aayush (External) 20 Reputation points

2023-08-18T19:19:10.2966667+00:00
Hello All,

I had raised a support ticket with Microsoft and found the following solution from MS.

Please go through this email extract from MS - Hope this helps.

"""

Got an update from the backend team, please find below 3 workarounds to mitigate the issue.

They are each standalone workaround. Let's go with the 1st one.

specify "EnableParquetDecimalKeepOriginalType": false under the format settings of sink

specify ""useParquetV2": true" under the format settings of both source and sink

when source and sink's parquet datasets have the same compression codecs, choose a copy behavior other than "merge files" under sink settings

Our team is investigating further on the RCA part and I will keep you updated.

"""

For me 3rd point worked, where I changed the copy behavior from "Merge Files" to "Preserve Hierarchy"

Hope this helps you in anyway possible.

Happy to help.
Akshaya Mandrekar 0 Reputation points

2023-08-20T22:46:43.5666667+00:00

Thanks Abhyankar, Aayush (External). I had a look at the workarounds given to you, but looks I don't see the options from point 1 and 2 in our pipeline. Regarding third workaround, unfortunately we need the merge. SatishBoddu-MSFT, any updates from your internal investigation? Should I be raising a support ticket separately?
QuantumCache 20,681 Reputation points Moderator

2023-08-20T23:19:04.4966667+00:00

Hello All,

Steven Dodd ,Torkkeli Julius,Akshaya Mandrekar,HelenMoniakis-1088,Dinesh Rangasamy,Abhyankar, Aayush (External)

The Product team is notified of this issue, and they are actively working on releasing the hot fix asap.

We will keep you updated on this issue, Monday PST morning hours.

Thanks for your understanding in this matter.
Anonymous

2023-08-21T13:17:49.2766667+00:00

Hello all, similar to Akshaya Mandrekar, we need the merge and can't implement option 1 and 2.

What worked for us is to cast the decimals as varchars coming in from on-prem source to datalake...and in the final view we re-cast as decimal so that they are the right datatype for power bi consumption. This would be a hassle if you had many columns to cast.
Steven Dodd 15 Reputation points

2023-08-21T14:18:40.55+00:00

Thanks Satish, Just as an update, we've successfully used a workaround from Aayush's post (thank you for sharing!) - we are using ADF (not Synapse) and adding "EnableParquetDecimalKeepOriginalType": false under the format settings of sink worked for us. Note that it doesn't appear to be an option in the UI, so we had to add it manually to the JSON for the copy activity.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
TOMA 0 Reputation points

2023-08-22T09:15:23.27+00:00

Any updates on this issue? Because today we experienced the same problem.
Leonardo Benitez 0 Reputation points

2023-08-25T07:54:30.61+00:00

Same problem here.
Workarounds 1 and 2 are highly problematic for us, since we don't know if they would introduce any other change in the data types.
Is there documentation for those two settings?

Answer accepted by question author

QuantumCache 20,681 Moderator

Hello all,

Update from Microsoft Product team:

Thanks to Kevin Nowell who reported this error on this forum, which really helped in the escalation process.

Summary of Impact: Between 05:10 UTC on 11 Aug 2023 and 11:02 UTC on 24 Aug 2023, Azure Data Factory have experienced error with copy activity with both source and sink's file format being Parquet, and copy behavior configured as "MergeFiles".**
Hotfix to be completed by the 24 Aug 2023. Some users may be seeing signs of recovery as our mitigation efforts progress.**

Stay informed about Azure service issues by creating custom service health alerts: https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation.

Hello @Kevin Nowell Please click "Accept Answer" and we will be good to close this case. Thanks for your understanding.

0 comments

1 additional answer

Your answer

QuantumCache 20,681 Reputation points Moderator

2023-08-16T20:52:45.66+00:00

Hello @Kevin Nowell Welcome to QnA forum,

The error message suggests that the error is related to a decimal column in the Parquet file.

Could you please check that the decimal column is correctly formatted and does not contain any invalid data?

Did you try any other method, such as ADF pipeline instead of Synapse pipeline?

Can we see any dummy source data to repro on our sde?
Anonymous

2023-08-16T21:03:27.16+00:00

Thank you for your reply Satish.

The error is not limited to any one column of any one table. We have multiple Synapse pipelines pulling from multiple sources and they all are experiencing the same error all of a sudden this morning when they were working before.

I can read the data in using the copy statement just fine. In fact an earlier step is reading the new data in from the data lake landing folder and saving to a SQL table and that part is working perfectly fine. It is only the step that is using the datalake as a source and as the sink, just copying from one location to another, that we are experiencing the error.

I have not tried using an ADF pipeline instead because I would even if that work I would like to fix this pipeline. I have however downloaded the parquet files and read them in and there does not seem to be any error in the data that I can see or any changes in the column names or anything like that.
QuantumCache 20,681 Reputation points Moderator

2023-08-16T21:31:37.4966667+00:00

Hello @Kevin Nowell, Thanks for the quick response.

Looks like we need to investigate all the pipeline logs.

Could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.

Your Email should have below information.

Thread URL: Link to this thread.

Your Email Subject: Attn Satish Boddu

Subscription ID: your Azure Subscription ID in which the issue is occurring.

Please let me know once you have done the same.
Anonymous

2023-08-16T23:35:05.87+00:00

Thank you Satish. I have sent you the email.
QuantumCache 20,681 Reputation points Moderator

2023-08-17T00:29:07.0533333+00:00

Got it , Thanks for sending the email.
Abhyankar, Aayush (External) 20 Reputation points

2023-08-17T10:22:33.3433333+00:00

Same issue for me. I am running multiple copy pipelines for transfer of parquet files from azure blob storage to SFTP. It was running well until 16th Aug,2023. Suddenly all the started to fail with same above error
QuantumCache 20,681 Reputation points Moderator

2023-08-17T15:35:28.5166667+00:00

Hello @Abhyankar, Aayush (External) Thanks for reaching out on this forum!

Do you have a support plan to reach out to the support team for immediate help? Let me know if you need any help with that?

We are working on this issue offline and we will keep you posted on this thread.
Anonymous

2023-08-17T16:25:39.0966667+00:00

Hello, we are also having this exact same issue with our load at 3AM this morning. Nothing has changed and the pipelines have been running for months with no issues. As of early this morning we can no longer run the load due to this exact issue (parquet file being copied from one location in gen2 datalake to another location in same datalake). Thank you.
QuantumCache 20,681 Reputation points Moderator

2023-08-17T16:30:08.6633333+00:00

Hello @HelenMoniakis-1088 Sorry for the inconvenience caused in this matter.

We have escalated this internally.
Dinesh Rangasamy 5 Reputation points

2023-08-17T16:55:25.57+00:00

Hi Satish, Same issue has been faced in pipelines copying Parquet between ADLS gen2 paths which was working fine before. But also for only for copying the Parquets with Decimal columns.
Anonymous

2023-08-17T17:22:42.9833333+00:00

Hello, I have confirmed that this is happening only to the parquet files that include decimal columns. All other parquet files are being copied successfully from 1 location to another in the same datalake.
Akshaya Mandrekar 0 Reputation points

2023-08-18T03:42:04.48+00:00

We are also having the same issue since yesterday
Torkkeli Julius 5 Reputation points

2023-08-18T11:20:51.0933333+00:00

Hello, do we have any update on this issue? Our solution is suffering from the same error code. In our case the storage is Gen1.
Steven Dodd 15 Reputation points

2023-08-18T11:51:39.67+00:00

We are affected by this issue also in a number of our key pipelines with the same symptoms as reported in other comments (copy activity, parquet, decimal). Nothing has been changed on our side, it started occurring yesterday. Any updates or ETAs on resolution would be great to know!
QuantumCache 20,681 Reputation points Moderator

2023-08-18T18:42:13.1933333+00:00

Hello All,

Steven Dodd ,Torkkeli Julius,Akshaya Mandrekar,HelenMoniakis-1088,Dinesh Rangasamy,Abhyankar, Aayush (External)

We really appreciate your patience in this matter.

Sorry for the inconvenience caused.

We are working with Product team on addressing this issue as soon as possible. Please stay tuned!
Abhyankar, Aayush (External) 20 Reputation points

2023-08-18T19:19:10.2966667+00:00

Hello All,

I had raised a support ticket with Microsoft and found the following solution from MS.

Please go through this email extract from MS - Hope this helps.

"""

Got an update from the backend team, please find below 3 workarounds to mitigate the issue.

They are each standalone workaround. Let's go with the 1st one.

specify "EnableParquetDecimalKeepOriginalType": false under the format settings of sink

specify ""useParquetV2": true" under the format settings of both source and sink

when source and sink's parquet datasets have the same compression codecs, choose a copy behavior other than "merge files" under sink settings

Our team is investigating further on the RCA part and I will keep you updated.

"""

For me 3rd point worked, where I changed the copy behavior from "Merge Files" to "Preserve Hierarchy"

Hope this helps you in anyway possible.

Happy to help.
Akshaya Mandrekar 0 Reputation points

2023-08-20T22:46:43.5666667+00:00

Thanks Abhyankar, Aayush (External). I had a look at the workarounds given to you, but looks I don't see the options from point 1 and 2 in our pipeline. Regarding third workaround, unfortunately we need the merge. SatishBoddu-MSFT, any updates from your internal investigation? Should I be raising a support ticket separately?
QuantumCache 20,681 Reputation points Moderator

2023-08-20T23:19:04.4966667+00:00

Hello All,

Steven Dodd ,Torkkeli Julius,Akshaya Mandrekar,HelenMoniakis-1088,Dinesh Rangasamy,Abhyankar, Aayush (External)

The Product team is notified of this issue, and they are actively working on releasing the hot fix asap.

We will keep you updated on this issue, Monday PST morning hours.

Thanks for your understanding in this matter.
Anonymous

2023-08-21T13:17:49.2766667+00:00

Hello all, similar to Akshaya Mandrekar, we need the merge and can't implement option 1 and 2.

What worked for us is to cast the decimals as varchars coming in from on-prem source to datalake...and in the final view we re-cast as decimal so that they are the right datatype for power bi consumption. This would be a hassle if you had many columns to cast.
Steven Dodd 15 Reputation points

2023-08-21T14:18:40.55+00:00

Thanks Satish, Just as an update, we've successfully used a workaround from Aayush's post (thank you for sharing!) - we are using ADF (not Synapse) and adding "EnableParquetDecimalKeepOriginalType": false under the format settings of sink worked for us. Note that it doesn't appear to be an option in the UI, so we had to add it manually to the JSON for the copy activity.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
TOMA 0 Reputation points

2023-08-22T09:15:23.27+00:00

Any updates on this issue? Because today we experienced the same problem.
Leonardo Benitez 0 Reputation points

2023-08-25T07:54:30.61+00:00

Same problem here.
Workarounds 1 and 2 are highly problematic for us, since we don't know if they would introduce any other change in the data types.
Is there documentation for those two settings?

Answer 1

Anonymous

Microsoft issued an update that has fixed our issue.

Anonymous

2023-08-26T21:38:43.2033333+00:00

I am not sure if the bug has been fixed. Tried today, the Sink Copy behavior using Merge files and got same error

Share via

Azure Synapse Copy Error

1 additional answer

Your answer