question

R-5499 avatar image
0 Votes"
R-5499 asked PRADEEPCHEEKATLA-MSFT commented

Error while reading a Parquet file (in Blob) in Azure Synapse via an External Table

Hi ,

I am trying to read data from a Parquet file on blob in Azure Synapse via an External Table , the file is generated from a SQL DB via ADF (Copy Activity)
The external table has datatype for all columns as varchar

While reading the data via external table , i am getting the below error :

HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: ClassCastException: class java.lang.Integer cannot be cast to class parquet.io.api.Binary (java.lang.Integer is in module java.base of loader 'bootstrap'; parquet.io.api.Binary is in unnamed module of loader 'app')

Is there any way , this issue can be resolved.

Edit :

I also tried updating the destination datatype to string in mapping in ADF (editing the JSON script) & tried to create a file (Refer below screenshot) , was still not able to read the file via an External Table with the same error


141679-image.png

Is there any specific reason for this ?

Appreciate your help.


azure-data-factoryazure-synapse-analytics
image.png (24.1 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @R-5499,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
1 Vote"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @R-5499,

Thanks for the question and using MS Q&A platform.

You may experience the following error if types are mismatched between Parquet and SQL or if you have unsupported Parquet data types: "HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: ClassCastException: ...".

To resolve this issue, make sure there is no mismatch between Parquet and SQL data types.

For more details, refer to Data loading strategies for dedicated SQL pool in Azure Synapse Analytics - Define the tables.

And also, you may refer to article ADF throws error: Unexpected error encountered filling record reader buffer ClassCastException which discuss similar issue.

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi ,

Thankyou for replying

I did try this method & it did work . Thankyou !

However I have also edited the question with a method I tried & was not able to have any success with , maybe you can help me on that too ?

Thanks!

0 Votes 0 ·

Hello @R-5499,

Could you please help me with the Data Type of the source?

To resolve this issue, make sure there is no mismatch between Parquet and SQL data types mentioned: https://docs.microsoft.com/en-us/azure/synapse-analytics/sql-data-warehouse/design-elt-data-loading#define-the-tables


142000-image.png


0 Votes 0 ·
image.png (26.6 KiB)

Hello @R-5499,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

0 Votes 0 ·