Unable to load full data set from on premises (oracle server ) to Synapse dedicated sql pool table using polybase

Question

Hello Team ,

While trying to load the table from oracle on prem table to Synapse sql table using polybase operation it is giving below error :

ErrorCode=PolybaseOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error happened when loading data into SQL Data Warehouse. Operation: 'Polybase operation'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,Source=.Net SqlClient Data Provider,SqlErrorNumber=107090,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=107090,State=1,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,},],'

I tried to load random 2-3 rows only then it loaded successfully but while loading full data set at a time it gives above error. .

Could you please help me out .

Thanks ,
Shreeya Jain

Answer 1

Hello @Jain, Shreeya ,

Thanks for the question and using MS Q&A platform.

If my understanding is correct here, you are receiving above error while trying to copy data from Oracle (on premise) to Synapse dedicated pool using Polybase. And this happens only when you load the complete table from source, but when you test with sampled data (limited rows) the copy pipeline run successfully without issues. Let us know of that is not correct.

When you copy a sampled data, the pipeline runs successful but fails with large dataset, this makes me think that the issue could be related to your source data. It could be corrupt data or a mismatch between datatypes. To make sure if the issue is related to data, I recommend enabling logging in copy activity which would route the incorrect data or the records/rows with incorrect data/dataTypes to your logging folder. From there you should be able to get more insights on what the actual root case is and how to overcome it.

If the issue is related to empty string('') or blank value columns, then try to set treatEmptyAsNull to true under polybase settings of your sink and make sure your destination corresponding column accepts NULLs.

In addition, you can also refer to doc if your issue is related to loading empty values to Decimal type column in synapse table: PolyBase troubleshooting

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators