Feature request - Datafactory copy activity default copy option to be polybase at the backend.

Shaik, Khaja 1 Reputation point
2021-09-24T06:41:48.25+00:00

Scenario:

  1. In azure synapse analytics, built in copy activity uses polybase as default.
  2. Pipeline in synapse with source as postgre and sink as synapse dedicated pool works fine with polybase as copy method.
  3. However datafactory throws following exception, with same set up of polybase.
  4. error:
    Operation on target Copy data1 failed: ErrorCode=PolybaseOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error happened when loading data into SQL Data Warehouse. Operation: 'Polybase operation'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,Source=.Net SqlClient Data Provider,SqlErrorNumber=107090,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=107090,State=1,Message=HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Not enough columns in this line.,},],'

This denotes that, there is something in ADF polybase setting, which is uncommon to Synapse polybase setting.

Feature Request:

  1. Can ADF engine use same polybase as in synapse.
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,672 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,136 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2021-09-25T00:06:17.98+00:00

    Hello @Shaik, Khaja ,
    Thanks for the feedback and using Microsoft Q&A platform .We will pass on this feedback to the ADF team and share if we get any updates from them .
    You can also pass the feedback from the ADF UI
    135158-image.png

    Thanks
    Himanshu

    -------------------------------------------------------------------------------------------------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments