COPY INTO erroring out because of Polybase size limit

Lokesh 211 Reputation points
2023-02-07T16:38:13.9466667+00:00

HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: The size of the schema/row at ordinal 44555 is 1408314 bytes. It exceeds the maximum allowed row size of 1000000 bytes for Polybase.

Getting about error while executing the below command.

any help would be highly appreciated!



COPY INTO table
FROM ''
WITH (
    FILE_TYPE = 'PARQUET'--,
	--MAXERRORS  = 100
	
)
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,342 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Lokesh 211 Reputation points
    2023-02-08T10:34:17.5166667+00:00

    BCP works, used the bulk insert from azure data factory
    There is no SQL script to perform the bulk operation in dedicated pool.

    0 comments
  2. BhargavaGunnam-MSFT 25,876 Reputation points Microsoft Employee
    2023-02-08T20:06:48.2566667+00:00

    Hello @Loki,

    Welcome to the MS Q&A platform.

    The max length of the intermittent external table in polybase(behind the scenes, PolyBase creates an External Table) is nvarchar(4000).

    So, if the max length of the source column is greater than nvarchar(4000), polybase option can’t be used.

    You can use the bulk insert option or copy into option in copy activity.

    Please see the Row size and data type limits using PolyBase.

    User's image

    Reference document:

    https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-sql-data-warehouse?tabs=data-factory#use-polybase-to-load-data-into-azure-synapse-analytics

    I hope this helps. Please let me know if you have any further questions.