SnowFlake Data Copy Error

Question

I have a copy data activity which is using a SnowFlake source. In the source under "Additional Snowflake copy options" I have added a parameter with the property name set to "SINGLE" and the value set to "FALSE". I am able to publish the pipeline without any issues.

When I run it, I get the error below. To double check I logged into my SnowFlake account and I can see the command ADF is sending does not have SINGLE = FALSE, it has SINGLE = TRUE.

Operation on target Copy data1 failed: ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [22000] Max file size (268435456) exceeded for unload single file mode. We recommend that you increase the max file size parameter, or disable single-file mode in the unload command and combine the unloaded files into a single file after you download them.,Source=Microsoft.DataTransfer.Runtime.GenericOdbcConnectors,''Type=System.Data.Odbc.OdbcException,Message=ERROR [22000] Max file size (268435456) exceeded for unload single file mode. We recommend that you increase the max file size parameter, or disable single-file mode in the unload command and combine the unloaded files into a single file after you download them.,Source=Snowflake,'

Answer 1

Hi @Albert, Christopher ,

Thanks for your query and apologizes for the delay in response.

According to this phrase - 'In the sink under "Additional Snowflake copy options" I have added a parameter with the property name set to "SINGLE" and the value set to "FALSE"' my understanding is that your sink data store is also a Snowflake, please correct me if I am incorrect.

If your sink is Snowflake, as per this doc (copy options of snowflake) , the default value for parameter SINGLE is FALSE. Could you please try removing that parameter from your sink settings (Additional Snowflake copy options) and run your pipeline and let us know what is the value you see in your Snowflake account ( FALSE , TRUE)

Also could you please try setting MAX_FILE_SIZE parameter value < 268435456 bytes under source Additional Snowflake copy options settings? (See MAX_FILE_SIZE limitations in this document : MAX_FILE_SIZE)

The strange part is, though you have set SINGLE parameter value to FALSE, you are noticing it as TRUE in your Snowflake account. In-order to further investigate on this issue, would you please share below details so that I can try to validate and escalate to internal team as needed.

• Pipeline runID
• Activity runID
• Data factory regions.
• JSON code of your pipeline.

Looking forward to your response.

[Note: Due to 1000 character limit in comment section of this Q&A platform, I am posting this in answer section]

----------

Thank you