Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,387 questions
DF-Executor-OutOfMemoryError" in azure synapse
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 19%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Shailendra Kad
11
Reputation points
I am having a json from ravenDB which is not valid json as it is having duplicate columns.
So my first step is to clean the json and if there are duplicates make separate json for each file.
I was able to do it for sample file and it ran successfully,
Then I tried for a 12 MB file and it also worked.
But when I tried for a full DB backup file which is 10GB in size , it is giving error.
This 10 GB file generates 3 separate json as it is having DOCS columns 3 times.
First file is 9.6GB and other 2 files are small like 120MB and 10KB.
For the first file when I am trying to load it in Synapse DWH I am getitng below error.
Job failed due to reason: Cluster ran into out of memory issue during execution. Also, Please note that the dataflow has one or more custom partitioning schemes. The transformation(s) using custom partition schemes: Json,Select1,FlattenDocsCS,Flatten2,Filter1,ChangeDataTypesDateColumns,CstomsShipment. 1. Please retry using an integration runtime with bigger core count and/or memory optimized compute type. 2. Please retry using different partitioning schemes and/or number of partitions.
I tried to publish the pipeline so that I am not running in debug mode and in a small cluster.
I changed cluster size to 32 cores and changed partition schemes in optimize tab to all possible things.
But still I a getting an error.
Kindly please help
Azure Synapse Analytics
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,677 questions
{count} votes
-
Shailendra Kad 11 Reputation points
2021-11-09T18:59:24.78+00:00 Also,
When I increase the Cores of integration Runtime and run the job for above error, I get below error,
Same error even though I start the job from failed activity many times.Internal Server Error:Failed to submit job on job cluster. Integration Runtime 'IR-Synapse-test', ActivityId: '410b7569-ff98-42cf-b8ca-a2e3ed30bae5'.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT 26,156 Reputation points2021-11-09T23:20:15.38+00:00 Hello @Anonymous . Since increasing the cluster size didn't help, can we try breaking up the data into smaller chunks? Is it possible for RavenDB to write out in partitions? Or are you using RavenDB's Backup feature?
I suspect that currently it is trying to read the entire file into one piece. -
Shailendra Kad 11 Reputation points
2021-11-10T07:19:15.893+00:00 Yes, we are using RavenDB backup file.
We want to currently migrate the data from RavenDb to synapse , first full load and then incremental load.Currently I am trying to do full load.
This is the file from TEST env. and it is for the one client.
I am sure file from PROD env. will be bigger and also for other customer files would also be bigger.I do not think there is an option to generate backup partition wise or in smaller chunks.
Is there any other way , atleast it should not give out of memory error.
It is okay if it takes some time but why it is taking whole file into memory?
Or I am doing something wrong.
Sign in to comment
Sign in to answer