Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,601 questions
2 answers
Schema Drift Issue in Dataflow with Sink-Delta
Hi Team, I was testing the schema drift option for my delta sink and am getting error while running my dataflow when added one new column to existing source schema. Pipeline ran successfully when I didn't change the schema. Error: I have selected Allow…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
1 answer
Renew Certification - Microsoft Azure Fundamentals
Hi Team, My Microsoft Azure Fundamentals Certification expires in June 2024. I Would like to renew the certification; can you please provide me with the steps. I don't see option to take Renewal assessment. I tried to go through this video, however it…
asked 2024-05-04T13:01:18.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 74%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETA%3C/text%3E%3C/svg%3E)
Tulika Anand
0
Reputation points
edited an answer 2024-05-04T13:14:56.4833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 76%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
VarunTha
3,290
Reputation points Microsoft Vendor
2 answers
How to update and delete a row using Azure Data Factory Change Data Capture
I am exploring Azure Data Factory CDC feature, and I am trying to perform the CDC from one SQL table to another SQL table, also the SQL table in the source has a primary key. Whenever a new row is added to the source table, it gets added to the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
1 answer
An csv file with 200 columns in adls. fetch only list of columns from config table. config table is in Azure SQL. config table has only 10 col names
An csv file with 200 columns in adls. fetch only the list of columns from the config table. config table is in Azure SQL.config table has only 10 column names. how to create pipeline with this scenario in Azure datafactory?
asked 2024-05-03T07:59:15.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 38%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E1P%3C/text%3E%3C/svg%3E)
19B01A04F3 P
0
Reputation points
answered 2024-05-03T19:27:13.7+00:00
AnnuKumari-MSFT
31,061
Reputation points Microsoft Employee
0 answers
CDC Start button disabled. Cannot Start CDC in Azure data factory
I created a CDC in adf and made a Git configuration to connect to Azure Devops Repos with a new branch created. Now, I checked out to a new branch. In the new branch, CDC is also there but i cannot start the CDC . It says Please publish to enable the…
asked 2024-05-03T13:15:19.5066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 71%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Sudip Bala
0
Reputation points
edited a comment 2024-05-03T19:16:21.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
26,306
Reputation points Microsoft Employee
0 answers
InsertDate and UpdateDate column addition in Sink-Delta from dataflow.
I am using dataflow to Upsert the data to my delta lake. My requirement is to have two columns - 1. insertdate 2.updatedate . If there is a new record then both insertdate and updatedate column should get updated with current_date() values. And, if…
0 answers
Split single column data into multiple columns in datafactory
Issue: I have some csv files in sftp location . I was planning to use copyactivity to load files from sftp location to datalake for archiving as well as upserting in deltalake. However, I am getting error while reading from source, as some file has…
1 answer
One of the answers was accepted by the question author.
How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory ?
How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory 2.from one table in log analytics workspace 1 to another table in log analytics 2 workspace using Azure Data Factory ?
asked 2024-03-14T00:37:19.5866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rishabh Srivastava
40
Reputation points Microsoft Employee
accepted 2024-05-03T16:08:33.41+00:00
Rishabh Srivastava
40
Reputation points Microsoft Employee
1 answer
How to copy the data from Amazon S3 to Azure SQL database incrementally.
Hello team, I wanted to wanted to copy data from Amazon S3(JSON format) to Azure SQL database. I wanted to learn the mechanism for incrementally updating the data from Amazon S3 to Azure SQL. Also, the environment has only SHIR deployed, so data flows…
asked 2024-05-01T20:06:01.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chavan
40
Reputation points
edited a comment 2024-05-03T16:06:06.96+00:00
Oury Ba-MSFT
16,471
Reputation points Microsoft Employee
0 answers
How to fix this error in azure data factory? My dataflow worked fine, now I receive this error.
Error: Spark job failed: { "text/plain":…
asked 2024-05-03T14:07:52.7066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 28.000000000000004%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFG%3C/text%3E%3C/svg%3E)
Fatemeh Ghorbani Sini
0
Reputation points
commented 2024-05-03T15:48:21.6766667+00:00
Fatemeh Ghorbani Sini
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to keep ARM Template in sync when modifying Azure Data Factory?
I'm unclear on the process for maintaining the ARM Template for deployments when modifying an Azure Data Factory (ADF). I see two options: Modify the ADF in the browser-based drag-n-drop editor and use the generated Json ARM Template as is. Manually…
asked 2024-05-02T13:38:26.7833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 93%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Moyer, Todd
0
Reputation points
commented 2024-05-03T13:36:10.6766667+00:00
Moyer, Todd
0
Reputation points
2 answers
Can you assist with a Synapse issue involving memory allocation in SQL DW Copy Command in Data Factory, without solely resorting to increasing DWU?
I'm encountering an issue while using the Copy Activity in Data Factory with the Polybase copy option. The error message I receive is as follows: “…..SQL DW Copy Command operation failed with error 'Unable to allocate 257707 KB for columnstore…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 40%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
0 answers
COPY ACTIVITY FROM SOURCE LIKED REST TO DESTINATION LOINED REST SERVICE
Want to copy from source table to destination table in the same log analytics workspace. Using Azure Data Factory. My configuration: Source Linked REST Service to read data from Log Analytics workspace w1 table t1 Destination Linked REST service to…
asked 2024-05-02T23:31:22+00:00
Rishabh Srivastava
40
Reputation points Microsoft Employee
commented 2024-05-03T12:02:27.48+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
9,760
Reputation points Microsoft Vendor
0 answers
Job failed due to reason: at Sink 'xxx': Connection string is invalid. Unable to parse.
Hi, In a pipeline there are 5 dataflows. When we debug each dataflow separately it works perfectly fine. But the moment we debug the pipeline, we get the error "Job failed due to reason: at Sink 'xxx': Connection string is invalid. Unable to…
asked 2024-05-03T10:54:25.3966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 70%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pratisti Satardekar
90
Reputation points
edited a comment 2024-05-03T11:43:19.97+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
5,840
Reputation points Microsoft Vendor
1 answer
ADF + SAP Tableconnector: Function Modules
Hi together, We have a setup to copy table data from our SAP system to our AzureDataFactory via the SAP table connector interface. This works well as long as we access SAP tables. Now there is the need to access function mudules in SAP with parameters to…
asked 2024-04-23T07:33:41.3666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 40%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM8%3C/text%3E%3C/svg%3E)
MarcusU-8790
0
Reputation points
answered 2024-05-03T11:29:34.7233333+00:00
MarcusU-8790
0
Reputation points
0 answers
How to Automatically create a Data Cycle diagram
Currently, the entire data cycle of a project has been developed in Azure Data Factory. The main objective is to have an automated visual scheme of the entire data cycle at a low level (from the basic activities, not the sequential execution of…
asked 2024-05-02T10:45:02.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 20%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAJ%3C/text%3E%3C/svg%3E)
ADM. Javier Gaig Mellado
0
Reputation points
commented 2024-05-03T11:29:06.8433333+00:00
ADM. Javier Gaig Mellado
0
Reputation points
4 answers
Why is the public ip range download blocked?
Hey, I use a script to download the public ip ranges from azure (https://www.microsoft.com/en-us/download/details.aspx?id=56519). If I view that URL in my browser, it displays and I am able to download the JSON. However, if I run my script that would…
asked 2024-05-01T11:40:10.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 81%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
karolyi
0
Reputation points
commented 2024-05-03T11:18:59.8766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 93%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBG%3C/text%3E%3C/svg%3E)
Ben Gimblett
3,410
Reputation points Microsoft Employee
1 answer
ADF Copy Activity : Data load from ADLS (parquet files) to Azure Synapse Table Failing after running for 24 hours
ADF pipeline copy failed after running for 24 hours. Timeout is set to 7days. Copy Activity is trying to copy data from ADLS (Parquet files) to Azure Synapse Table on Production Environment . Could you please suggest what could be the possible issue for…
asked 2024-05-02T08:48:59.7533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 24%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kapoor, Sipra (S)
5
Reputation points
commented 2024-05-03T10:13:53.3866667+00:00
Smaran Thoomu
9,760
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
ADF GLobal Parameters error after trigger (debug works fine)
Hi, I have set a Global Parameter "maximale_ouderdom" which I use in an If Condition. When I debug the pipeline all works fine. The global parameter is being used in the condition. However, when the actual…
asked 2020-09-17T08:15:32.647+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 52%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
markharing@bisqq.nl
61
Reputation points
commented 2024-05-03T09:52:57.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 46%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E0%3C/text%3E%3C/svg%3E)
09251703
0
Reputation points
1 answer
How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute
Databricks has recently announced serverless compute for workflows: https://learn.microsoft.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…
asked 2024-05-01T12:12:06.9033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
edited a comment 2024-05-03T09:47:03.68+00:00
phemanth
5,840
Reputation points Microsoft Vendor