Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,007 questions
0 answers
How to import a new scheme from an excel source to data factory?
Greetings community, I am trying to update my schema in one of my datasets, but it throws this error message: Failed to load: Invalid excel header with empty value, filename is '', sheetname is 'Mayo', the row number is '', the column number is ''.…
asked 2024-07-03T21:58:58.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 9%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGP%3C/text%3E%3C/svg%3E)
Genesis Pena
0
Reputation points
asked 2024-07-03T21:58:58.84+00:00
Genesis Pena
0
Reputation points
1 answer
Error on Data Types string to nvarchar when copying data from Netsuite to Azure SQL database in ADF
I am trying to copy a table from Netsuite to an Azure SQL database using an ODBC connector from Netsuite with SuiteAnalytics Connect. Most of the tables are coming across fine, there are a couple giving this error. I realize it is something to do with…
asked 2024-01-11T21:59:43.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 85%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECC%3C/text%3E%3C/svg%3E)
Chantelle Carley
0
Reputation points
commented 2024-07-03T20:40:19.1133333+00:00
susmit sarker
0
Reputation points
2 answers
ADF Dataflow Debug is not refreshing data
Hi All, I am trying to run my Dataflow pipeline which consists of an inline dataset pointing to Common Data Model and When I run my pipeline for the first time data is loaded properly to sink but it is not fetching latest data in the consecutive runs…
asked 2024-07-03T10:52:32.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 93%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
K J, Sujanya
0
Reputation points
answered 2024-07-03T19:32:38.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 6%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Sujanya KJ
0
Reputation points
2 answers
How do you stop/pause or restart a Workflow Orchestration Manager instance?
We use git-sync with our WOM instance. For DAGs, this works fine, changes to DAGs show up very quickly in Airflow. However, some changes require a restart of the instance, for example changes in the plugin (since those are only loaded once at startup).…
asked 2024-07-03T10:50:27.2933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 19%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
aeteq
0
Reputation points
edited an answer 2024-07-03T17:56:03.6933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,531
Reputation points Microsoft Employee
0 answers
Issue with copy data from Azure Synapse Link for Dataverse using ADF Dataflow
Hi there, We use Export to Data Lake option to copy the data from D365 and want to transform to Azure Synapse Link for Dataverse because Export to Data Lake option is obsoleting from 1st Nov 2024. We have configured Azure Synapse link for Dataverse in…
asked 2024-07-01T11:07:06.5533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 96%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERI%3C/text%3E%3C/svg%3E)
Raheel Islam
21
Reputation points
commented 2024-07-03T17:48:41.4633333+00:00
AnnuKumari-MSFT
31,806
Reputation points Microsoft Employee
0 answers
Why are so many Workflow Orchestration Manager configurations locked? Especially AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION is annoying.
What is the reason that so many configuration settings are locked, some to non-default values? For example, we would like to turn AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION to True but it is currently locked to False. This greatly hinders our workflow:…
asked 2024-07-03T10:45:02.1733333+00:00
aeteq
0
Reputation points
edited a comment 2024-07-03T17:43:07.7+00:00
BhargavaGunnam-MSFT
28,531
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Data Flow in Azure Data Factory has a problem with previews in debug mode
When I refresh the preview (Data preview section--> Refresh button) I can't get new data. It shows again old data! It seems it is using a cache memory and I am not able to disable it. Is there any setting to avoid this problem or is it just a bug? …
asked 2021-06-17T10:24:07.067+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 15%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMC%3C/text%3E%3C/svg%3E)
MASSIMO CAULI
66
Reputation points
answered 2024-07-03T17:42:14.76+00:00
Sujanya KJ
0
Reputation points
0 answers
Is it possible to upgrade the Airflow version of a Workflow Orchestration Manager while it is running?
Currently, only 2.6.3 is available to choose, but when newer versions become available, what is the upgrade path for existing instances? Is it always required to create a completely new instance when upgrading? How would you preserve the Meta-DB with the…
asked 2024-07-03T10:40:56.18+00:00
aeteq
0
Reputation points
commented 2024-07-03T17:28:28.32+00:00
BhargavaGunnam-MSFT
28,531
Reputation points Microsoft Employee
0 answers
When will Workflow Orchestration Manager be updated to newer Airflow versions?
The WOM has been stuck at Airflow Version 2.6.3 for a while now. The current upstream Airflow version is 2.9.2. Is there any (public) timeline on when newer versions will be available? Is there a general release schedule?
asked 2024-07-03T10:38:45.5966667+00:00
aeteq
0
Reputation points
edited a comment 2024-07-03T17:17:42.9433333+00:00
BhargavaGunnam-MSFT
28,531
Reputation points Microsoft Employee
0 answers
I wan use Show billing report by pipeline in ADF, and enabled this via CI/CD
I want enabled the feature "Show billing report by pipeline " in azure Data Factory, but I don't know how can I enabled this via Terraform deployment or ARM template deployment. How Can I do that ?
asked 2024-07-03T15:15:45.9433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 9%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Benjamin Baele
0
Reputation points Microsoft Employee
edited a comment 2024-07-03T17:03:24.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
8,075
Reputation points Microsoft Vendor
0 answers
My Copy Activity only downloads partially my file to blob storage
Hello, I started to receive only partial data using a copy activity in ADF. Recently, I detected that my pipeline is only downloading 1,743,275 rows from an HTTP endpoint. I tried hardly to debug my pipeline and it is not changing the output. I do not…
asked 2024-07-03T11:00:10.6266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 7.000000000000001%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Cristian Rodriguez
0
Reputation points
commented 2024-07-03T16:45:00.89+00:00
ShaikMaheer-MSFT
38,321
Reputation points Microsoft Employee
1 answer
How much would it cost to create an ADF Pipeline with two servers
I need to retrieve data from a MySQL backup file on Cyberduck, restore it in MySQL Azure then move it to our data warehouse in SQL Azure, how much would it cost for building the pipeline using Azure ADF.
asked 2024-07-03T14:38:13.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 97%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYN%3C/text%3E%3C/svg%3E)
Youness Naciri
0
Reputation points
edited the question 2024-07-03T16:29:47.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 16%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Gurram
5,070
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
I'm encountering inconsistencies with FetchXML in Dynamics 365; it fails to load all the all attributes each time I run the query. Despite setting up FetchXML to filter and retrieve all attributes, it only returns a subset. Why is this ?
Here is the query that I am using ADF copy activity: <fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false"> <entity name="@{item().tablename}"> …
asked 2024-06-26T05:05:57.9766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 12%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENZ%3C/text%3E%3C/svg%3E)
Namrata Zalewski (She/Her)
20
Reputation points
accepted 2024-07-03T15:46:28.7066667+00:00
Namrata Zalewski (She/Her)
20
Reputation points
0 answers
Can SAP CDC Connector support variables (parameter) of CDS Views
We want to load data from S/4HANA to Azure with the CDC Connector. The CDS View contains variables for exchange rate type and currency. Question: Is it possible to maintain these parameters/variables within the CDC Connector and transfer them back to…
asked 2024-07-03T13:41:20.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 20%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVL%3C/text%3E%3C/svg%3E)
von Lehsten Tjarko, B2B-PAP-BAP-IND-BC
0
Reputation points
asked 2024-07-03T13:41:20.73+00:00
von Lehsten Tjarko, B2B-PAP-BAP-IND-BC
0
Reputation points
1 answer
How do you add Airflow requirements to a Workflow Orchestration Manager instance without it crashing?
The Workflow Orchestration Manager has an option to add pypi dependencies. However, when I add for example "apache-airflow-providers-databricks", the instance does not start anymore (it tries to start for ~1 hour and then stops with an…
asked 2024-07-03T10:48:07.2333333+00:00
aeteq
0
Reputation points
answered 2024-07-03T13:28:53.79+00:00
Amira Bedhiafi
18,501
Reputation points
0 answers
How can you import data into the Workflow Orchestration Manager Database?
We have been using Airflow on premise for a long time and would like to migrate to WOM without losing our execution history. Is there a way to import data into the Airflow Meta-DB?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
2 answers
Trying to send a file from ADF blob storge to SFTP server but getting an error
Getting an error - SocketErrorCode: 'ConnectionReset'.,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=Renci.SshNet,' while…
asked 2024-07-01T13:35:28.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 12%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENT%3C/text%3E%3C/svg%3E)
Nilesh Tayde
0
Reputation points
commented 2024-07-03T12:58:17.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
4,126
Reputation points Microsoft Vendor
1 answer
How do you access the Airflow CLI/API/DB in Workflow Orchestration Manager?
Working with Airflow, one often requires access to the Airflow CLI/API or even access to the underlying meta database. For example: to import or export connections start backfills manage Variables (using scripts) manage DB items that are…
asked 2024-07-03T10:33:38.4866667+00:00
aeteq
0
Reputation points
edited a comment 2024-07-03T12:20:07.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,090
Reputation points Microsoft Vendor
5 answers
Why does Data Factory Salesforce Connector not pull correct column formats?
We had a data factory pipeline connecting to salesforce using integration runtime 4.9 and everything worked perfect. We upgraded our integration runtime to 5.1 because of a sql server connector issue and now our salesforce connector does not work…
asked 2024-06-18T12:37:25.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 55.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBD%3C/text%3E%3C/svg%3E)
Blasko, Dan W
0
Reputation points
answered 2024-07-03T12:06:55.04+00:00
Adam VEGH
0
Reputation points
2 answers
unable to connect synapse db
Hi, Cannot connect to SQL Database. Please contact SQL server team for further support. Server: 'tcp:ag-syn-bi-group-d.sql.azuresynapse.net,1433', Database: 'bronze', User: ''. Check the linked service configuration is correct, and make sure the SQL…
asked 2024-07-02T14:07:35.56+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
265
Reputation points
commented 2024-07-03T11:08:57.7566667+00:00
Smaran Thoomu
12,090
Reputation points Microsoft Vendor