Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,805 questions
1 answer
One of the answers was accepted by the question author.
SQL Dynamic query not working
How can I replace a question mark "?" in a SQL dynamic query with pipeline parameters in Azure Data Factory? When I attempt to pass the parameters, ADF does not recognize the "?" symbol, and the parameters are not correctly passed.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 64%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
1 answer
Authentication Token Issues with Azure Data Factory When Accessing Microsoft Graph API
I am currently facing challenges with an Azure Data Factory pipeline that's configured to access data from the Microsoft Graph in a production environment. While I can successfully fetch authentication tokens using a POST request…
asked 2024-05-29T19:46:22.4333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 91%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EXI%3C/text%3E%3C/svg%3E)
Xavier Ibanez-Padron
0
Reputation points
answered 2024-06-01T17:36:37.5266667+00:00
Amira Bedhiafi
16,951
Reputation points
1 answer
Updating Oracle on-premise table from a Synapse table
Hi, I need to update an Oracle on-premise table from a Synapse table by joining the Oracle table with the Synapse table avoiding to build a staging table on Oracle platform to copy the Synapse data. It is a problem to obtain the authorization from my…
asked 2024-06-01T12:00:58.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 41%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
pmscorca
817
Reputation points
answered 2024-06-01T17:20:04.4833333+00:00
Amira Bedhiafi
16,951
Reputation points
2 answers
One of the answers was accepted by the question author.
How to copy files from sharepoint drive that has huge heirarcy of folders and subfolders
We want to get all files from the sharepoint into our ADLS container. In sharepoint drive there are arround 60k files but through our existing pipeline we are getting only 600 files. We have a web activity with below…
asked 2024-05-12T03:16:23.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 79%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Rayne
40
Reputation points
edited a comment 2024-06-01T14:30:36.69+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 35%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMY%3C/text%3E%3C/svg%3E)
Mansi Yadav
40
Reputation points
1 answer
One of the answers was accepted by the question author.
How to compare datetime stamp of string type in ADF
I'm using graph api drives/drive-id/items/root/children to get all items in root and the I'm filtering files and folders. I am also passing lastrun time stamp in pipeline parameter 2024-05-30T09:03:00Z . I want to add one more condition in filter for…
asked 2024-05-30T06:13:20.6366667+00:00
Mansi Yadav
40
Reputation points
accepted 2024-06-01T13:40:28.5266667+00:00
Mansi Yadav
40
Reputation points
1 answer
ByPass custom logic executions in Azure Data factory copy activity
Hi, I am planning to do data refresh project using Azure data factory and like to disable the custom logics execution. is there any way I can do in ADF copy activity? Like…
asked 2024-05-31T08:32:26.37+00:00
Maniraj K V
20
Reputation points
commented 2024-06-01T13:39:05.6666667+00:00
Maniraj K V
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Moving files after a copy from a blob storage
Hi, I've implemented a pipeline to read a set of txt files from a blob storage, searched by a certain string, and then to write them on a shared folder on-premise. For a such task I've used a copy activity. Now, how I could move the copied files from the…
2 answers
Azure to AWS
Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…
asked 2024-05-20T08:39:35.8366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sourav
80
Reputation points
edited an answer 2024-06-01T10:27:52.2833333+00:00
Sumarigo-MSFT
44,251
Reputation points Microsoft Employee
1 answer
Delta to Parquet mapping data flow resulting in one empty partition of 2
Hi, I've been working on something, but I can't get it to work. I seem to have found the issue, but I"m not able to fix it. I've isolated in a single run which I'll explain below: I have a Delta table on ADLSv2 (based on one partition) that I want…
asked 2024-05-31T13:29:25.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 8%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEW%3C/text%3E%3C/svg%3E)
Erp, Wessel van
1
Reputation point
answered 2024-06-01T09:18:52.72+00:00
Amira Bedhiafi
16,951
Reputation points
1 answer
Failed to convert the value in 'xxxx' property to 'System.String' type in Data Lake Connector
Environment: Azure Synapse I am developing a pipeline that has a lookup to SQL that is passing the values to GetMetaData. When using @dataset.name of parameter in the dataset, it fails with "Failed to convert the value in name of parameter…
asked 2024-05-31T19:42:29.6733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 18%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDC%3C/text%3E%3C/svg%3E)
Doreen Cecil
0
Reputation points
answered 2024-06-01T09:09:28.1+00:00
Amira Bedhiafi
16,951
Reputation points
1 answer
How to flatten dynamic Json in ADF/Synapse
I have a issue where I have dynamic json and enabled schema drift in my source, To flatten the json I am coding it as array(byName('Columnname')) in unroll by dynamic content section. Here columnname is the array which need to be unrolled. I am getting…
asked 2021-10-02T02:06:35.697+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 97%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rashmi Shetty
6
Reputation points
commented 2024-06-01T05:24:48.3433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 8%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHB%3C/text%3E%3C/svg%3E)
Harman Brar
0
Reputation points
1 answer
Copy activity throwing an error, when a auto incremented column is there on the Sink side
Hello Team, I am trying to copy data from Amazon S3 to Azure SQL database. On Azure SQL side I have set an autoincrement column(Row_Index) in a table, however the copy activity throws an error that "Column 'Row_Index' does not allow…
asked 2024-05-15T22:53:21.2033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 0%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shweta Chavan
40
Reputation points
commented 2024-05-31T22:33:28.77+00:00
Oury Ba-MSFT
16,901
Reputation points Microsoft Employee
1 answer
Failed to get access token from your token endpoint. Error message: No MediaTypeFormatter is available to read an object of type 'OAuth2AccessToken' from content with media type 'text/html
In Azure Data Factory I am trying to connect to a REST API that uses OAuth 2.0 Authentication. When supplying Data Factory with all the necessary credentials and testing the connection I get the error: Failed to get access token from your token…
asked 2024-05-29T16:36:57.1933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 50%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Josh Starita
0
Reputation points
edited a comment 2024-05-31T20:47:52.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
4,485
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
DF-SRC-002 at Sink 'DatasetSink'(Line 176/Col 13): 'tableName' (Table Name) is required
Hi, I have a dataflow that reads data from a source (Azure dataset), here is the source config: I have the desired output before the sink. The sink configuration as as follows: The problem is when I try to preview the data, it gives…
asked 2022-03-17T15:22:56.727+00:00
Obaid Ur Rehman
86
Reputation points
commented 2024-05-31T20:28:10.1133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 0%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Braun, Bill
5
Reputation points
2 answers
Azure AD Token Customization
I have registered an application in azure ad through which I am getting the token which is used for authentication. I have added some app roles to the application which is mapped at group level in enterprise application. While decoding the token in…
asked 2024-05-30T04:44:09.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 76%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDP%3C/text%3E%3C/svg%3E)
Dinesh Prajapati
126
Reputation points
edited a comment 2024-05-31T19:29:31.9633333+00:00
Amira Bedhiafi
16,951
Reputation points
1 answer
How to ignore the records by applying an auditing filed column condition using ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
asked 2024-05-30T09:28:46.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
45
Reputation points
commented 2024-05-31T19:27:52.55+00:00
Amira Bedhiafi
16,951
Reputation points
1 answer
Azure Data Factory pagination using QueryParameters and body field
Hello everyone, I am currently working with a pagination system and facing a challenge with the URL and parameter concatenation. I am trying to append a specific string to the URL retrieved from the first page of an API response. Here's the setup: API…
asked 2024-05-09T22:23:30.69+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 15%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Max Teitelbaum
0
Reputation points
commented 2024-05-31T18:07:37.13+00:00
Max Teitelbaum
0
Reputation points
1 answer
How to fix these errors and warnings with Integration Runtime (Self-hosted) Host Service on Windows
Our Windows server administrator found that the Windows event log is so full that it can only hold entries for the last few hours. The following error occurs 3 times in a row in the Event Viewer every couple of hours: Log Name: Integration…
asked 2024-05-30T15:17:54.2933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 89%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Richard Martinez
0
Reputation points
commented 2024-05-31T17:27:09.4933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
1 answer
How to add the "Azure Machine Learning workspace name" which exists in the data factory Azure Machine Learning linked service in the override parameters in the release pipeline?
I have two adf dev and prod. I have the linked service "Azure Machine Learning". Inside this linked service I have some parameters such us, subscriptionID, resourceGroupName and Azure Machine Learning workspace name**.** In my pipeline…
asked 2024-05-23T11:02:38.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 30%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dina
0
Reputation points
edited a comment 2024-05-31T15:39:31.19+00:00
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
0 answers
Missing cerificate
I passed the Azure Data Fundamentals exam(2nd certificate) via Certiport, but the certificate isn't linking to my account even though my email ID is linked and also, I've already completed the azure fundamentals exam(1st certificate) and only that…
asked 2024-05-31T15:31:07.3866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 13%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPG%3C/text%3E%3C/svg%3E)
PAVITHRA GUNASEKARAN
0
Reputation points
asked 2024-05-31T15:31:07.3866667+00:00
PAVITHRA GUNASEKARAN
0
Reputation points