Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
0 answers
How should I connect DataFactory to a SFTP Server?
Hi, I'm trying to connect My DataFactory to SFTP Server. I have all the details necessary to connect. But I'm currently facing: Invalid Sftp credential provided for 'SshPublicKey' authentication type. openssh key type: ssh-rsa is not supported …
asked 2024-07-01T15:16:18.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 79%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
G S, HarisKumar
0
Reputation points
edited the question 2024-07-01T15:30:50.0133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 16%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Gurram
5,070
Reputation points Microsoft Vendor
1 answer
Trying to send a file from ADF blob storge to SFTP server but getting an error
Getting an error - SocketErrorCode: 'ConnectionReset'.,Source=Microsoft.DataTransfer.ClientLibrary.SftpConnector,''Type=System.Net.Sockets.SocketException,Message=An existing connection was forcibly closed by the remote host,Source=Renci.SshNet,' while…
asked 2024-07-01T13:35:28.1766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 12%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENT%3C/text%3E%3C/svg%3E)
Nilesh Tayde
0
Reputation points
commented 2024-07-01T15:26:43.7+00:00
Nilesh Tayde
0
Reputation points
1 answer
How to handle multiple different output streams in ADF
Hi team, I have a scope script that calls a module with a function that has multiple output rowsets and I was wondering if this is supported in adf. Ultimately, we want to push the data to Kusto in the same different output streams without mixing them.…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 2%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFN%3C/text%3E%3C/svg%3E)
0 answers
Google BigQuery updated linked service error: "Message=No current element is available" for empty result
I started to receive failure with error "Message=No current element is available" for Copy activity with the new Google BigQuery linked service. The legacy one doesnt throw an error/failure when the Google SQL query gives an empty result. I…
asked 2024-07-01T14:42:56.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 0%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQD%3C/text%3E%3C/svg%3E)
Quyen Dang
11
Reputation points
asked 2024-07-01T14:42:56.19+00:00
Quyen Dang
11
Reputation points
2 answers
Formatting a datetime referring to a 24-hours clock
Hi, in an ADF pipeline I'm trying to use this expression: @formatDateTime(utcNow(),'yyyy-MM-ddTHH:mm:ss') but this return a value that shows the AM/PM indicator. I'd like to show the time referring to the 24-hours clock. Any suggests to me, please?…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 41%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
2 answers
CopyData in ForEach
Dear all, I want to use the copydata activity in my pipeline. I have a csv file in my storage account, which has the list of the data i want to pass into the copy activity. I create a lookup activity first and then I try to use ForEach and put the copy…
asked 2024-06-27T09:48:48.74+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 75%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZM%3C/text%3E%3C/svg%3E)
Zhuoqiong Mo
40
Reputation points
answered 2024-07-01T14:27:01.7666667+00:00
Zhuoqiong Mo
40
Reputation points
0 answers
Transform a regular array into an array of objects inside a Mapping Data Flow in Azure Data Factory
In azure data factory, I have a column which is an arrayof strings like in this (image below): costGroup = ["GH", "APT"] Because I need to output in Json format later on, I need to transform it to an array of objects like…
asked 2024-07-01T11:56:30.9366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 88%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJM%3C/text%3E%3C/svg%3E)
Jose Martinez
0
Reputation points
commented 2024-07-01T14:26:31.8+00:00
Jose Martinez
0
Reputation points
2 answers
Event based approach for SharePoint list in ADF
Hi All, I have a share point list and Iam able to connect the same and get data and load it into blob storage I am now interested to know is it possible to do anything event based approach to load data the data into blob
asked 2024-06-26T07:00:26.5833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 50%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Suraj Sasidaran
0
Reputation points
answered 2024-07-01T14:07:16.62+00:00
Suraj Sasidaran
0
Reputation points
0 answers
Does Airflow on ADF (Managed Workflow Orchestration) require Data Factory Contributor role with AAD?
I want to use Airflow on ADF (Managed Workflow Orchestration), but when I try to add a user, they do not get access even though they have been given "Viewer", "User", and "Op" roles in the Airflow UI. I integrated AAD during…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 85%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
1 answer
ErrorCode=ODataFailedClientCreation
we would like transfer data about odata to sql. Our linked service is working. Only the ADF pipeline doesn't work with folling error... …
asked 2024-06-12T05:44:04.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 15%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETA%3C/text%3E%3C/svg%3E)
T1 ADM Sven ***
0
Reputation points
commented 2024-07-01T14:01:07.6133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
7,825
Reputation points Microsoft Vendor
1 answer
How to sync/upload files from Sharepoint to Azure Blob Storage
I am currently trying to create a chatbot that has access to a Azure Blob Storage, so that I can ask questions about the files I upload. Most of the files are located in Sharepoint, so I would like the files to automatically be updated in the Blob…
asked 2024-06-27T21:41:40.9633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 62%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELF%3C/text%3E%3C/svg%3E)
Luke Field
0
Reputation points
commented 2024-07-01T13:59:24.5933333+00:00
phemanth
7,825
Reputation points Microsoft Vendor
1 answer
Copy Activity - ADF
Hi, I am new to azure, while I am learning I am facing an issue that copy activity running twice in a loop. Use case: Copy files from raw to silver container based on file extension. So for this I have created If loop(true==csv, false==JSON) but false is…
asked 2024-06-28T05:22:38.5433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 64%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Nikhil k
0
Reputation points
commented 2024-07-01T13:40:54.1233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,085
Reputation points Microsoft Vendor
2 answers
Is there any way to make Azure Data Factory pipelines wait in queue if infrastructure reaches maximum capacity and not fail?
I have an Event based trigger in Azure Data Factory which executes ETL pipelines every 5 mins. The pipeline has some Databricks notebook activities which executes via a cluster pool. In some cases, the pipelines are failing throwing this error:…
asked 2024-06-26T14:57:09.1266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 23%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDA%3C/text%3E%3C/svg%3E)
Danish Ahmed Mehmuda
0
Reputation points
edited a comment 2024-07-01T13:32:28.7033333+00:00
Smaran Thoomu
12,085
Reputation points Microsoft Vendor
1 answer
The costing for the Batch services and I have trouble running my Python job in Azure Batch because the required directories are not present on the Batch node.
I would like to know how Batch service is charged. When I run my Python jobs through ADF's Batch service, I get som troubles. For example: My test.py script and the utils folder are located in the run_python folder. The test.py script calls the…
asked 2024-06-20T05:46:01.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 9%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZX%3C/text%3E%3C/svg%3E)
谢志青 Zhiqing Xie
40
Reputation points
commented 2024-07-01T13:28:46.83+00:00
Smaran Thoomu
12,085
Reputation points Microsoft Vendor
2 answers
ADF Pipeline -> "Azure Key Vault"
Hi, I have created a web activity in a pipeline, I want to type the password and not select the password from Azure Key Vault. I'm not able to type the password. Can you please guide? Thanks and Regards, Pratisti Pratap Satatdekar
asked 2024-06-24T06:36:10.0266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 70%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pratisti Satardekar
130
Reputation points
commented 2024-07-01T13:21:10.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
5,420
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Which is best option for an environment with analytic reporting to be done in the cloud and require on premise data and it require frequent updates during the day
Hi, I would like to understand what is the best solution to implement with the following requirements: Cloud environment for analytical reporting where data will be sourced from on premise and then combined with data loaded from excel along with have…
asked 2024-06-28T12:55:21.7733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 24%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Naidoo, S. (Sushiela)
20
Reputation points
accepted 2024-07-01T13:18:55.0033333+00:00
Naidoo, S. (Sushiela)
20
Reputation points
1 answer
Unroll By button for flatten is grayed out
I'm new to Azure Data Factory. In Azure Data Factory I have an XML file I want to flatten and convert to a CSV file. I add a Data Flow, and to the Data Flow I add a "source1" where I have already done an "Import projection". I added…
asked 2024-06-24T14:24:20.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 77%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Chuck Roberts
65
Reputation points
commented 2024-07-01T12:32:20.4766667+00:00
Chuck Roberts
65
Reputation points
1 answer
ODBC Error [HY104] during Data Transfer to Oracle via Azure Data Factory: Character, decimal, and binary parameters cannot have a precision of zero
Hello, I am encountering an issue while transferring data to an Oracle database using Azure Data Factory copy activity. The operation fails with the following error…
asked 2024-07-01T08:57:18.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 97%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETD%3C/text%3E%3C/svg%3E)
Tommaso Di Gennaro
0
Reputation points
commented 2024-07-01T11:18:58.7333333+00:00
Harishga
5,420
Reputation points Microsoft Vendor
0 answers
Issue with copy data from Azure Synapse Link for Dataverse using ADF Dataflow
Hi there, We use Export to Data Lake option to copy the data from D365 and want to transform to Azure Synapse Link for Dataverse because Export to Data Lake option is obsoleting from 1st Nov 2024. We have configured Azure Synapse link for Dataverse in…
asked 2024-07-01T11:07:06.5533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 96%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERI%3C/text%3E%3C/svg%3E)
Raheel Islam
21
Reputation points
asked 2024-07-01T11:07:06.5533333+00:00
Raheel Islam
21
Reputation points
1 answer
Facing issue while using ADF Copy data activity from Azure to Kusto Bugs table
Dear Team, We are experiencing an issue with data transfer from an Azure Storage Table to a Kusto Table using Azure Data Factory (ADF). Specifically, the System_AssignedTo field, which is blank in the Azure Storage Table, is being populated with…
asked 2024-06-28T09:29:30.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 33%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVV%3C/text%3E%3C/svg%3E)
Vikas Verma (HCL TECHNOLOGIES CORPORATE SER)
25
Reputation points Microsoft Vendor
answered 2024-07-01T10:44:26.2766667+00:00
Amira Bedhiafi
18,341
Reputation points