Microsoft Q&A

Azure Data Factory

6,716 questions

An Azure service for ingesting, preparing, and transforming data at scale.

Browse all Azure tags

6,716 questions with Azure Data Factory tags

Sort by: Updated
1 answer

Azure Data Factory Rest Linked service returns Array og Json objects instead of Set of Json Objects

I have set up a Linked Service of type "REST" to copy data from an API to Blob storage. I have a copy activity, which uses a dataset from the Linked Service and a Json dataset on top of a Blob storage Linked service. The API that I am using,…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-07T17:11:36.48+00:00
Thomas Sørensen 0 Reputation points
commented 2023-03-30T18:50:55.65+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
0 answers

How to read .xls file with content type application/octet-stream using copy activity

Hi Team, I am not able to copy .xls file which having content type as application/octet-stream. Can you please help me to resolve this issue without changing file extension.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
Excel
Excel
A family of Microsoft spreadsheet software with tools for analyzing, charting, and communicating data.
309 questions
asked 2023-03-30T07:27:35.15+00:00
Sharad Gawade 0 Reputation points
commented 2023-03-30T18:45:31.3166667+00:00
BhargavaGunnam-MSFT 10,636 Reputation points Microsoft Employee
1 answer

Dataflow Error: The spark job of Dataflow completed, but the runtime state is either null or still InProgress.

Hi Microsoft Team, I have this error whenever I run with "Debug" a pipeline that contains a Data Flow. I tried to create new pipeline, new Data Flow but still the same error. It can be reproduced all the time. My last Pipeline Run ID is:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-06T09:11:03.01+00:00
Cosmin Nicolae Cristea 0 Reputation points
commented 2023-03-30T18:33:01.2733333+00:00
Bert Natenstedt 1 Reputation point
1 answer

DAG runs stuck in running state using Azure Data Factory Managed Airflow (version 2.4.3)

I'm following Microsoft's tutorial on how does managed airflow work using the tutorial.py script referenced in the documentation (see code block below). I've set up my airflow environment in azure data factory using the same configuration in the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-02-28T03:13:18.7466667+00:00
Pete Trenkwalder 5 Reputation points
commented 2023-03-30T17:56:50.8233333+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
0 answers

How to handle pagination with $skipToken in Azure Data Factory

I'm curious how to handle pagination with $skipToken in Azure Data Factory According to the documentation (Pagination support) it seems quite simple in case when we have @odata.nextLink to use as next URL (Microsoft Graph API), but it's more complicated…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-02-27T20:15:26.1933333+00:00
Bartosz Pelikan 36 Reputation points
commented 2023-03-30T17:47:40.3433333+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
1 answer

ADF Rest API - How To Handle "too many requests" error (429)

Hello, I am struggling with integrating API source into Azure SQL DB using Azure Data Factory. Problem is that I am getting "too many requests" error (429) randomly when trying to get data from API source. It looks like error is caused by some…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-02-27T14:54:11.87+00:00
Tomasz Swinarski 0 Reputation points
edited a comment 2023-03-30T17:46:21.0966667+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
1 answer

SFTP Error of Host Not Found in Azure Data Factory

Hello Team, Greetings of the Day!!! Getting error while connecting to SFTP in Azure Data Factory. Doing Azure Data Factory HOL Lab from GitHub :-…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
711 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,543 questions
asked 2023-02-22T11:26:06.9466667+00:00
Niral Dave 5 Reputation points
commented 2023-03-30T17:30:16.42+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
0 answers

how to dynamically transform one activity output and feed in another activity in ADF

If I directly give (@{item().id} @{item().col1} @{item().col2} @{item().name} @{item().col5}) in set variable is gives back result "C0004\t1000\t\tAT020\t1102" but I give the activity output (@activity('Lookup_DB_column…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-29T13:47:52.91+00:00
Arundhati Sen 101 Reputation points
commented 2023-03-30T17:28:11.8933333+00:00
BhargavaGunnam-MSFT 10,636 Reputation points Microsoft Employee
1 answer

String was not recognized as a valid DateTime

Hi team, The data in my excel is in format dd-MM-yyyy but while copying to azure postgressql database, I am getting error saying that "String was not recognized as a valid DateTime". Could you please help me on this asap

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-02-22T11:13:15.66+00:00
Dinesh Prajapati 86 Reputation points
commented 2023-03-30T17:24:33.5133333+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
0 answers

Pulling service now data into azure blob using Azure Service Now connector through ADF pipeline

I am trying to pull entire incident table history from service now. Using following items in my ADF Pipeline. Service Now ADF Connector. Using basic Service Now authentication and tested the connection, it works Able to preview the records…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-30T17:06:21.9+00:00
Isukapati, Chandra 0 Reputation points
1 answer

Azure Data Factory - Smartsheet Linked service errors

The new Smartsheet linked service in Azure data Factory has some errors and I couldn't find any reliable solutions. If there is one please let me know. below are the errors I found: This linked service doesn't work when using it with Azure Key…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2022-07-20T14:19:47.267+00:00
Potlapally, Harsha 1 Reputation point
edited the question 2023-03-30T17:04:26.6766667+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

ADF's File Exists Trigger does reads the files but doesn't fire off Switch activity in pipeline, any idea?

So our ADF V2 File exists trigger we setup was able read the input files but doesn't fire off the linked Switch activity. Specifically, in our case, it correctly read the file "Mashgin_Order_Items_20221001.csv" from the Azure Blob storage…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-27T17:02:25.2033333+00:00
WarrenL 20 Reputation points
commented 2023-03-30T17:04:02.42+00:00
WarrenL 20 Reputation points
3 answers

Azure Data Factory (ADF) Escape Character Issue When Quoting

I've got a pipeline built in Azure Data Factory that's pulling data from SQL and storing it as csv. My Dataset for the csv has Escape Character set to a blackslash (\) and Quote Character set to Double Quote ("). The problem this causes is when I…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2020-10-03T19:40:10.38+00:00
AzureHero 41 Reputation points
edited the question 2023-03-30T17:00:05.7133333+00:00
HimanshuSinha-msft 18,536 Reputation points Microsoft Employee
1 answer

Azure Data Factory Copy Data From ADLS Gen 2 to Synapse failed

Hello, I am trying to copy data from ADLS Gen 2 to Synapse, but encounter the error "An error occurred while sending the request". Message is "The remote server returned an error: (503) Server Unavailable". The synapse is using…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,877 questions
asked 2021-09-27T03:27:59.587+00:00
Jacky 41 Reputation points
commented 2023-03-30T16:57:12.4+00:00
KranthiPakala-MSFT 35,932 Reputation points Microsoft Employee
1 answer

Can't import Live resources to GitHub Repo

I have an existing GitHub repository and I want to import resources from my ADF Live environment to the repo. I can connect to it fine and browse the repos in my account, but when I click Import I get this error. You don't have permission to import…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-23T14:15:46.97+00:00
Mark Allison 126 Reputation points
commented 2023-03-30T16:28:40.1133333+00:00
Mark Allison 126 Reputation points
3 answers

Issue with ADF Pipeline loosing Connectivity Refreshing Analaysis services Model

Erroor Recieved - Operation on target ThrowErrorOnFailure failed: Error calling the endpoint ''. Response status code: 'NA - Unknown'. More details: Exception message: 'NA - Unknown [ClientSideException] Invalid URI: The URI scheme is not valid.'. …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,877 questions
asked 2022-05-08T11:43:38.72+00:00
Elvin Caldwell 1 Reputation point
commented 2023-03-30T15:37:45.3833333+00:00
83390558 0 Reputation points
1 answer

ADF Dataflows flatten a JSON using unroll by not working.

Hi I have a JSON like this in my file and i am trying t flatten it using the flatten transformation in dataflow. when i click on unroll by all the elements are displayed but i am unable to select any.. how can i do that .? or is there any other way i…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-29T23:53:26.4+00:00
Tarapareddy,Saikrishna 61 Reputation points
answered 2023-03-30T14:58:54.98+00:00
KranthiPakala-MSFT 35,932 Reputation points Microsoft Employee
0 answers

Error code 3608: 'Invoking Azure function failed with HttpStatusCode - InternalServerError.'

Dear Team, Pipelines are getting failure because of this Error associated with Azure Function. Please assist me with your solution.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
2,808 questions
asked 2023-03-30T13:29:29.4633333+00:00
Rajkumar Govindaraj 20 Reputation points
edited the question 2023-03-30T14:54:56.6433333+00:00
Rajkumar Govindaraj 20 Reputation points
0 answers

ADF - Run an activity if any one of a set of activities fail and capture failed activity details in that activity

Currently I am running one generic activity when any of the activities in the pipeline fail based on the answer provided in https://learn.microsoft.com/en-us/answers/questions/208812/adf-run-an-activity-if-any-one-of-a-set-of-activit But I want to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
asked 2023-03-30T14:32:10.3+00:00
Prasad Ambulkar 0 Reputation points
edited the question 2023-03-30T14:53:37.7233333+00:00
Prasad Ambulkar 0 Reputation points
0 answers

Nested Execute Pipeline issue in ADF master pipeline

Hi ,In my case i have a master pipeline which has nested Execute pipelines like execute1 pipeline contains lookup and foreach inside foreach execute2 .inside execute 2 get metadata and and foreach again inside foreach execute3 .execute3 contains lookup…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,716 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,877 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
937 questions
asked 2023-03-30T14:36:54.07+00:00
Prapul Kumar Dongari 71 Reputation points