9,643 questions with Azure Data Factory tags

Sort by: Updated
Updated Created Answers
1 answer One of the answers was accepted by the question author.

Set variable from lookup activity result

Hi, I have a lookup activity in and ADF pipeline connected to a set variable activity: The query in lookup activity is: Select name from [dbo].[someTable] where [ID] = '@{pipeline().parameters.pl_ID}' name is actually an integer …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-23T11:04:28.687+00:00
Obaid UrRehman 166 Reputation points
accepted 2022-04-05T16:35:59.64+00:00
Obaid UrRehman 166 Reputation points
1 answer

Copy Activity Fails while copying Multiple JSON Files in ADLS to Azure SQL DB

I have Multiple JSON Files in ADLS GEN2 ( 100s ) ,when there is no Error File the ADF V2 Copy Activity, Copies Entire Files from ADLS Gen2 to Azure SQL DB. But when there is one error File in the List then the Entire Copying gets failed. I need to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-31T20:03:28.737+00:00
Suchit Kumar Battula 6 Reputation points
commented 2022-04-05T16:25:19.477+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
1 answer

Invoking RESI API and Uploading File through API Management using ADF Data Pipeline

Hello, We have a requirement to Invoke a REST API call through API Management using ADF Data Pipeline. Data Pipeline would have the source as ADLS Gen2 where CSV files would be extracted and invoke External REST API call through API Management API,…

Azure API Management
Azure API Management
An Azure service that provides a hybrid, multi-cloud management platform for APIs.
1,782 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,355 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-29T02:29:04.047+00:00
Jayaprakash Kumarasamy 1 Reputation point
commented 2022-04-05T16:22:52.303+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
0 answers

ADF pipeline errors out with pagination logic for Rest API

I have a pipeline having data flow to get the data from a Rest API where I am sending the page size, page number (SET TO 0) date as filter as the API request. The response contains the json data and total available records and the total pages based on…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-25T12:23:11.203+00:00
Geeta 1 Reputation point
commented 2022-04-05T16:18:36.32+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
1 answer

SAP Table to Azure Data Lake - Partition Option

Scenario: Copy data from SAP table to Azure Data Lake Leveraging 'Partition option' in Copy data activity as without it we are getting memory error in SAP when pulling data Questions: => How to load bulk data if I do not have a partition…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,422 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-18T23:25:06.747+00:00
Satwik Kashyap 11 Reputation points
commented 2022-04-05T15:59:44.75+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
2 answers

Implement the Rest API pagination rule with Twitter API

Hi everyone, I'm trying to get tweets data in a json file with Microsoft Azure Data Factory. I've created a copy data activity which takes the url for the get tweets api with the needed parameters (those parameters are included in the dataset…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-01T16:52:05.51+00:00
Mikael BURBAN 1 Reputation point
commented 2022-04-05T15:48:12.823+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
1 answer

Azure Data Factory Error : Operation on target Azure Function1 failed: Call to provided Azure function 'XXX' failed with status-'NotFound'

Hi, I have created a pipeline in ADF , with DataFlow Activity from container Storage -> Azure SQL. On success , my HTTP Function is triggered , which in turn processes the SQL Data inserted. The Blob -> SQL Dataflow is working fine , but my…

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,321 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2021-06-23T08:34:00.193+00:00
Tina.Jain.Gupta 1 Reputation point
commented 2022-04-05T12:23:59.51+00:00
MayankBargali-MSFT 68,746 Reputation points
1 answer One of the answers was accepted by the question author.

Azure Data Factory CPU usage in dataflows

Hi all, We have a dataflow in Azure Data Factory that is transforming data from a table in a database. We create several activities like Derived Columns, Joins, Filters.. which all use this table as the source. The question is if the CPU from that…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-04-04T14:44:43.87+00:00
Anonymous
accepted 2022-04-05T10:10:41.273+00:00
Anonymous
1 answer One of the answers was accepted by the question author.

Logging Pipeline exection

I am creating a parameterized pipeline to load data from blob storage to Cosmos DB . Trying to implement a logging framework for the pipeline execution details. The logging framework which everyone proposes is based on tables in Azure SQL Databases .(I…

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,454 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2021-12-09T05:08:34.303+00:00
Litty Kuriakose 106 Reputation points
accepted 2022-04-05T08:54:14.017+00:00
Litty Kuriakose 106 Reputation points
1 answer One of the answers was accepted by the question author.

pipeline output Run Start

Hello, Do you know why in the pipeline output window, under Run start column, I see the date and time where time is one hour behind my local time(uk)? Thank you

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-31T13:12:55.107+00:00
arkiboys 9,646 Reputation points
accepted 2022-04-05T06:23:52.4+00:00
arkiboys 9,646 Reputation points
3 answers One of the answers was accepted by the question author.

How to Trigger ADF pipeline from another ADF?

I'm trying to trigger one adf(transformation) pipeline from another adf(ingestion) pipeline, Both the adf's are under same subscription but different resources. We don't want to use even hubs. I'm looking for alternative solutions.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-30T08:01:51.467+00:00
Santosh 21 Reputation points
accepted 2022-04-05T06:04:55.45+00:00
Santosh 21 Reputation points
2 answers

Lift and shift ADF using powershell

I need to lift and shift ADF (all contents) from subscription A to subscription B using release pipeline (CI/CD). I am able to do that using ARM template. But my requirement is to achieve it using powershell script. I need to design a powershell…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
Azure Automation
Azure Automation
An Azure service that is used to automate, configure, and install updates across hybrid environments.
1,131 questions
Windows Server PowerShell
Windows Server PowerShell
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.PowerShell: A family of Microsoft task automation and configuration management frameworks consisting of a command-line shell and associated scripting language.
5,389 questions
asked 2022-03-31T04:27:15.287+00:00
Racquelle 1 Reputation point
commented 2022-04-05T05:57:48.477+00:00
PRADEEPCHEEKATLA-MSFT 78,331 Reputation points Microsoft Employee
2 answers

ADF SQL Managed Instance (SHIR) Service Principal Authentication - Inline vs Credential(Preview) - Test Connection Oddity

Notable Scenario here that has different behavior switching settings back and forth that is reproducible: - SQL Managed Instance Linked Service - AAD Service Principal Authentication (using certificate stored in keyvault as credential) - Self Hosted…

Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,135 questions
Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
19,661 questions
asked 2021-09-02T20:13:24.25+00:00
Chad Bentz 26 Reputation points Microsoft Employee
commented 2022-04-05T04:30:57.117+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

reading .json data

Hello, I have many large .json files. In it there is a field name as companyNames for each region some regions have one companyName and other regions have more than one companyNames And so the files may as follows: ... region1[ ..., …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-23T05:17:27.163+00:00
arkiboys 9,646 Reputation points
commented 2022-04-04T17:38:15.43+00:00
arkiboys 9,646 Reputation points
1 answer

Add Dynamic column name to additional column in copy activity

Hi Team, I am creating copy from SQL database to D365 and i have to do mapping between one studentID column to two fields from D365 F&O. I am using additional column from copy activity to add duplicate column from SQL database. but…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-24T14:52:43.42+00:00
Mahesh Totageri 21 Reputation points
commented 2022-04-04T16:40:52.927+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
1 answer

Dynamic Expression By Timezone Broke

For over a year I've been running the following dynamic expression in order to get the most recent sunday in a specific format in eastern time. Then this week it broke: …

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,422 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-25T12:30:36.257+00:00
B K 41 Reputation points
commented 2022-04-04T16:36:50.047+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
1 answer

Copy multiple CSV Files to SQL Database (validations and foreign keys)

Hi, I'm new here and also using the Data Factory tool and well, I explain. I managed to insert the information from the csv files to their respective tables, with the activities: Get MetaData, forEach (activity: copy data) . As the example of this…

Azure SQL Database
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,721 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-25T22:07:28.173+00:00
Mvdit0 1 Reputation point
commented 2022-04-04T16:34:05.797+00:00
ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
1 answer

Can I use DataFactory to access a 3rd party data file and pull it into Azure?

I am investigating DataFactory. I have a 3rd party who happens to be in the same Azure region. They will be creating a file that I need to process in my Azure tenant. I was hoping that through the DataFactory pipeline that I would be able to connect…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-04-04T15:10:31.36+00:00
Ed Hickey 11 Reputation points
answered 2022-04-04T15:20:20.427+00:00
Ed Hickey 11 Reputation points
1 answer One of the answers was accepted by the question author.

Want to decrypt csv files coming from aws s3 bucket using adf in blob using pgp encryption/decryption

Hi, I have created an adf pipeline which is copying .csv files from s3 to blob . The files in the s3 are encrypted using PGP encryption. Now since these are copied to blob so now i need to decrypt these files using PGP in adf so that i could use them…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,449 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-31T15:10:26.153+00:00
Amar Agnihotri 891 Reputation points
commented 2022-04-04T13:32:11.553+00:00
Amar Agnihotri 891 Reputation points
1 answer One of the answers was accepted by the question author.

replace expression

At present in dataflow1, there are two derived columns derivedcolumn1 : Column_1 --> replace(Column_1, '"address":{', '"address":[{') derivedcolumn2: Column_1 --> replace(Column_1, '},"office"',…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,643 questions
asked 2022-03-30T15:16:11.953+00:00
arkiboys 9,646 Reputation points
accepted 2022-04-04T12:03:48.613+00:00
arkiboys 9,646 Reputation points