Microsoft Q&A

Azure Data Factory

6,303 questions

An Azure service for ingesting, preparing, and transforming data at scale.

Browse all Azure tags

6,303 questions with Azure Data Factory tags

Sort by: Updated
1 answer

How to skip/ignore zip files that have been unzipped successfully before?

I want to use this system within my work environment by letting users put zip files into the Azure Blob Storage and use Azure Data Factory to automate the process of extracting/unzipping zip files everyday at the specific time. I have followed the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,406 questions
asked 2023-02-03T06:47:32.7033333+00:00
Niramit Cheunprapanusorn 0 Reputation points
answered 2023-02-04T07:54:21.74+00:00
Subashri Vasudevan 4,296 Reputation points
1 answer

Azure Data Factory - replace expression in derived column transformation using Mapping Data Flow

Dear Team, I am trying to use Azure Data Factory for a Derived column transformation task one of my tasks is mentioned below, DESCRIPTION_TEXT : UNILEVER GROUP ##### GBR Remove trailing country code (only when it equals to country) and ##### if they…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-03T11:56:03.47+00:00
Khatri, Amisha 0 Reputation points
answered 2023-02-04T07:04:17.3133333+00:00
Subashri Vasudevan 4,296 Reputation points
1 answer

why am I unable to get display of 3,00,000 records in azure sql database with price-tire-50 /100 DTUs?

Hello There, I recently started using Azure SQL Database as part of my Azure DE learning purpose. I tried to run the query 'select (*) from' over a table that has around 3,00,000 records -- to just fetch the records ; tried this in Query-editor within…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure SQL Database
asked 2023-02-03T18:57:55.9366667+00:00
ArunRaaman 936 Reputation points
edited an answer 2023-02-04T03:36:25.7766667+00:00
Alberto Morillo 24,801 Reputation points Microsoft MVP
0 answers

Pagination of REST API call with "Link" in Header and rel="next"

Hello, I have a hard time doing pagination in ADF dataflow activity. Tried many of the below, nothing worked. My Link in Header is like below - Any idea how to work on it?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,625 questions
asked 2023-02-02T01:11:52.59+00:00
Kumar, Arun 21 Reputation points
commented 2023-02-04T01:22:22.2+00:00
Kumar, Arun 21 Reputation points
1 answer

Where is the storage account key for cosmos db?

I have a copy data pipeline which loads data from a json file within a blob to a cosmos DB collection. Now the sink is the cosmos dB connection . At creation time account selection method is Azure subscription , subscription is selected and then Azure…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
917 questions
asked 2023-01-27T14:46:21.0766667+00:00
edited a comment 2023-02-04T00:35:25.92+00:00
Oury Ba-MSFT 9,566 Reputation points Microsoft Employee
0 answers

Change Data Capture from Data Lake source using Common Data Model?

I have some D365 sample data exported to Data Lake using Common Data Model. In the Azure Data Factory Data Flow, I have configured a source to read from the manifest file under Changefeed folder and selected in this case the "HcmWorker" entity.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
895 questions
asked 2023-01-26T14:36:46.0866667+00:00
Kim Beck 0 Reputation points
commented 2023-02-03T23:08:49.12+00:00
Kim Beck 0 Reputation points
0 answers

ADF is not able to access Keyvault secrests via ADB notebook

We have stored credentials in Azure keyvault we have provided access to both ADF and ADB to access keyvault details.(Get,List). we are able to access secret directly from ADF if we create sample pipeline to check the access. we are able to access secret…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,214 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-01-10T12:04:57.297+00:00
Amardeep Aklujkar 26 Reputation points
edited a comment 2023-02-03T23:04:37.6333333+00:00
KranthiPakala-MSFT 33,136 Reputation points Microsoft Employee
1 answer

Copy Method in ADF - Polybase / Copy Command / Bulk Insert

Hi, We have data of million rows in delimited files placed in Azure Blob storage. We need to transfer to dedicated sql pool of Azure Synapse via Azure Data Factory So which copy method would be suitable in ADF - Copy command / polybase / Bulk Insert.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,625 questions
asked 2023-02-02T07:19:52.46+00:00
Murugeswari 441 Reputation points
commented 2023-02-03T22:40:19.0466667+00:00
KranthiPakala-MSFT 33,136 Reputation points Microsoft Employee
0 answers

ADF Copy Activity - Disable chunking when unchecked does not copy a xlsx file

Hi Team, The issue is while using Copy activity in ADF. Issue occurs when disable chunking is unchecked (default value). As soon as we set disable chunking to true that file is copied successfully. The file has about 36000 records. We receive the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-02T03:48:27.89+00:00
edited a comment 2023-02-03T22:38:59.7233333+00:00
KranthiPakala-MSFT 33,136 Reputation points Microsoft Employee
0 answers

My data flow window is empty even though the code is still there

Hello everyone! I was working on my data flow yesterday and it was all going well. When I reopened my browser this morning, I noticed that my data flow window is completely empty. It's all blank, not even the parameters show up or the option in an empty…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-01T09:52:14.9833333+00:00
Campos, Francisco 0 Reputation points
commented 2023-02-03T22:36:42.5466667+00:00
KranthiPakala-MSFT 33,136 Reputation points Microsoft Employee
0 answers

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'IR(Manage Private Network)',

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime. Error code :4502. PL was running last 3 months and suddenly consistently failing with above error. I tried use high IR…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-01T01:51:12.3633333+00:00
Adf2023 5 Reputation points
commented 2023-02-03T22:35:03.0933333+00:00
KranthiPakala-MSFT 33,136 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Script activity failed: Argument {0} is null or empty. Parameter name: paraKey

Hi, Please see this topic for reference: https://learn.microsoft.com/en-us/answers/questions/763892/script-activity-failed-argument-0-is-null-or-empty.html I'm getting the same error in the Script1 activity in the following JSON: { …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2022-03-17T12:00:49.327+00:00
Francisco Dominguez 356 Reputation points
edited the question 2023-02-03T22:24:40.5466667+00:00
KranthiPakala-MSFT 33,136 Reputation points Microsoft Employee
0 answers

IndexError: string index out of range - ADF - Databricks - Python

I'm passing a UDF value from a JSON table through Azure Data Factory as the following: "schema_array":…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,214 questions
asked 2023-02-03T17:44:09.69+00:00
Jered Willoughby 1 Reputation point
1 answer

Webhook activity for Python3 runbook execution

Hello, I created an Automation Account and Python3 runbook. Python script takes files from Blob storage and makes some transformations on them and after that new files are created in Blob storage. I am testing on simple Python script like…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-02T09:15:38.5266667+00:00
Maryna Paluyanava 86 Reputation points
answered 2023-02-03T16:23:27.1766667+00:00
ShaikMaheer-MSFT 28,106 Reputation points Microsoft Employee
0 answers

Issue with Dynamic Schema mapping in azure data factory for JSON Data

Hi Team, When I am doing dynamic schema mapping for Json data in ADF, that expression is showing me below error. Please let me know what I am missing here and that's the way I am doing. My Source has multiple fields and it is showing the same error for…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-03T15:57:29.35+00:00
pankaj chaturvedi 81 Reputation points
1 answer

[DataFactory] - Start/Stop Tumbling Triggers does not repect recurrence

Hello, I'm trying to make the following senario work without errors Have 1 pipeline that copies incremental data if table existes and if not it creates me that table. Have 2 tumbling trigger, 1 for running throughout the day and 1 to running at night…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-01T16:43:27.6466667+00:00
Gustavo Balbino 0 Reputation points
answered 2023-02-03T15:10:33.79+00:00
ShaikMaheer-MSFT 28,106 Reputation points Microsoft Employee
0 answers

ADF unable to call internal API via web activity with self-host Integration Runtime

Hi, I am working on a project with Azure Data Factory. We have setup an internal application and provided a API for updating record, and we would like to trigger a POST call by web activity in ADF (with Self host IR), but failed with the follow…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-03T14:55:20.9366667+00:00
Eric Kwan (HSL) 0 Reputation points
0 answers

Transfering .bak file from sftp_server to blob storage and storing as "PAGE BLOB", using Azure Data Factory

Hi, Im trying to move a ~3GB .bak file from an ftp_server to azure blob storage using sftp. I want the transfer to happen automatically once a day, as a new .bak file with the same name is placed at the FTP_server every night. I have looked into logical…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,406 questions
asked 2023-02-01T13:23:50.89+00:00
August Asheim Birkeland 20 Reputation points
commented 2023-02-03T14:51:22.8166667+00:00
August Asheim Birkeland 20 Reputation points
1 answer One of the answers was accepted by the question author.

Azure Data Factory IR cannot be stopped or deleted

I have created an IR in Azure Data Factory. The option to stop or delete the IR is missing. IR Type is Azure. Please advise on how I can proceed please.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
asked 2023-02-02T18:03:12.59+00:00
Michael Corcoran 20 Reputation points
commented 2023-02-03T14:29:33.59+00:00
Michael Corcoran 20 Reputation points
0 answers

Can Azure Data Factory connect to Azure MySql Flexible via private endpoints?

I have an Azure MySql Flexible instance, and an Azure Data Factory instance. Both resources are in the same subscription, connected to private endpoint on the same vnet (each in it's own /29 subnet). On my Data Factory instance, I followed the steps…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,303 questions
Azure Database for MySQL
Azure Database for MySQL
An Azure managed MySQL database service for app development and deployment.
440 questions
asked 2022-04-08T12:58:54.11+00:00
EJ Marmonti 106 Reputation points
commented 2023-02-03T14:27:09.32+00:00
Jammigumpula, Kalyan 0 Reputation points