Microsoft Q&A

Azure Data Factory

7,983 questions

An Azure service for ingesting, preparing, and transforming data at scale.

Browse all Azure tags

7,983 questions with Azure Data Factory tags

Sort by: Updated
1 answer

insert record from azure datafactory into a delta table generated in databricks

Hi team, i'm trying to insert value into a delta table from datafactory, the table has been created in databricks and in the table properties i setted "delta.enableChangeDataFeed=true", for this reason the protocol minWriterVersion of my delta…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,596 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2022-03-02T12:08:37.73+00:00
Antonio Nuzzo 11 Reputation points
commented 2023-09-24T02:03:35.55+00:00
Tim Frazer 0 Reputation points
1 answer

Generated ADF pipelibe using Python

Hello, I am working on the dynamic data pipeline framework were i can generate Azure data pipeline programatically. till now, i have done for creating copy activiyt and dataflow. the final step is, need to link between this two by creating another…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-19T11:17:33.5466667+00:00
CON-Thirusenthilkumar Pandiyan 5 Reputation points
edited a comment 2023-09-23T12:04:43.12+00:00
CON-Thirusenthilkumar Pandiyan 5 Reputation points
2 answers

Adding Azure Data Factory Managed Identity to Azure SQL DB Failes - Error Msg 33132

When trying to grant a Data Factory instance access to an Azure SQL DB I get the following error. "Msg 33132, Level 16, State 1, Line 1 This principal type is not supported in Azure SQL Database." Using this command - CREATE USER…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-02-15T02:06:09.79+00:00
Scott Cooke (Agile Analytics) 5 Reputation points
commented 2023-09-23T11:41:05.98+00:00
David EDWARDS 0 Reputation points
1 answer

how to fix -Your account is under review

Hi , I am facing below message while login Azure free Account. Your account is under review Kindly help met to get solved. Thanks in advance.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-17T14:22:05.4766667+00:00
mkeshwar 0 Reputation points
commented 2023-09-22T19:26:51.3933333+00:00
Rahul Randive 4,291 Reputation points Microsoft Employee
2 answers

Stored procedure with parameters not working in azure data factory

In my Azure Data Factory pipeline, I have an "if" activity. Within the "if" activity, I have a "stored procedure" activity that executes a delete statement and requires a parameter. In the ADF "stored procedure"…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-15T17:12:36.1233333+00:00
Deirdre Clark 0 Reputation points
commented 2023-09-22T18:24:02.87+00:00
Deirdre Clark 0 Reputation points
1 answer

Dataset works with pipeline but does not connect with dataflow

Hello, On azure data factory I created some datasets that work flawlessly on pipelines: The pipeline execution works as expected: And I get the data I need. But when I try to use that dataset in a dataflow I get a connection error: It says it could be…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2022-05-30T07:22:16.02+00:00
Sergi Arbós Mauri 6 Reputation points
commented 2023-09-22T17:40:26.02+00:00
Manish Kumar 0 Reputation points
0 answers

ADF Ingestion

What different ways you can ingest the data using ADF and Synapse

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,536 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-20T08:42:05.09+00:00
Anshal 1,201 Reputation points
commented 2023-09-22T17:26:52.07+00:00
BhargavaGunnam-MSFT 19,316 Reputation points Microsoft Employee
1 answer

How to connect to Azure PostgreSQL with AAD login, from ADF or Synapse pipeline?

Hello, I have a PostgreSQL (PG) database that exclusively permits Azure Active Directory (AAD) logins. I want to establish a linked service in Azure Synapse to import data from this database. If I obtain an authentication token from a user who belongs…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,536 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
Azure Database for PostgreSQL
asked 2023-09-22T14:57:51.6933333+00:00
Deba Nayak 41 Reputation points
edited a comment 2023-09-22T17:11:05.7066667+00:00
Amira Bedhiafi 4,071 Reputation points
1 answer

Data Flow using Temp File on SFTP by Default

I am creating a process that picks up a txt file based on "blob created" event trigger in ADF Data Flow. Once this txt file triggers, my data flow picks up the file translates and maps it to a CSV. On my sink I have it connected to an SFTP…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-22T15:10:16.6066667+00:00
Sasha Oelsner 0 Reputation points
commented 2023-09-22T16:23:00.8866667+00:00
Amira Bedhiafi 4,071 Reputation points
1 answer

How to bring through text readable fields (for lookup/choice fields in dataverse) when copying data using the azure synapse link or azure data factory

I have two situations where I copy data from Dataverse into other storage locations: using azure synapse link to copy data from dataverse into azure data lake storage using azure data factory copy activity to copy data from dataverse into azure sql…

Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
559 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,536 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-22T15:07:03.49+00:00
Hannah Chant 20 Reputation points
answered 2023-09-22T16:11:08.49+00:00
Amira Bedhiafi 4,071 Reputation points
0 answers

Azure Data Factory set variable add backslash before double quote.

In Azure Data Factory, I need to copy the output of a REST API, which is an array of nested JSONs, to a CSV file. I used a loop for the array and dynamic content for the copy activity's mapping. I have a Pipelines_Mappings variable, it contains a JSON…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-22T11:01:30.3233333+00:00
Mehrsa 0 Reputation points
commented 2023-09-22T15:38:11.86+00:00
Amira Bedhiafi 4,071 Reputation points
1 answer

ADF - Error at sink side - Rest API- Key Content-Type conflicts in AuthHeaders/AdditionalHeaders

Hello, I am getting error as below when I am trying to use the copy activity which as source as Json file from container and sink as Rest API. I have defined auth token in linked service . Content Type header is defined in additional headers in…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2022-09-26T11:59:52.21+00:00
Anurag Yadav 1 Reputation point
answered 2023-09-22T14:54:25.3266667+00:00
1 answer One of the answers was accepted by the question author.

How to clone/copy Azure Data Factory to another tenant?

My organization is moving to another Azure tenant. We have an Azure Data Factory with 50 pipelines, 40 datasets, 15 linked services and 5 triggers configured in the current tenant that needs to be available in the new tenant. I know that moving Azure…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-19T16:29:15.1666667+00:00
Pitawat 321 Reputation points
accepted 2023-09-22T13:42:24.6033333+00:00
Pitawat 321 Reputation points
1 answer

While copying multiple files in for each file, i am getting AzureStorageOperationFailedConcurrentWrite issue. Even though folder and files names are diffrent

Using look-up activity i am trying to read a file which contains the api data information(with folder structure) . Next I used for each activity to loop those records. In for each activity i have placed a copy activity to copy those files as per folder…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-22T10:30:23.79+00:00
Mohan Parashetti Veerabhadrappa 0 Reputation points
edited an answer 2023-09-22T12:50:34.4633333+00:00
Suba Balaji 8,386 Reputation points
1 answer

Unable to replace valid phone number

https://examregistration.microsoft.com/ I am unable to register the exam as it fails to update my valid number exists in USA "The Phone Number you provided is invalid, please provide a valid phone number and try again."

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,536 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,596 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,191 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-18T15:49:46.7433333+00:00
Divya Nagarajan 0 Reputation points
commented 2023-09-22T12:35:05.7566667+00:00
Suba Balaji 8,386 Reputation points
1 answer

How to create single dataset for multiple azure data flows?

I have about 50 files that i need to transform using a Azure Data Flows. Is there a way i can create a single data set for multiple data flows? I want to avoid creating a dataset for every single file.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-19T13:47:03.2733333+00:00
Pankaj Mandania 21 Reputation points
commented 2023-09-22T11:43:06.1133333+00:00
Harishga 90 Reputation points Microsoft Vendor
0 answers

Error when using Window activity in Azure Data Factory Data Flow

Hi, I am currently attempting to use the window activity in a data flow to recreate partition over order by. What I want is for the activity to rank rows with the same RECID and DML_Action by their LastProcessedChange_DateTime value, with the most recent…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
Azure Data Lake Analytics
asked 2023-09-22T11:38:00.53+00:00
Tobias Rognstad 0 Reputation points
1 answer

ADF -- Extract from SAP with RFC table option, filter on 2 or more columns

Hello Is there any way we can use multiple conditions on the RFC Table Option field of a copy activity? Something like this: Doing this on 1 column works. However having multiple conditions doesn't seem to work. I've seen that 'AND' operator or 'OR'…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-13T18:17:24.0766667+00:00
Yannick Seynhaeve 0 Reputation points
commented 2023-09-22T11:16:31.63+00:00
Smaran Thoomu 165 Reputation points Microsoft Vendor
0 answers

can we bulk copy and migrate mysql database tables using azure data factory

i know we can bulk copy SQL database tables using for each in azure data factory, but can we do the same for mysql database tables?, if it is possible please suggest some solutions.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
3,536 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-09-22T06:06:29.24+00:00
piyush pant 0 Reputation points
commented 2023-09-22T10:32:11.44+00:00
piyush pant 0 Reputation points
1 answer

Azure Data Factory - Copy activity using REST connector - Unable to access response headers

Hi team, I use ADF pipeline comprising copy activity to pull data from a remote endpoint using REST connector and then place the data on the blob. I do not have a need to use the pagination feature available with REST connector as the response is not…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,983 questions
asked 2023-07-31T03:06:55.41+00:00
Balachandran, Karthikeyan 0 Reputation points
commented 2023-09-22T09:40:19.0366667+00:00