Microsoft Q&A

Azure Data Factory

6,261 questions

An Azure service for ingesting, preparing, and transforming data at scale.

Browse all Azure tags

6,261 questions with Azure Data Factory tags

Sort by: Updated
1 answer

Data Factory: turn on data flow debug error

Hello! I need the "Data Preview" for checking, and it require me to turn on "Data flow debug". However, error received and no ErrorCode. I can't do anything. Data flow debug Failed to setup debug session. Error: An error…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2021-12-12T16:55:00.157+00:00
Ken 1 Reputation point
commented 2023-01-29T03:41:34.73+00:00
Ali Omran 0 Reputation points
0 answers

Long running Databricks notebook but cumulative processing time is short

Hi Microsoft experts, I have a long running Databricks notebook tonight, and I can't see a reason why. My Data Factory pipeline which calls the notebook has been running reliably everyday for about 8 months, but this week it has had some problems and now…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,199 questions
asked 2023-01-13T09:29:24.8366667+00:00
James Hooi 0 Reputation points
commented 2023-01-29T03:12:39.82+00:00
James Hooi 0 Reputation points
0 answers

How the pass a session ID for a SOAP API in ADF

Hi all, Struggling for a while now with creating an API request in ADF to load data from a SOAP API. I'm having difficulties with passing a session ID to another API request which will provide the transactional data. step 1: API request (OpenSession)…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure API Management
Azure API Management
An Azure service that provides a hybrid, multi-cloud management platform for APIs.
1,075 questions
asked 2023-01-28T22:42:24.6466667+00:00
Remon van Ramele 0 Reputation points
1 answer

Merging 2 Array variables

Hello All. I might be missing some simple aspect here but wanted to confirm : Is there any way in ADF to merge 2 Array variables without the use of any iteration activity. Like ask is as below: Input1= [1,2,3] Input2=[1,4,5] Output=[1,2,3,1,4,5] Note:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-28T07:35:35.7433333+00:00
Nandan Hegde 21,606 Reputation points
answered 2023-01-28T20:38:46.8233333+00:00
Dillon Silzer 33,176 Reputation points
0 answers

change column name in delta parquet file

hello, data fatory data flow is being used to upsert into delta parquet files. The delta parquet files have several columns. How is it possible to change the name of a column without affecting the data using the delta sink in the data flow? for…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-28T16:25:23.9866667+00:00
arkiboys 7,861 Reputation points
1 answer

Your account or password is incorrect. If you don't remember your password, reset it now.

Hello! When I try to enter incorrect password, I give this error: Your account or password is incorrect. If you don't remember your password, reset it now.also when I try to click on reset now, I give this message: Verify your identity How would you like…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
1,540 questions
Azure Active Directory
Azure Active Directory
An Azure enterprise identity service that provides single sign-on and multi-factor authentication.
12,575 questions
Exchange Server Management
Exchange Server Management
Exchange Server: A family of Microsoft client/server messaging and collaboration software.Management: The act or process of organizing, handling, directing or controlling something.
6,074 questions
asked 2023-01-28T15:58:43.3833333+00:00
Anthony Hello 0 Reputation points
answered 2023-01-28T16:20:52.7966667+00:00
Dave Patrick 328.6K Reputation points Microsoft MVP
0 answers

Change Data Capture in Azure Data factory High level resources does not ask for DelimitedText value, instead it takes comma(,) by default

Hi I am trying to implement CDC High Level Resources through Blob to SQL DB. But my mapping was incorrect as my source is delimited by "|", due to which I am unable to implement. Can we look into this? on how to change default comma(,) to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-26T23:04:18.0766667+00:00
Krishnamohan Nadimpalli 386 Reputation points
commented 2023-01-28T13:14:38.62+00:00
Krishnamohan Nadimpalli 386 Reputation points
0 answers

Can we convert R script to M code (Power Query)

As we are having few scripts in R which we are consuming in another scope, as per the new scope that needs be migrated to M (Power Query). If so, are there any best practices to migrate existing scripts to M code. Do we have any tools that can be used…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
Azure Analysis Services
Azure Analysis Services
An Azure service that provides an enterprise-grade analytics engine.
337 questions
asked 2023-01-27T12:38:18.8+00:00
Manohar Kondragunta 0 Reputation points
edited the question 2023-01-28T04:47:39+00:00
AjayBathini-MSFT 0 Reputation points Microsoft Employee
0 answers

ADF copy data to cosmosdb producing errors

I created a copydata pipline in azure data factory to move data from a blob to cosmos mongo api database. All necessary dataset connections connected successfully .But while i debug or execute i get the errors below relating to the sink. Is there a way…

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
908 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-23T02:01:56.39+00:00
commented 2023-01-28T04:17:39.3433333+00:00
BhargavaGunnam-MSFT 7,311 Reputation points Microsoft Employee
0 answers

pivot hour to minutes workflow

I need to pivot each row in a table as follows. From: To: Each hour of the day represented from h1 to h24 listing total minutes worked minus break. What would be the best workflow for this process?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
2,577 questions
asked 2023-01-21T17:49:58.8666667+00:00
Petar Novakovic 21 Reputation points
edited the question 2023-01-27T22:49:17.8866667+00:00
Petar Novakovic 21 Reputation points
1 answer

Where is the storage account key for cosmos db

I have a copy data pipeline which loads data from a json file within a blob to a cosmos DB collection. Now the sink is the cosmos dB connection . At creation time account selection method is Azure subscription , subscription is selected and then Azure…

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
908 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-27T14:46:21.0766667+00:00
answered 2023-01-27T22:35:58.55+00:00
Dillon Silzer 33,176 Reputation points
0 answers

Airtable API Patch Request Fails in ADF

I'm trying to update data in Airtable using Rest API in ADF. I get an error. I'm trying to execute a PATCH. Here's my mapping. Example request from Airtable API documentation: Why is this not working? Error Message: Response:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-27T22:23:38.65+00:00
Jon Carlisle 66 Reputation points
0 answers

Upsert to Azure SQL Database Failed because Interim Table Exists

I have created a Copy Activity on Azure Data Factory that moves the data from a CSV file hosted on Data Lake to Azure SQL Database. When I select the write behavior to be INSERT, the pipeline runs fine. But when I changed to UPSERT and provide the proper…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure SQL Database
asked 2023-01-27T22:07:31.5066667+00:00
Su Wang 0 Reputation points
1 answer One of the answers was accepted by the question author.

How to update data into Azure storage table from datafactory?

1) Can i use storage table for capture log details of pipeline via datafactory. will it allow insert new values in table? 2) I want to update specific column in storage table. how do i update the column which activity I should use? like I want to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
891 questions
asked 2020-10-13T12:07:05.143+00:00
THIMMAIAH GARI,PRASHANTH,, 191 Reputation points
commented 2023-01-27T17:43:38.41+00:00
Mehdi Nikkhah 0 Reputation points
0 answers

Error to set lookup field in N:1 relationship while writing data to Dynamics CRM from Azure Data Factory

While trying to sync data from sql staging table to Account entity in CRM,  I faced an issue in populating the multi-lookup column. The input column is Manager Name that is of string data type. The destination field is dfws_sareportingmanager (GUID data…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-25T21:03:04.89+00:00
AnaC 0 Reputation points
commented 2023-01-27T17:40:38.6833333+00:00
Pratik Somaiya 4,026 Reputation points
1 answer

Azure Storage explorer stuck on Loading.. any suggestion

i am facing this problem on my intel arc gaming pc running on windows 11 rightnow. when i Try to connect to blob storage in Azure Storage Emulator its stuck on Loading.. Was working last week before I did a windows update and storage explorer update. I…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
Azure Active Directory
Azure Active Directory
An Azure enterprise identity service that provides single sign-on and multi-factor authentication.
12,575 questions
asked 2023-01-27T08:07:38.8933333+00:00
AARON JEFFERY 0 Reputation points
answered 2023-01-27T16:02:55.7933333+00:00
Marshaljs 26,191 Reputation points
0 answers

Problem with ADF Dataflow pipeline when writing data to SQL Server

I have an ADF Dataflow pipeline that reads data from a data lake and writes it to a SQL Server table. I have a problematic column called "gdtxft" that I converted from binary to string using the 'toString' function in the Derived column…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-25T10:06:58.0633333+00:00
commented 2023-01-27T15:55:32.6633333+00:00
Subashri Vasudevan 4,296 Reputation points
1 answer

Does Interactive authoring of the integration runtime have to be activated manually?

Hello! I just wanted to know if there's a way to automate the enabling of the interactive authoring of the AutoResolveIntegrationRuntime. Thanks!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
SQL Server Integration Services
SQL Server Integration Services
A Microsoft platform for building enterprise-level data integration and data transformations solutions.
1,840 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,600 questions
asked 2022-06-29T16:07:00.26+00:00
commented 2023-01-27T15:53:05.7033333+00:00
Marc Scirri 1 Reputation point
1 answer

How to convert data row into valid json in ADF Dataflow

I have data in tabular format. As part of dataflow transformation want add another derived column like below. Basically converting column data into valid Json string

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-27T10:48:56.6433333+00:00
Vivek, Anand 0 Reputation points
edited an answer 2023-01-27T14:47:38.1833333+00:00
AnnuKumari-MSFT 19,116 Reputation points Microsoft Employee
0 answers

Very Old System: Failed to load SOAP Service definition file.

defunct quetion, posted it before editing, will do another to fix it

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,261 questions
asked 2023-01-25T13:12:45.45+00:00
Bing sling 0 Reputation points
commented 2023-01-27T14:41:01.3633333+00:00
BhargavaGunnam-MSFT 7,311 Reputation points Microsoft Employee