Microsoft Q&A

Azure Data Factory

6,672 questions

An Azure service for ingesting, preparing, and transforming data at scale.

Browse all Azure tags

6,672 questions with Azure Data Factory tags

Sort by: Updated
0 answers

How to get data for all the fields listed there in CRM using Azure data factory

For getting full load data from the CRM database to the Azure SQL database, I am using the table option in the copy source and getting the data for all the columns. But when I am trying to get incremental data using Query option for which I am using…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-02-02T19:45:47.8566667+00:00
Jayasree Gade 0 Reputation points
commented 2023-03-27T01:08:36.6+00:00
Ryan Abbey 1,081 Reputation points
1 answer

Azure Data Factory json manipulation

Hi, Good day. We have an input JSON that is of the following schema: { "result": [ { "table": "test1", "field": "completed_on", "sys_id": "7d6a0a84dc0", …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-23T06:50:11.99+00:00
MS Prog 391 Reputation points
answered 2023-03-27T00:20:09.94+00:00
MS Prog 391 Reputation points
0 answers

It must be software level, energy saving is powerful antivirus

Tell me how to keep energy saving always enabled on Desktop without battery that also means laptops.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
Windows 10
Windows 10
A Microsoft operating system that runs on personal computers and tablets.
6,675 questions
Windows 11
Windows 11
A Microsoft operating system designed for productivity, creativity, and ease of use.
4,093 questions
asked 2023-03-26T21:31:44.4466667+00:00
Nick B 241 Reputation points
commented 2023-03-26T21:33:39.6+00:00
Nick B 241 Reputation points
0 answers

Custom Activity runs for hours and never completes in ADF (trying to run a sample python script)

Hello, I am trying to run python script in a custom activity as shown in this article. https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory Followed it word by word and trying creating pools multiple times and ran…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-26T01:07:21.01+00:00
N2120 41 Reputation points
edited a comment 2023-03-26T20:47:55.5933333+00:00
N2120 41 Reputation points
0 answers

Iam receiving json data from restapi link, with multiple arrays how to pull data into SQL db

@ShaikMaheer-MSFT data structure like this { } Json. [ ] Links { } 0 { } 1 [ ] Results{} Entity - under again [] links

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-24T12:22:56.7433333+00:00
Venki Bimma 0 Reputation points
commented 2023-03-26T20:16:01.2633333+00:00
Ronen Ariely 14,141 Reputation points MVP
1 answer

Copy data activity import CSV issue when Line Feeds are mixed in column data and conflict with end of file delimiter CR LF

You cannot set the value to \r\n as a distinct option which doesn't make sense considering it is a standard row delimiter on most text extracts, forcing it is not an option nor can you create a parameter to set it manually. The error generated when you…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-01T03:41:21.4733333+00:00
Sam Miller 0 Reputation points
commented 2023-03-26T01:46:49.54+00:00
Sam Miller 0 Reputation points
2 answers

How writing partitioned parquet files from a csv file

Hi, I need to know how I can write partitioned parquet files into an Azure Blob storage reading a single csv file. I'd like to partition by year, creating year folders and putting the year partitioned files in the corresponding folder. Any helps to me,…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-24T22:21:07.8466667+00:00
pmscorca 186 Reputation points
edited a comment 2023-03-25T15:40:22.34+00:00
pmscorca 186 Reputation points
1 answer

ADF-SAP Table connector Memory Error for SAP ECC Source

Hello, I am trying to retrieve the data from SAP ECC-ERP 6.0 on Netweaver 7.5 using ADF-SAP Table connector, I can access the SAP tables using Preview Data option, however while trying to ingest the data to ADLS getting the below error: Could you please…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-25T12:58:11.03+00:00
Tyagi, Amit 0 Reputation points
answered 2023-03-25T13:34:17.34+00:00
Erkan Sahin 470 Reputation points
1 answer One of the answers was accepted by the question author.

Azure Data Factory | Web Activity

Hi I want to get a token from graph api using the Web Activity The token request is a multipart/form data type I am unable to send data in this format Let me know the steps for this We are using graph api from documentation

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-24T06:42:20.22+00:00
Abhay Chandramouli 641 Reputation points
accepted 2023-03-25T01:19:51.1866667+00:00
Abhay Chandramouli 641 Reputation points
0 answers

Upsert options not showing in Copy Acivity after initially creating pipeline. Settings are still in JSON?

I am not getting an error, but I think this might be UI bug? After initially creating Upsert and adding keys, those options no longer show up in UI. The settings are in the JSON, but not are not displayed in UI. Anybody else see this behavior? My…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-24T09:58:26.3466667+00:00
Derek Horrall 141 Reputation points
commented 2023-03-25T00:53:11.9433333+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
1 answer

ADF Copy Data REST API Pagination not ending on condition

Hi there, I am trying to fix my endCondition on my paginated copy date REST api activity. My API only returns the current page and the total pages. So I tried to add a endCondition of... EndCondition:$['page']Const$['total'] aka. end if current page is…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-22T12:32:54.4533333+00:00
Jon 0 Reputation points
answered 2023-03-25T00:30:18.93+00:00
KranthiPakala-MSFT 35,597 Reputation points Microsoft Employee
1 answer

Enable Interactive Authoring Under Managed Virtual Network in Azure ADF pipeline

Enable Interactive Authoring Under Managed Virtual Network in Azure ADF pipeline. I have a requirement wherein I have to enable the interactive authoring option under Managed Virtual Network in Azure ADF pipeline via AzureCLI/Azure Powershell or any…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-07T05:11:01.63+00:00
Dinesh Prajapati 86 Reputation points
answered 2023-03-24T21:16:27.4633333+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
2 answers

How to copy data from Azure SQL Database to SAP system using Azure Data Factory?

I am looking for a way to copy data from Azure SQL Database table to SAP system using Azure Data Factory pipeline (Push model). Is this possible? When checked in linked services I cannot find SAP connectors in SINK DATASET. Unfortunately I am unable to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-14T07:13:48.22+00:00
Atharva Jadhav 0 Reputation points
commented 2023-03-24T21:11:13.66+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
1 answer

What is the best way to copy all files from a sharepoint site to Azure blobs using data factory?

I have followed a few tutorials but it either seems to not work or it only copies one file. I don't have the paid version of Power Automate that gives me the premium items. I was under the impression that I can copy multiple files to an Azure Storage…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
6,457 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,530 questions
asked 2023-03-20T19:01:18.8733333+00:00
Brett Johnson 21 Reputation points
commented 2023-03-24T21:08:25.3166667+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
0 answers

How to bearer access token generate in data flows as a source transformation

How to generate bearer access token generate in data flows as a source transformation

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-20T13:39:44.6266667+00:00
Sai 0 Reputation points
commented 2023-03-24T20:52:34.6+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
0 answers

Can't import Live resources to GitHub Repo

I have an existing GitHub repository and I want to import resources from my ADF Live environment to the repo. I can connect to it fine and browse the repos in my account, but when I click Import I get this error. You don't have permission to import…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-23T14:15:46.97+00:00
Mark Allison 126 Reputation points
commented 2023-03-24T20:45:06.1733333+00:00
MartinJaffer-MSFT 25,556 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Schema drift not working on Unpivot activity/transformation on Azure Data Factory

Hi, I'm trying to use the "Unpivot" transformation on a Source that doesn't have schema pre-defined (Schema Drift is activated). The "unpivot" transformation generates the following error: at Unpivot 'UnpivotForPlaysOrViews': web |…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-21T00:07:08.05+00:00
Vivek Komarla Bhaskar 521 Reputation points
commented 2023-03-24T20:31:03.8933333+00:00
KranthiPakala-MSFT 35,597 Reputation points Microsoft Employee
1 answer

ADF load data to Rest API Sink

My data is present in Azure SQL Database table, and I have to PUT the data in Blackline API in text format, using ADF Blackline API example : https://developer.blackline.com/docs/data-ingestion-accounts-api/1/routes/dataingestion/accounts/put URL :…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
Azure SQL Database
asked 2023-03-23T18:02:43.38+00:00
Arundhati Sen 101 Reputation points
edited a comment 2023-03-24T20:26:13.3+00:00
Arundhati Sen 101 Reputation points
1 answer

DateTime source values showing as NULL in Azure Data Factory - Data Flows

Hi, Is there a reason why my data flow cannot read/treat the DateTime input column? In my source file, I have a column named 'Date' with DateTime values of format 'yyyy-MM-dd HH:mm:ss', but the data preview always shows NULL values. Hence, to fix this…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-22T23:15:04.3233333+00:00
Vivek Komarla Bhaskar 521 Reputation points
commented 2023-03-24T19:48:42.3033333+00:00
Vivek Komarla Bhaskar 521 Reputation points
1 answer One of the answers was accepted by the question author.

Rule based mapping - Azure data factory

Hi, I am trying to use rule-based mapping on my input schema, which is a CSV file. For some reason, I can't achieve what I want. This is the screenshot of my input schema, as you can see I have columns that contain %Concurrent% & %Plays%. What I'm…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
asked 2023-03-20T11:13:31.64+00:00
Vivek Komarla Bhaskar 521 Reputation points
accepted 2023-03-24T19:45:48.42+00:00
Vivek Komarla Bhaskar 521 Reputation points