Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,672 questions
Content
0 answers
How to get data for all the fields listed there in CRM using Azure data factory
For getting full load data from the CRM database to the Azure SQL database, I am using the table option in the copy source and getting the data for all the columns. But when I am trying to get incremental data using Query option for which I am using…
asked 2023-02-02T19:45:47.8566667+00:00
Jayasree Gade
0
Reputation points
commented 2023-03-27T01:08:36.6+00:00
Ryan Abbey
1,081
Reputation points
1 answer
Azure Data Factory json manipulation
Hi, Good day. We have an input JSON that is of the following schema: { "result": [ { "table": "test1", "field": "completed_on", "sys_id": "7d6a0a84dc0", …
0 answers
It must be software level, energy saving is powerful antivirus
Tell me how to keep energy saving always enabled on Desktop without battery that also means laptops.
0 answers
Custom Activity runs for hours and never completes in ADF (trying to run a sample python script)
Hello, I am trying to run python script in a custom activity as shown in this article. https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory Followed it word by word and trying creating pools multiple times and ran…
0 answers
Iam receiving json data from restapi link, with multiple arrays how to pull data into SQL db
@ShaikMaheer-MSFT data structure like this { } Json. [ ] Links { } 0 { } 1 [ ] Results{} Entity - under again [] links
asked 2023-03-24T12:22:56.7433333+00:00
Venki Bimma
0
Reputation points
commented 2023-03-26T20:16:01.2633333+00:00
Ronen Ariely
14,141
Reputation points MVP
1 answer
Copy data activity import CSV issue when Line Feeds are mixed in column data and conflict with end of file delimiter CR LF
You cannot set the value to \r\n as a distinct option which doesn't make sense considering it is a standard row delimiter on most text extracts, forcing it is not an option nor can you create a parameter to set it manually. The error generated when you…
asked 2023-03-01T03:41:21.4733333+00:00
Sam Miller
0
Reputation points
commented 2023-03-26T01:46:49.54+00:00
Sam Miller
0
Reputation points
2 answers
How writing partitioned parquet files from a csv file
Hi, I need to know how I can write partitioned parquet files into an Azure Blob storage reading a single csv file. I'd like to partition by year, creating year folders and putting the year partitioned files in the corresponding folder. Any helps to me,…
1 answer
ADF-SAP Table connector Memory Error for SAP ECC Source
Hello, I am trying to retrieve the data from SAP ECC-ERP 6.0 on Netweaver 7.5 using ADF-SAP Table connector, I can access the SAP tables using Preview Data option, however while trying to ingest the data to ADLS getting the below error: Could you please…
asked 2023-03-25T12:58:11.03+00:00
Tyagi, Amit
0
Reputation points
answered 2023-03-25T13:34:17.34+00:00
Erkan Sahin
470
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory | Web Activity
Hi I want to get a token from graph api using the Web Activity The token request is a multipart/form data type I am unable to send data in this format Let me know the steps for this We are using graph api from documentation
asked 2023-03-24T06:42:20.22+00:00
Abhay Chandramouli
641
Reputation points
accepted 2023-03-25T01:19:51.1866667+00:00
Abhay Chandramouli
641
Reputation points
0 answers
Upsert options not showing in Copy Acivity after initially creating pipeline. Settings are still in JSON?
I am not getting an error, but I think this might be UI bug? After initially creating Upsert and adding keys, those options no longer show up in UI. The settings are in the JSON, but not are not displayed in UI. Anybody else see this behavior? My…
asked 2023-03-24T09:58:26.3466667+00:00
Derek Horrall
141
Reputation points
commented 2023-03-25T00:53:11.9433333+00:00
MartinJaffer-MSFT
25,556
Reputation points Microsoft Employee
1 answer
ADF Copy Data REST API Pagination not ending on condition
Hi there, I am trying to fix my endCondition on my paginated copy date REST api activity. My API only returns the current page and the total pages. So I tried to add a endCondition of... EndCondition:$['page']Const$['total'] aka. end if current page is…
asked 2023-03-22T12:32:54.4533333+00:00
Jon
0
Reputation points
answered 2023-03-25T00:30:18.93+00:00
KranthiPakala-MSFT
35,597
Reputation points Microsoft Employee
1 answer
Enable Interactive Authoring Under Managed Virtual Network in Azure ADF pipeline
Enable Interactive Authoring Under Managed Virtual Network in Azure ADF pipeline. I have a requirement wherein I have to enable the interactive authoring option under Managed Virtual Network in Azure ADF pipeline via AzureCLI/Azure Powershell or any…
asked 2023-03-07T05:11:01.63+00:00
Dinesh Prajapati
86
Reputation points
answered 2023-03-24T21:16:27.4633333+00:00
MartinJaffer-MSFT
25,556
Reputation points Microsoft Employee
2 answers
How to copy data from Azure SQL Database to SAP system using Azure Data Factory?
I am looking for a way to copy data from Azure SQL Database table to SAP system using Azure Data Factory pipeline (Push model). Is this possible? When checked in linked services I cannot find SAP connectors in SINK DATASET. Unfortunately I am unable to…
asked 2023-03-14T07:13:48.22+00:00
Atharva Jadhav
0
Reputation points
commented 2023-03-24T21:11:13.66+00:00
MartinJaffer-MSFT
25,556
Reputation points Microsoft Employee
1 answer
What is the best way to copy all files from a sharepoint site to Azure blobs using data factory?
I have followed a few tutorials but it either seems to not work or it only copies one file. I don't have the paid version of Power Automate that gives me the premium items. I was under the impression that I can copy multiple files to an Azure Storage…
asked 2023-03-20T19:01:18.8733333+00:00
Brett Johnson
21
Reputation points
commented 2023-03-24T21:08:25.3166667+00:00
MartinJaffer-MSFT
25,556
Reputation points Microsoft Employee
0 answers
How to bearer access token generate in data flows as a source transformation
How to generate bearer access token generate in data flows as a source transformation
asked 2023-03-20T13:39:44.6266667+00:00
Sai
0
Reputation points
commented 2023-03-24T20:52:34.6+00:00
MartinJaffer-MSFT
25,556
Reputation points Microsoft Employee
0 answers
Can't import Live resources to GitHub Repo
I have an existing GitHub repository and I want to import resources from my ADF Live environment to the repo. I can connect to it fine and browse the repos in my account, but when I click Import I get this error. You don't have permission to import…
asked 2023-03-23T14:15:46.97+00:00
Mark Allison
126
Reputation points
commented 2023-03-24T20:45:06.1733333+00:00
MartinJaffer-MSFT
25,556
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Schema drift not working on Unpivot activity/transformation on Azure Data Factory
Hi, I'm trying to use the "Unpivot" transformation on a Source that doesn't have schema pre-defined (Schema Drift is activated). The "unpivot" transformation generates the following error: at Unpivot 'UnpivotForPlaysOrViews': web |…
asked 2023-03-21T00:07:08.05+00:00
Vivek Komarla Bhaskar
521
Reputation points
commented 2023-03-24T20:31:03.8933333+00:00
KranthiPakala-MSFT
35,597
Reputation points Microsoft Employee
1 answer
ADF load data to Rest API Sink
My data is present in Azure SQL Database table, and I have to PUT the data in Blackline API in text format, using ADF Blackline API example : https://developer.blackline.com/docs/data-ingestion-accounts-api/1/routes/dataingestion/accounts/put URL :…
asked 2023-03-23T18:02:43.38+00:00
Arundhati Sen
101
Reputation points
edited a comment 2023-03-24T20:26:13.3+00:00
Arundhati Sen
101
Reputation points
1 answer
DateTime source values showing as NULL in Azure Data Factory - Data Flows
Hi, Is there a reason why my data flow cannot read/treat the DateTime input column? In my source file, I have a column named 'Date' with DateTime values of format 'yyyy-MM-dd HH:mm:ss', but the data preview always shows NULL values. Hence, to fix this…
asked 2023-03-22T23:15:04.3233333+00:00
Vivek Komarla Bhaskar
521
Reputation points
commented 2023-03-24T19:48:42.3033333+00:00
Vivek Komarla Bhaskar
521
Reputation points
1 answer
One of the answers was accepted by the question author.
Rule based mapping - Azure data factory
Hi, I am trying to use rule-based mapping on my input schema, which is a CSV file. For some reason, I can't achieve what I want. This is the screenshot of my input schema, as you can see I have columns that contain %Concurrent% & %Plays%. What I'm…
asked 2023-03-20T11:13:31.64+00:00
Vivek Komarla Bhaskar
521
Reputation points
accepted 2023-03-24T19:45:48.42+00:00
Vivek Komarla Bhaskar
521
Reputation points