Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,642 questions
0 answers
Getting "Found invalid data while decoding" error while running Copy Activity for REST API
Hi, I am trying to get data from REST API and trying to save it in CSV. However, each time i run the copy activity i am getting below error. Any help will be great appreciate it. This REST API required a Header value as below to get uncompressed…
asked 2024-05-09T08:22:18.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 1%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAY%3C/text%3E%3C/svg%3E)
Avnish Yadav
0
Reputation points
commented 2024-05-10T14:34:31.5333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,085
Reputation points Microsoft Vendor
0 answers
InsertDate and UpdateDate column addition in Sink-Delta from dataflow.
I am using dataflow to Upsert the data to my delta lake. My requirement is to have two columns - 1. insertdate 2.updatedate . If there is a new record then both insertdate and updatedate column should get updated with current_date() values. And, if…
0 answers
Connection issue with Oracle Autonomous Cloud (ADW) and Azure Data Factory
Hi, I am trying to connect Oracle Autonomous Database (ADW) with Azure Data Factory using self hosted integration runtime. After setting up Oracle DataDirect wire protocol 8.0 driver still i am unable to connect. i am getting this error below ERROR…
asked 2024-05-10T11:29:44.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 41%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Keshaw Bhardwaj
0
Reputation points
commented 2024-05-10T13:18:42.0266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
3,745
Reputation points Microsoft Vendor
2 answers
Recommended technologies for creating a dashboard of ongoing jobs.
I'm looking for recommendations on technologies to use for the following project. I need to create a dashboard of summarized job data from a network of hundreds of users. Our current application is used in the field for inputting and tracking job costs…
asked 2024-05-09T14:00:50.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 77%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECF%3C/text%3E%3C/svg%3E)
COW Farmer
0
Reputation points
answered 2024-05-10T12:24:35.2066667+00:00
COW Farmer
0
Reputation points
1 answer
how to fix task canceled error from Azure Data Factory, Pipeline
Hello The error below has occurred. Let me know what I'm gonna do. Error : 'Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,' A pipeline trigger was executed in Azure Data Factory. Thank you.
asked 2023-02-19T23:43:04.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 68%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Arthur Choi
5
Reputation points
edited a comment 2024-05-10T11:36:57.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 21%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESY%3C/text%3E%3C/svg%3E)
Shrayansh Yadav
0
Reputation points
2 answers
How to maintain same folder structure as source while sink the processed file
I have a requirement to process the JSON to parquet on daily basis. I have folder A,B,C needs to sink the file to another container with same structure as A,B,C for example if I'm processing a file from folder A it should sink to output container folder…
asked 2024-04-26T08:36:30.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 11%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Venkatesh Srinivasan
0
Reputation points
edited an answer 2024-05-10T11:31:43.6433333+00:00
Venkatesh Srinivasan
0
Reputation points
0 answers
Azure Data Factory Oracle to Parquet datatype mismatch
Hi Team, I have build a dynamic pipeline to load the data from Oracle tables to datalake as Parquet files. Now, when I am trying to copy the data from source to sink through CopyActivity, then I am not able to convert the decimal datatype of the…
asked 2024-05-09T10:06:20.1066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vaibhav
105
Reputation points
commented 2024-05-10T11:27:40.9033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
10,080
Reputation points Microsoft Vendor
0 answers
Cannot read excel file which is in using adls using load_workbook of openpyxl in databricks
Cannot read excel file which is in using load_workbook of openpyxl but can read if copied to dbfs
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 53%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Setup ADF authen with personal github
I have error when I try setup repo git integration with new credential it's not working. check in tab network it return 401 but I don't see any step input credential or token???? why it have token in here?? where it come from??
asked 2024-05-10T05:21:26.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 41%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Duy Nguyen
0
Reputation points
commented 2024-05-10T09:34:13.7066667+00:00
PRADEEPCHEEKATLA-MSFT
78,331
Reputation points Microsoft Employee
0 answers
Azure Data Factory pagination using QueryParameters and body field
Hello everyone, I am currently working with a pagination system and facing a challenge with the URL and parameter concatenation. I am trying to append a specific string to the URL retrieved from the first page of an API response. Here's the setup: API…
asked 2024-05-09T22:23:30.69+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 15%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Max Teitelbaum
0
Reputation points
edited a comment 2024-05-10T08:17:17.7566667+00:00
phemanth
6,085
Reputation points Microsoft Vendor
4 answers
CosmosDB Analytical store delete changes in sql sink
Hi I am trying to capture delete changes from Azure cosmosdb analytical store using change data capture in Azure datafactory, Source is Cosmosdb, sink is Azure sql database. In between I am flattening my file using dataflow. My insert, update is…
asked 2024-04-30T13:52:46.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 15%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kuldeep Bhati
0
Reputation points
edited a comment 2024-05-10T08:01:14.7766667+00:00
Harishga
3,745
Reputation points Microsoft Vendor
1 answer
ADF lost access to Dynamics CRM / Dataverse after MFA was turned on
Today my ADF pipelines started throwing errors: Operation on target Process order failed: ErrorCode=DynamicsFailedToConnect,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to connect to Dynamics: Unable to Login to…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 20%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E14%3C/text%3E%3C/svg%3E)
0 answers
Mapping Address Values to Relative URLs in Azure Data Factory
Hello, I am working in ADF to post data to Business Central Temp table. I have set up my source dataset as reading from blob to REST as sink. I am having this issue where I have to send invoices to the respective temp tables in each company in Business…
asked 2024-05-09T14:04:53.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 28.000000000000004%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKO%3C/text%3E%3C/svg%3E)
Kayode Ogidan
0
Reputation points
commented 2024-05-10T07:13:19.5+00:00
Harishga
3,745
Reputation points Microsoft Vendor
0 answers
The Integration Runtime (Self-hosted) node has encountered an error during registration.
I am not able to register Self Hosted Integration Runtime on my Machine. This is the error I am getting : The Integration Runtime (Self-hosted) node has encountered an error during registration. Unable to determine the Integration Runtime (Self-hosted)…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 36%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE6%3C/text%3E%3C/svg%3E)
1 answer
Error in Linked Service connected to azure sql db using Azure Key vault
The linked service is not working after performing CICD from dev to qa branch. All the resources are copied along with linked service. However, The linked service could not setup correctly with this error. What does this error indicate and how to…
asked 2024-05-09T15:16:14.0666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 71%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Sudip Bala
0
Reputation points
commented 2024-05-10T06:16:03.64+00:00
Sudip Bala
0
Reputation points
1 answer
Split single column data into multiple columns in datafactory
Issue: I have some csv files in sftp location . I was planning to use copyactivity to load files from sftp location to datalake for archiving as well as upserting in deltalake. However, I am getting error while reading from source, as some file has…
asked 2024-05-03T18:07:32.02+00:00
Vaibhav
105
Reputation points
commented 2024-05-10T05:15:16.5266667+00:00
Smaran Thoomu
10,080
Reputation points Microsoft Vendor
0 answers
Unable to resolve Azure database for mysql name from Data Factory
I am creating a linked service from my Azure Data Factory environment, to access my Azure Database for Mysql database. I select "Azure database for Mysql", and I can see all my mysql servers. I select "From Azure subscription" as the…
asked 2024-05-07T07:39:15.9766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 71%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIS%3C/text%3E%3C/svg%3E)
Ignacio Sales
5
Reputation points
commented 2024-05-10T03:19:30.3833333+00:00
PRADEEPCHEEKATLA-MSFT
78,331
Reputation points Microsoft Employee
1 answer
Error from ADF pipeline loading data into Dynamics 365 table
I have an Azure pipeline that is copying data from a table in an azure sql db to a table in Dynamics 365 CE. The D365 has MFA attached so I am using an azure service principal in the linked service connection for the pipeline. The Service principal is…
asked 2024-05-08T05:28:41.8833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 10%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWB%3C/text%3E%3C/svg%3E)
Wayne Bartkowski
656
Reputation points
answered 2024-05-09T22:54:54.8+00:00
Wayne Bartkowski
656
Reputation points
1 answer
How to Get REST API Data when return rows are more than 30,000 and store in one single csv
How to Get REST API Data when return rows are more than 30,000 and store in one single csv. I tried pagination rule with AbsoluteUrl to $.Response. How to get copy activity go through each record with offset and limit automatically? How to write all…
asked 2024-05-09T08:27:59.8866667+00:00
Avnish Yadav
0
Reputation points
answered 2024-05-09T21:55:14.2333333+00:00
Amira Bedhiafi
15,831
Reputation points
1 answer
How to parameterize a dataset so I can use multiple excel files in a blob folder to pass to PowerQuery one by one?
I am in Data factory and I need help to parameterize a dataset so I can use multiple files in a blob folder to process in PowerQuery. I do not want to make 100 datasets if I want to process the 100 excel files in the same way. I have successfully…
asked 2024-05-09T08:24:14.3866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 2%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
Mayank Dhami
0
Reputation points
answered 2024-05-09T20:21:08.5066667+00:00
Amira Bedhiafi
15,831
Reputation points