Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,740 questions
0 answers
Unable to connect to Epicor CMS DB2 with Azure Data Factory
Hello, I am moving from an Azure VM(with SQL Server) to an Azure SQL server. My source systems are varied including SQL server, SAP, Epicor CMS DB2, etc. Using Data Factory I have no problem connecting to any of my source systems that are on SQL…
asked 2022-04-13T13:35:23.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 61%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Robert Perry
1
Reputation point
commented 2022-04-22T21:29:58.053+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,761
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
dataflow - no sink
dataflow1 returns fields in a select activity (Field1 which has multiple distinct values) and so there is no sink dataflow2 expects is to loop through each of the values from dataflow1 Here is what I have but I get an error when trying to debug: …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
alter row transformation
Hello, In dataflow of the ADF, I am using alter row transformation before the sink which is set to use inline sinktype and Delta dataset type. My goal is for the delta files to have delete as well as upsert. In sink key columns I have placed…
1 answer
One of the answers was accepted by the question author.
ADF completed notification
Is there a video to teach me how to have a final activity in ADF to send a message to external apps to indicate that the ADF has completed its task and succeeded or failed? Thank you
1 answer
One of the answers was accepted by the question author.
subscribing to completion of ADF
How is it possible for the external applications to be notified that my ADF has completed its work? Is this related to event grid? Do the external applications have to subscribe to something? Thanks
0 answers
Source HTTP response is greater than 4 MB for copy activity
Hi , I am working on a pipeline which contains copy activity to copy data from HTTP response to Azure SQL database. I am getting the response and the activity is completed successfully but no rows written to the SQL server table. After doing google…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 81%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How to read image from azure blob storage as a byte format
Please help me here in order to achieve the below solution? I have a API for posting image. API accept byte as input. in this case, i need to read a image from azure blob storage and convert that image to byte and then post that byte value to API…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 38%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
ability to see column names on compressed files (.csv.gz)
Hi, We have a simple data flow whereby we can copy data from compressed csv files (.csv.gz) and load into a sink in azure and when use auto-mapping works fine. However, when we say need to translate a column from string to date and therefore want to…
asked 2022-04-19T15:17:26.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 22%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPO%3C/text%3E%3C/svg%3E)
Philip O'Rourke
111
Reputation points
commented 2022-04-22T08:21:58.797+00:00
Philip O'Rourke
111
Reputation points
1 answer
One of the answers was accepted by the question author.
Missing timestamp using inline Common Data Model dataset of ADLS gen2
I am using inline dataset to read data from CDM in dataflow. It is strange that I found the CREATEDDATETIME and MODIFIEDDATETIME are missing value, while other fields look ok. The source data is from D365 F&O which we exported by using "Export…
asked 2022-04-21T01:50:58.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 82%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Achates Chen
21
Reputation points
commented 2022-04-22T07:29:40.46+00:00
Achates Chen
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory CI/CD release is generating empty csv from data flow activity pipeline
Hi, I have tried to create automated CI/CD ADF deployment. So I have created 3 data factories 1)dev-df 2)uat-df 3)prod-df and In dev-df I have created managed IR, managed private endpoints, linked services, data sets, data flow and pipeline with data…
asked 2022-04-20T06:25:21.757+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 86%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Dhanush Narayanan
21
Reputation points
commented 2022-04-22T07:09:48.297+00:00
ShaikMaheer-MSFT
38,126
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure logic apps to send the failure ADF pipelines email notifications
How much will it cost to use using Azure logic apps to send the failure ADF pipelines email notifications?
asked 2022-04-11T08:59:59.183+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 7.000000000000001%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJP%3C/text%3E%3C/svg%3E)
Jagadeesh Pinninti
21
Reputation points
accepted 2022-04-22T04:21:10.793+00:00
Jagadeesh Pinninti
21
Reputation points
1 answer
Azure Managed VNET Integration Runtime pricing not availabe for Central US Region
We have our data center in central US region. But when we try to create Azure data factory Azure Managed VNET Integration Runtime doesn't have pricing for central US. Does this mean that we cannot create ADF in Central US? If Yes, is it worth to…
asked 2022-04-12T17:05:16.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 44%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVV%3C/text%3E%3C/svg%3E)
Venkat Velicheti
1
Reputation point
commented 2022-04-22T03:55:44.583+00:00
PRADEEPCHEEKATLA-MSFT
79,381
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
loop in ADF
Hello, in my ADF, There is a lookup activity which reads a column from a csv file. After the lookup, there is a loop activity which loops through each of the records in the csv. Inside the loop activity there is a copy activity which calls an api…
1 answer
One of the answers was accepted by the question author.
parallel processing
There is a url which accepts ids, i.e. https://someaddress/id in {@Anonymous } example1: https://someaddress/id in {'vxyz'} https://someaddress/id in {'ITYG'} https://someaddress/id in {'FTRT'} https://someaddress/id in {'OOOO'} …
1 answer
One of the answers was accepted by the question author.
not pulling data from .json in dataflow source data preview
in the dataflow, dataset of source shows data inside the source data preview, now rows are shown. only column names are shown. Any suggestions? Thanks
6 answers
Post file from Blob to Web Activity in Azure Data Factory
Hi, I have been working on this problem for days, weeks and I can't seem to get it right. What I am trying to do is send a file from a blob storage to a website using Web activity on ADF. Attached is what I have so far. I can't figure…
asked 2020-11-18T18:07:29.137+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 46%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED2%3C/text%3E%3C/svg%3E)
DannyV-2978
6
Reputation points
commented 2022-04-21T18:17:11.817+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 10%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJE%3C/text%3E%3C/svg%3E)
Jonathan Espindola (MINDTREE LIMITED)
1
Reputation point
1 answer
How to run a Python script from ADF?
I want to execute a Python script which will extract data from pdf stored in Azure blob storage using tabula and save the extracted data into a excel/csv file and upload the excel/csv file back to the blob storage from ADF.
asked 2022-03-25T07:22:33.183+00:00
G Shivani Murthy
26
Reputation points
commented 2022-04-21T11:02:53.357+00:00
G Shivani Murthy
26
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure data factory - Copy filterred bulk data from one container to another container in Azure blob storage
Hello, I would like to create Azure data factory pipeline to copy the filtered bulk data from one container to another container in the same Azure blob storage. Here are the high level details: In Excel file one of the column contains list of…
asked 2022-04-07T20:12:27.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 37%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKD%3C/text%3E%3C/svg%3E)
Kumar, Dinesh
26
Reputation points
commented 2022-04-21T10:21:19.427+00:00
AnnuKumari-MSFT
31,721
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
first date of the month - dataflow
In pipeline I want to pass in dates to dataflow, I use this to get the current date --> @convertTimeZone(utcNow(),'UTC','GMT Standard Time') How do I get the first DATE of the current month? example: 2022-04-01 Thank you
2 answers
One of the answers was accepted by the question author.
Want to store .sql files to azure blob using adf
Hi , I am using third party tool which exports .sql scripts daily . I want to store those .sql scripts in azure blob (if possible without using azure mysql db instance) using adf in the form of tables and then i will use those tables for analytical…
asked 2022-04-14T06:16:34.01+00:00
Amar Agnihotri
916
Reputation points
commented 2022-04-21T06:31:42.263+00:00
Amar Agnihotri
916
Reputation points