Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,662 questions
2 answers
Extract files from .tar.gz files store in Blob Container
Is it possible to extract files from .tar.gz using Azure Data Factory or Functions App to be ingested by ETL process in ADF? I tried to use 7zip in Functions Apps, which worked fine to extract a test .tar.gz file uploaded to the Functions App but…
asked 2020-09-11T17:23:46.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 83%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
elastiSol
1
Reputation point
commented 2020-09-21T17:07:02.647+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
What does publishing do in Azure Data Factory do?
What does publishing a pipeline/dataset in data factory do? Does it publish the data somewhere where other users can access? Is it related to publishing a Power BI notebook?
asked 2020-09-19T15:02:01.35+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 94%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Siddhanth Shetty
66
Reputation points
accepted 2020-09-21T11:35:21.723+00:00
Siddhanth Shetty
66
Reputation points
1 answer
Mapping Data Flow Sink to SQL on IaaS VM
Does Mapping Data Flow allow us to sink the data into a SQL DB hosted on a IaaS VM? Currently, there is no option other than Azure Blob Storage or Azure SQL. Any inputs here are highly appreciated. Thanks.
asked 2020-09-15T20:05:01.807+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 73%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Priyaranjan Midde
1
Reputation point
commented 2020-09-19T00:05:59.85+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
0 answers
ADF SQL to CRM Copy Data Pipeline Error
I am currently working on a Azure Data Factory Mapping. SQL Server to CRM There is a Entity Reference look up field that I am trying to update. Do not know how to. The field is: new_districtlink --> Territory/Territories. My Source does a…
asked 2020-09-14T03:42:15.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 55.00000000000001%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
Shamsuddin, Natasha
1
Reputation point
commented 2020-09-18T23:31:29.247+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF pipeline stuck in "Queued" status while writing data in SFSC
I am writing to upsert / insert records in SFSC from sql server object . The connection to SFSC is working fine and successful. But when I run the pipeline the status is stuck in "Queued" . It never changes.
asked 2020-09-10T16:52:55.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 61%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Alinkil, Mithun
21
Reputation points
accepted 2020-09-18T20:23:20.773+00:00
Alinkil, Mithun
21
Reputation points
1 answer
Copy Activity Sink to SFTP "useTempFileRename": false option not always working
Has anyone else experienced this issue, is it a bug? When using Copy Activity and ftp to a site which does not allow file rename I set "useTempFileRename": false. But even though ADF will say copy was successful the file sometimes is not…
asked 2020-09-15T14:03:02.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 31%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jay
561
Reputation points
commented 2020-09-18T17:54:12.513+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
protect JSON-file /ARM template Azure Data Factory
Hi, I have developed an ETL procedure for a specific ERP system in Azure Data Factory. I want to sell the entire procedure to customers so that they can use the ARM template in their own Azure tenant. However, I do not want them to make a copy of…
asked 2020-09-18T11:34:13.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 52%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
markharing@bisqq.nl
61
Reputation points
accepted 2020-09-18T12:28:13.007+00:00
markharing@bisqq.nl
61
Reputation points
1 answer
One of the answers was accepted by the question author.
How to manually cancle a ADF pipline when an activity fails or completes?
Hi, I have a pipeline which has two concurrent list of activities running at the same time lets say the first activity has a flag which makes the until loop in the second list of activity to complete if flag is true. Now the problem is the flag is…
asked 2020-09-14T04:39:40.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 21%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Stramzik
21
Reputation points
commented 2020-09-17T20:25:56.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,061
Reputation points
1 answer
Error occurs at source transformation when trying to source csv file from folder in data lake storage gen2 while debugging the pipeline
While using mapping data flow activity, In source transformation I am trying to read a csv file from a folder in Azure data lake gen2. I am able to preview the data in "Data Preview" in source transformation but when trying to debug the…
asked 2020-08-31T03:05:19.917+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 86%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rushabh Shah
1
Reputation point
commented 2020-09-17T16:55:13.657+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Data Factory Pipeline to Azure SQL Server - fails on firewall IP
Hi Team, I'm not sure what I've done wrong here. I've got an Azure Data Factory pipeline that copies data from an on-prem SQL Server (using the Self-Hosted Integration Runtime) up into a database running on an Azure SQL Database. I can grab the…
asked 2020-09-15T08:12:13.74+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 20%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMR%3C/text%3E%3C/svg%3E)
Matt Russell
36
Reputation points
accepted 2020-09-17T07:27:38.333+00:00
Matt Russell
36
Reputation points
2 answers
ADF Copy activity output into multiple parquet snappy file?
In ADF copy activity output, pipeline is creating very large parquet file. Will it possible to split the output into multiple small parquet snappy files, so that synapse external table can use parallelism (polybase)? like Similar to DBrick data frame,…
asked 2020-08-17T11:34:57.857+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 55.00000000000001%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
ever
1
Reputation point
answered 2020-09-16T08:59:47.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
[ADF] Azure Data Factory Copy Activity number of files
We've a simple ADF pipeline that copies data from an on premise sql server database (throught one IR node) to ADLS Gen 2 We are trying to optimize job duration/performance by writing multiple files sink side but the parallel copy parameters / DIU…
asked 2020-08-05T15:10:16.443+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 23%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Aminouvic
1
Reputation point
answered 2020-09-16T08:58:10.18+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Alternative way to Deploy SSIS Package without Data factory
I need to deploy only few SSIS Package to Azure Database for that I cannot afford the Data factory, is there any Feasible way to deploy the package like using dtex option in Premises Server in Azure
asked 2020-09-13T11:41:51.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 22%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAW%3C/text%3E%3C/svg%3E)
Alternative way for deploying SSIS PackageAzure Db
1
Reputation point
commented 2020-09-16T08:48:36.217+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
data flow drops column in different data factory
I have a data flow which is not displaying a column on the last step the sink when I view the the data in Date Preview. The source has been pushed from for our repository. Checking the data flow JSON matches between the Dev data factory and the…
asked 2020-09-14T00:18:04.963+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 23%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Peter McLarty
1
Reputation point
commented 2020-09-16T08:41:30.57+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Please help me to fix this error 2200
"ErrorCode=UserErrorAzureMysqlNetBatchWriteTimeOut,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=batch write time out, more than…
asked 2020-09-11T03:55:28.243+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 55.00000000000001%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJJ%3C/text%3E%3C/svg%3E)
Jackson Jacob
1
Reputation point
commented 2020-09-15T08:16:16.097+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Azure Data Factroy - select files from a folder based on a wildcard filename
I'm new to ADF and thought I'd start with something which I thought was easy and is turning into a nightmare! I'm trying to do the following. Copy files from a ftp folder based on a wildcard e.g. 'PN'.csv and sink into another ftp folder. (wildcard*…
asked 2020-09-11T10:51:18.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 87%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Ja
1
Reputation point
commented 2020-09-15T08:14:51.743+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF - Cosmos DB linked service - using managed identity
When creating a ADF linked service for cosmos db connection, I do not see a managed identity option. Only see Connection String / Azure Key Vault options to connect. Can we not connect from ADF to Cosmos db using managed identity. Thanks, Kothai.…
asked 2020-09-11T11:29:10.243+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 6%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-09-15T03:10:07.243+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
1 answer
Azure Data Factory - Technical Bug when ingesting data from on-premise File System to Azure File Storage
Hi Team, I was working on a pipeline to move data from on-premise File System to Azure File Storage. I found a strange bug which is causing error when trying to create a Azure File Storage linked service using UI but it is working smoothly when…
asked 2020-07-27T13:02:31.163+00:00
Manas Agarwal
11
Reputation points
answered 2020-09-15T00:17:48.957+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Pipeline activities are executing twice
I've been noticing an issue over the past week where pipeline activities are being executed twice, with no apparent reason based on the pipeline design. For example, I have a pipeline that simply calls two sub-pipelines sequentially: For some…
asked 2020-09-07T01:52:15.257+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 87%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBF%3C/text%3E%3C/svg%3E)
Ben Forbes
26
Reputation points
commented 2020-09-14T12:19:15.243+00:00
Ben Forbes
26
Reputation points
0 answers
Azure Data Factory - Failed to save dataflow. Error: Failed to fetch
Hi, I have the ADF data flow created in Git repository. After my modification in the data flow, when i click on save, getting the error message like 'Failed to save <data flow name>. Error: Failed to fetch'. Recently started facing this issue.…
asked 2020-09-10T11:24:12.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 47%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
GGN
6
Reputation points
commented 2020-09-14T07:50:37.687+00:00
HarithaMaddi-MSFT
10,136
Reputation points