Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,459 questions
1 answer
Parsing Error
Hello experts, I am reading txt files (which has sql queries) and concatenating it to a array variable. Then I convert it back to a string variable using the following expression @replace(replace(join(variables('sql_query'),' '),…
asked 2021-02-02T22:04:19.503+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 3%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Puneet Kumar
1
Reputation point
commented 2021-02-04T12:56:41.547+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 28.000000000000004%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
Haritha Maddi
26
Reputation points
1 answer
Calling Databricks activity with MSI resulting in Error Code 3207
We are following this link1922818 for setting up Databricks activity in our Datafactory. When pipeline is executed, we are getting below error Any help to resolve this is much appreciated. We confirm following things... Datafactory…
asked 2021-02-01T02:17:00.183+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 33%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Porsche Me
131
Reputation points
commented 2021-02-04T09:17:40.497+00:00
PRADEEPCHEEKATLA-MSFT
87,971
Reputation points Microsoft Employee
1 answer
How to load the different api data dynamically into corresponding tables
Hi @Anonymous , Do you have any idea on how to load the api data dynamically into corresponding tables? As mentioned in the earlier question i have created tables for the url's ,schema mapping and corresponding destination tables. As part of…
asked 2021-01-28T10:13:49.807+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 4%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
sivaranjani somana
26
Reputation points
commented 2021-02-04T05:28:00.333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Rest API as a data source and output in azure sql database
Hello, i am using web activity as a source to call rest api in adf, also my transformation is debugging properly but in my out as a SQL database it is reading only the object data not the array in it. Please tell me how i can achieve. Regards
asked 2021-01-27T06:37:15.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 23%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Punit Parmar
21
Reputation points
commented 2021-02-03T21:46:58.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,076
Reputation points
2 answers
One of the answers was accepted by the question author.
ADF, Pipeline, Copy Activity
All, I have to copy lot of tables from On-Prem to AzureSQL. I setup a pipeline and copy activities to accomplish the copy. How do I disable IDENTITY insert off to ensure that the identity column values are brought over from the source as they are…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 51%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How can I return the last cell in a column from excel file in set variable activity adf?
I have an excel sheet and need to extract the data in the last cell of a column to put into a new column in my sink file. I am using a set variable activity, is there a way to specify and return the last cell in the column? I can only make it work if I…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 98%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
1 answer
Gathering Analytics from txt files from Blob container with complex file paths
I want to be able to perform analytics on several txt files that are stored in a blob container within a storage account. The file path looks something like shoppingcenter\location\2020-05\shop\myfile.txt there's several location folders, several…
asked 2021-01-26T01:07:52.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 90%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
HarriS
1
Reputation point
commented 2021-02-03T18:24:26.813+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
ADF in Azure Synapse Analytics Workspace: Trigger Now gives error message "Sync and publish your changes before clicking "Trigger now"."
We have a simple ADF developed in Azure Synapse Analytics Workspace. Running it via "Debug" works fine. Running it via "Trigger Now" gives the error message Sync and publish your changes before clicking "Trigger now". …
asked 2021-01-26T13:00:22.25+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 81%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHS%3C/text%3E%3C/svg%3E)
Herven Stefan (SEIT)
41
Reputation points
commented 2021-02-03T17:39:42.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 50%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMG%3C/text%3E%3C/svg%3E)
Marcos Germano
1
Reputation point
1 answer
One of the answers was accepted by the question author.
How to write If --else if--- statement in Azure Mapping Data Flow Expression
Hello I am new to Azure Mapping data flows? I wanted write expression in mapping dataflows equivalent to If ( Account = 'A100') Then 'Normal' Else If (Account='B200') Then 'Gross' How I can write above logic in mapping data…
asked 2021-01-27T18:14:38.583+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 23%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Swapnil Kalkar
66
Reputation points
accepted 2021-02-03T16:41:54.17+00:00
Swapnil Kalkar
66
Reputation points
1 answer
One of the answers was accepted by the question author.
exporting data factory pipeline
hi, i have multiple adf pipelines in data factory, i want to move the entire pipeline structure to another data factory in different resource group. can you help me how i can i do that?
asked 2021-02-01T12:30:41.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 28.999999999999996%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Navdeep Kaur
21
Reputation points
accepted 2021-02-03T16:07:58.177+00:00
Navdeep Kaur
21
Reputation points
1 answer
One of the answers was accepted by the question author.
how to copy data from bloob storage to a output file by using copy pipeline activity
Hi Team, I have a requirement in azure data factory, That is in azure blob storage we have 2 files separately named employee_info and Department_info I have to create a output file with the below transformations 1.need Create Pipeline to copy the…
asked 2021-02-03T13:42:26.8+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 40%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEN%3C/text%3E%3C/svg%3E)
Eshwar Nakkala
106
Reputation points
commented 2021-02-03T14:48:28.85+00:00
Nasreen Akter
10,791
Reputation points
1 answer
MSDN Forum: Data flows to Onprem sql connectivity not enabled in Azure data factory
I was trying to copy the data flow Blob to Onprem SQL server using data flows. When trying to choose a SQL Server (On Prem) as sink, sql server icon is diabled in the sink linked service. Any reson for the same? Anyhow able to connect with normal…
asked 2021-02-01T06:39:50.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 8%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
ashok basani
1
Reputation point
commented 2021-02-03T11:49:44.15+00:00
PRADEEPCHEEKATLA-MSFT
87,971
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure SSIS IR custom DLL installation with main.cmd
I have used a azure SSIS IR to execute my SSIS package from on premise to azure adf but its failing as we epplus dll are used in script task of SSIS packages and same I tried to add using main.cmd but still facing issue in this .
asked 2021-01-26T10:23:47.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 46%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKA%3C/text%3E%3C/svg%3E)
Khatri, Ashish
96
Reputation points
accepted 2021-02-03T07:29:29.76+00:00
Khatri, Ashish
96
Reputation points
0 answers
HOW TO REPLACE \ SLASH IN AZURE DATA FACTORY WITH FUNCTIONS
HOW TO REPLACE \ SLASH IN AZURE DATA FACTORY WITH FUNCTIONS
asked 2021-02-01T10:22:39.593+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 36%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
ADF C2
1
Reputation point
commented 2021-02-03T06:17:57.303+00:00
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
Copy Activity - Enable Logging - Need to provide log file name
I am using the Enable logging option to log the details of the copy activity into a blob. Below are the options that I have set. The details of the copy activity are logged correctly in the blob. However, the log file name is…
asked 2021-01-29T08:27:16.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 65%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan (SC-ALT)
51
Reputation points Microsoft Employee
commented 2021-02-03T03:58:36.51+00:00
Nandan Hegde
31,781
Reputation points MVP
1 answer
One of the answers was accepted by the question author.
Data Factory Stored Procedure Error Cannot find the object
I have a stored procedure to truncate tables and it works absolutely fine in SQL Server EXEC [staging].[USP_TruncateTables] @tableName = 'example', @SchematableName = 'staging.example'; So I can use this truncate Script for all the staging tables However…
asked 2020-10-08T09:41:21.453+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDE%3C/text%3E%3C/svg%3E)
Debbie Edwards
521
Reputation points
commented 2021-02-02T18:56:24.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 18%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bala Murali Krishna
1
Reputation point
0 answers
The Read method cannot be called
Hello, I am having an issue with the npgsql connector to postgres. Error message below: 'Type=Npgsql.NpgsqlException,Message=Exception while reading from stream,Source=Npgsql,''Type=System.NotSupportedException,Message=The Read method cannot be…
asked 2021-01-20T14:45:15.307+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 60%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERR%3C/text%3E%3C/svg%3E)
Robert Riley
21
Reputation points
commented 2021-02-02T17:57:29.783+00:00
Robert Riley
21
Reputation points
0 answers
Synapse query failed after 6 minutes from Data factory
Hi, I'm using the copy activity to extract data from Synapse analytics (SQL DWH). After exactly 6:38 minutes the query fails and the error is a Timeout Error. If a run the query from ssms, works. I set up the max time for query timeout: …
asked 2021-01-25T12:31:30.273+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 17%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJR%3C/text%3E%3C/svg%3E)
Jhonatan Reyes
61
Reputation points
commented 2021-02-02T17:27:22.957+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Cannot connect to SQLMI database when using Azure Auto Resolve Integration Runtime
Hi All, I am loading data from ADLS Gen2 into SQLMI. When i am tried to create a linked service using Azure Auto resolve IR to connect to SQLMI using SQL authentication , i am getting the below azure message. Enabling Azure services in NOT an option…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 7.000000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
1 answer
Data Factory v1 - edit linked service with dependancies.
We've moved all of our data factory v1 jobs to v2 except for a few that we're unable to at this point. I need to edit the values in a linked service. I understand that that is no longer supported in the portal and must be done via powershell or rest…
asked 2021-01-24T13:49:26.087+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 21%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
bhb252
1
Reputation point
commented 2021-02-02T10:08:57.51+00:00
HarithaMaddi-MSFT
10,136
Reputation points