Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
1 answer
One of the answers was accepted by the question author.
How to read Data Partitioned files from ADLS using ADF mapping data flow source options
Hi All, Good Day!! We have a requirement to join multiple files which are available in ADLS folders which we have ingested from SQL Source and derive business logic accordingly. Since we store the Table data in ADLS with table name and date…
Azure Data Factory
asked 2021-07-09T17:50:03.347+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 83%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Abdulla Mahammad Khan Dawood
186
Reputation points
commented 2021-07-16T07:37:55.56+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
1 answer
Data Factory Copy Task Running Indefinitely
Good Day I have a problem where I have an Excel .xlsx file in the Data Lake, the size of the file is 48MB. I am trying to copy this file into an Azure SQL DB. This task works for many other smaller files but when it gets to this file it just hangs…
Azure SQL Database
Azure SQL Database
An Azure relational database service.
6,363 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-09T05:58:24.13+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 85%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
Keagan Jordan
1
Reputation point
commented 2021-07-16T06:33:57.24+00:00
Keagan Jordan
1
Reputation point
0 answers
ADF Mapping for Copy with Source being a Web REST Dataset times out before I can get Schemas?
@Anonymous @Anonymous @Anonymous We have a long running REST Source in a Copy Activity where I adjusted the Timeout to 5 mins and it works. HOWEVER, when I try to Map this to a DB Table it times out when I hit Import Schema. Is there any way…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-13T17:07:56.633+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 49%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mike Kiser
1,536
Reputation points
commented 2021-07-16T06:05:37.217+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
1 answer
Cant get hold of the array from a ForeEach activity
I get this error: The expression 'length(activity('Copy Excel to Cache Sink').output.value)' cannot be evaluated because property 'value' doesn't exist, available properties are 'runStatus, effectiveIntegrationRuntime, billingReference,…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-09T13:24:45.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 16%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKN%3C/text%3E%3C/svg%3E)
Klaus Nørregaard
1
Reputation point
commented 2021-07-16T05:58:47.643+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
0 answers
Can't publish Azure Data Factory
Hey when I try to publish my ADF I get this error: Error code: BadRequest Inner error code: BadRequest Message: The document creation or update failed because of invalid reference 'source_vp'. Resource:…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-08T11:31:16.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 89%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Michael van der Bend (student)
1
Reputation point
commented 2021-07-16T05:57:36.217+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
1 answer
ServiceNow Integration with Azure data factory for incremental data extraction
I am trying to create a pipeline in ADF where ServiceNow is a source and Azure blob storage is my target as and storing extracted files n json format. I have to get only incremental data from SNOW and load that into blob storage. Please provide any…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-08T08:58:23.397+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 85%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKG%3C/text%3E%3C/svg%3E)
Kapil Ganweer
1
Reputation point
commented 2021-07-16T05:55:56.797+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
2 answers
One of the answers was accepted by the question author.
Copy Activity - mapping translator
According to copy-activity-schema-and-type-mapping, it is possible to define the schema for the copy activity. We have been using this process to alter column names in the sink which has been working well. However, we've noticed that all columns are…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-14T03:50:12.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 77%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Ryan Abbey
1,186
Reputation points
commented 2021-07-15T20:15:50.58+00:00
Ryan Abbey
1,186
Reputation points
1 answer
Azure Data factory got error when getting information from HubSpot
Encounter problem suddenly when preparing data from HubSpot error message: Operation on target ForEach_Table failed: Activity failed because an inner activity failed; Inner activity name: Copy_HubspotTBL, Error: Failure happened on 'Source' side.…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-06-03T06:36:00.833+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 5%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Simon TC Kong (Tricor HK/IT)
6
Reputation points
answered 2021-07-15T20:07:11.923+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
0 answers
ADF How to add a line feed to a long text file? Urgent....
Hi @Anonymous @Anonymous @Anonymous My ADF pipeline created a very long text fixed field file where the following is just a few of the first several characters. 10E20131216330785887BMECOL 00 …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-11T18:43:26.68+00:00
Mike Kiser
1,536
Reputation points
commented 2021-07-15T19:49:15.99+00:00
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
1 answer
How to receive a http post in Data Factory?
Function 1 sends a message body by HTTP post every 5 minutes. In Data Factory I want to receive this message and pass this body to Function 2 as input. { "body": "run" } How can I receive this message in the…
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,953 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-12T05:52:16.243+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 43%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Mohsen Akhavan
831
Reputation points
commented 2021-07-15T19:47:06.687+00:00
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
2 answers
Azure SQL DB to Snowflake
Hello Team, I am trying to upload the data from Azure SQL DB Table to Snowflake db table.As snowflake is in AWS and we are creating the pipeline in azure data factory. Please refer the below error : Regards Rohit
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-12T11:47:04.313+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rohit Kulkarni
731
Reputation points
commented 2021-07-15T19:45:22.483+00:00
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
Adf schema mapping without an key
In an copy data activity I have a hierarchical source to tabular sink mapping. We extract data from an api. within the data we have the following node *, "businessCommunication": { "emails": [ …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-06-03T15:29:12.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 62%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Palios, Jason
81
Reputation points
commented 2021-07-15T19:29:17.66+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,156
Reputation points
1 answer
One of the answers was accepted by the question author.
Copying 100MB of data using ADF causes cosmos db partitions to increase from 2 to 50
We are copying data from West US 2 to North Europe using Azure Data factory. From Azure Data Factory, we see that around 104 MB of data was copied before the pipeline failed with Timeout Exception: …
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,922 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-12T21:36:41.557+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 47%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Nikhil Kumar
21
Reputation points Microsoft Employee
commented 2021-07-15T18:06:41.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,851
Reputation points Microsoft Employee
Moderator
0 answers
How to specify numbers of recors,rows to read in a copy activity?
How can I set the number of records to process so my Copy Activity won't crash. I'm copying data from Dynamics365 to Azure BLOB but i think the data in Dynamics is so big so Azure datafactory can't read it and it doesn't copy nothing here…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-01T20:31:00.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 65%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
Pablo CR
21
Reputation points
commented 2021-07-15T17:09:05.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,491
Reputation points Microsoft Employee
Moderator
2 answers
One of the answers was accepted by the question author.
Azure Data Factory: Date Conversion
I am creating a pipeline where the source is csv files and sink is SQL Server. The date column in CSV file may have following values like 12/31/2020 10162018 20201017 31/12/1982 1982/12/31 I do not find the function which checks the format of the…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2020-10-22T14:26:22.023+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 61%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Abhishek Vyas
101
Reputation points
answered 2021-07-15T14:02:30.003+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 95%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Venkata Subbaiah Kuruva
26
Reputation points
1 answer
One of the answers was accepted by the question author.
Synapse analytics for data ingestion
Hi All, I have seen a architecture which suggests Synapse Analytics for data ingestion and afterwards data is stored in ADLS Gen2 and again data pushed in to Synapse as well as Azure SQL database . Synapse used again third time and finally data being…
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,385 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-14T12:23:05.987+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 12%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Samy Abdul
3,376
Reputation points
commented 2021-07-15T10:34:28.477+00:00
PRADEEPCHEEKATLA
90,736
Reputation points
Moderator
0 answers
Approach to copy dynamically growing files and containers from one ADLS Gen2 storage account to another
Hi Team, We are looking at approach to copy data on a daily basis from one ADLS Gen2 storage account to another ADLS Gen2 storage account. Containers in our storage account are dynamically growing and folder structure is not pre-defined. We need to…
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,563 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-07-12T15:08:40.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 28.000000000000004%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
Haritha Maddi
26
Reputation points
commented 2021-07-15T08:43:23.127+00:00
Haritha Maddi
26
Reputation points
2 answers
One of the answers was accepted by the question author.
Getting following error (Client IP not authorized to access the API) in azure Data factory while publishing new pipeline changes
Seems new change happened to Azure Data factory and we are not able to publish the changes due to my IP is not registered - can anyone suggest where i need to add my IP? Error code: OK Inner error code: ClientIPNotAuthorized Message: Client IP not…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2020-11-06T22:13:47.153+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 31%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Srinivasa Rao Gattupalli (HCL America Inc)
31
Reputation points Microsoft External Staff
answered 2021-07-14T21:43:45.507+00:00
MartinJaffer-MSFT
26,156
Reputation points
0 answers
ADF Web activity can't connect to API When API Key has special character
I am trying to use a Web activity in an ADF pipeline to issue a GET to an API. The API authentication is through an Authorization header with an API key provided to be by the API developer. The API Key is of the form username:password (note the colon) …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-06-04T11:17:16.133+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 61%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENB%3C/text%3E%3C/svg%3E)
Nick Baker
1
Reputation point
commented 2021-07-14T17:06:25.95+00:00
MartinJaffer-MSFT
26,156
Reputation points
1 answer
One of the answers was accepted by the question author.
Error loading Excel file from Blob
We get Excel files uploaded to our Blob storage from another group. We are trying to copy the contents to an Azure SQL table. The issue I am having is with the format of the Excel file. It is using row groupings and is causing errors when being read…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 59%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)