Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,661 questions
0 answers
Data Factory doesn't store credentials in Git. I could see credentials saved.
This para mentions that Data Factory doesn't store credentials in Git. https://learn.microsoft.com/en-us/azure/data-factory/source-control#using-passwords-from-azure-key-vault But when I create/publish a linked service in a feature branch, I can see that…
asked 2024-10-03T04:51:30.0766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 28.999999999999996%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Ravikiran Srini
0
Reputation points
asked 2024-10-03T04:51:30.0766667+00:00
Ravikiran Srini
0
Reputation points
0 answers
Azure Data Factory with Dataverse Sink (mapping data flow) Upsert broken?
Simple ADF data flow with a dataverse sink (Dynamics 365). Created an alternate key on a custom dataverse table and attempting to do an Upsert. I have the Alter row transform with "Upsert if" -> true() I have set the sink to Upsert…
asked 2024-09-30T19:16:42.8+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 12%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENG%3C/text%3E%3C/svg%3E)
Noah Gilbert
26
Reputation points
commented 2024-10-03T04:26:38.3466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
10,330
Reputation points Microsoft Vendor
1 answer
Microsoft Fabric Copy Data Activity - Filter Out Files that Start With "~"
I need to copy files from a network folder to a Lakehouse and I am building a pipeline to handle that tasks. Some of these files are temporary excel files that Microsoft generates natively as other users are making edits. I need to exclude files that…
asked 2024-10-02T22:46:19.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 28.000000000000004%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJF%3C/text%3E%3C/svg%3E)
Jordan F
0
Reputation points
answered 2024-10-03T04:23:27.53+00:00
Nandan Hegde
32,256
Reputation points MVP
2 answers
One of the answers was accepted by the question author.
Read sharepoint list data and convert to csv using ADF
Is it possible to read sharepoint list data using ADF with having 60 sites and 100+ list items and automate XML response to CSV files ? We already did POC using databricks, its working fine, but want to know whether ADF can handle without…
asked 2024-09-23T14:39:19.3766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 88%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sankon437
41
Reputation points
commented 2024-10-03T02:18:58.0433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
AllenXu-MSFT
19,961
Reputation points Microsoft Vendor
2 answers
Azure Data Factory ServiceNow Connector Copy Activity Dynamic Date Filter Query Builder
Hello all, We recently received a notice to update our legacy ServiceNow connection driver and while updating our copy pipeline activities, I am struggling to add a filter on a date column. In the legacy connection driver, we were able to use a query…
asked 2024-10-01T16:38:41.4566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 24%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDG%3C/text%3E%3C/svg%3E)
David Gonzalez
0
Reputation points
commented 2024-10-03T01:26:27.7166667+00:00
phemanth
10,330
Reputation points Microsoft Vendor
3 answers
One of the answers was accepted by the question author.
How to reference Json fields in copy data (ADF)?
I am currently integrating an API response into a SQL Server database using Azure Data Factory. The response from the API is structured in JSON format, as shown below: The goal is to extract the id values from each batch and insert them into a SQL…
asked 2024-09-26T16:54:25.8933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 32%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Max Teitelbaum
20
Reputation points
commented 2024-10-03T01:01:21.13+00:00
phemanth
10,330
Reputation points Microsoft Vendor
2 answers
How to override an ARMTemplateParametersForFactory.json [Array as value] in GitHub Actions
I already got help from this "You can override the parameters in parameter file by explictly calling it out next to the parameter file. Example : parameters: repo/ARM/azuredeploy.resourcegroup.parameters.json rgName=${{env.RESOURCEGROUP_NAME}} This…
asked 2024-10-02T08:43:20.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 6%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECK%3C/text%3E%3C/svg%3E)
Chayut Kanthawong
0
Reputation points
answered 2024-10-02T19:55:09.7333333+00:00
Amira Bedhiafi
24,376
Reputation points
1 answer
How do I make a dataset based on an SQL statement with multiple tables?
How do I make a dataset based on an SQL statement with multiple tables, not a single table? I'm using MS Azure Data Factory online via a web browser. I'm not a total beginner but I know a few things and have completed a tutorial on Azure. This…
asked 2024-10-02T09:55:25.5133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 77%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Chuck Roberts
105
Reputation points
answered 2024-10-02T19:48:38.0166667+00:00
Amira Bedhiafi
24,376
Reputation points
2 answers
Switch from Legacy BigQuery to new BigQuery Linked Service
I have a BigQuery routine that takes start date and end date as input parameter and both input parameter is declared of type String in routine. This routine is working fine without errors. I want to invoke the BigQuery routine from Azure Data Factory.…
asked 2024-09-24T10:13:48.1233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 87%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
ShilpaGopal
100
Reputation points
commented 2024-10-02T18:43:10.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 14.000000000000002%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOR%3C/text%3E%3C/svg%3E)
Omar Rose
0
Reputation points
2 answers
What does ADF SSIS Error "Disk Full Exception Caught" mean
Hi, we are running an Azure-SSIS Type Integration Runtime in Azure Data Factory. This morning this IR suddenly got Unavailable for some reason. Fortunately we had the Diagnostic Settings ADFSSISIntegrationRuntimeLogs enabled and I found the following…
asked 2024-09-25T10:26:26.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 89%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Cedric Ahlers
35
Reputation points
edited the question 2024-10-02T16:20:36.3766667+00:00
Cedric Ahlers
35
Reputation points
0 answers
Error during batchpoint invoke: Access to this resource is denied. Please check your ACL rules on the resource
Hello, I am getting a "Access to this resource is denied. Please check your ACL rules on the resource." when attempting to invoke a batch endpoint. I have tried both with Azure CLI and Azure Data Factory to invoke a batch endpoint, with both…
asked 2024-07-22T11:59:17.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 39%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKT%3C/text%3E%3C/svg%3E)
Khai-Tam Nguyen
15
Reputation points
commented 2024-10-02T14:35:54.0533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 64%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Maui Bar
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to increase throughput in ADF copy activity From Table Storage to Table Storage
Hello, Short background - I'm using ADF for creation of Table storage Backups for specific Storage Accounts (each account is different pipeline with separate copy activities for each Table), creating new Table storages from old ones using copy…
asked 2024-09-26T13:10:27.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 98%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETB%3C/text%3E%3C/svg%3E)
Tomas Blažauskas
40
Reputation points
accepted 2024-10-02T13:43:45.01+00:00
Tomas Blažauskas
40
Reputation points
1 answer
One of the answers was accepted by the question author.
Power Query merge query on two colums
Hello: I have two datasets (UserQueries) in Power Query, I now want to merge them on two columns that are common. Should be easy, right. I use Merge Queries on the tool bar, get a dialogue box as follows: The col's are correct and the order is correct.…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
Data Factory Pipeline source pointing to Azure Data Lake failing
Hello: I have a Source in a Pipeline which points to a DataSet. If I test the DataSet the connection tests fine and I can preview the data. The schema is also being picked up correctly. There is an extra attribute in the first position, called Prop_0. …
1 answer
Automating execution of ".bat" file using ADF / LogicApp
Hello Team, Currently we are manually executing a .bat file in one of the client VMs and we need to automate the same using ADF / Logic App. The bat file is inside a folder and it executes a .jar file which resides within the same folder. The .jar file…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 53%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENC%3C/text%3E%3C/svg%3E)
1 answer
ADF - converting list of lists into a proper JSON format
Hello, I'm pretty new to ADF and I can't wrap my head around one case. To keep it simple: I have a source (REST API response), which theoretically is JSON but it doesn't have a "proper" JSON format. It's supposed to be a simple table but it…
asked 2024-09-25T18:29:34.62+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 22%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Patryk Sobczak
0
Reputation points
edited a comment 2024-10-02T12:28:02.0366667+00:00
Patryk Sobczak
0
Reputation points
1 answer
Salesforce connector : I am getting TaskCanceledException error while doing full load for Service__c and Service_Step__c entity.
Hello, We have recently transitioned to the Salesforce Bulk API 2.0 Connector in Azure Data Factory (ADF) for our integration processes. As part of our use case, we perform a full data load for all Salesforce entities once a month. However, we are…
asked 2024-10-01T07:32:09.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 0%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDU%3C/text%3E%3C/svg%3E)
Dhaval Uday Shah
0
Reputation points
commented 2024-10-02T12:02:31.69+00:00
Vinodh247
20,141
Reputation points
1 answer
One of the answers was accepted by the question author.
When to use deployment jobs vs. normal jobs in Azure Pipelines for Bicep deployments?
As a data engineer working on infrastructure-as-code using Bicep, I'm trying to understand the best practices for structuring my Azure Pipeline. I have separate environments for DEV and PRD, and I'm not sure when to use a deployment job versus a normal…
asked 2024-09-26T14:37:01.7233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 86%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
Melvin Tran
40
Reputation points
accepted 2024-10-02T07:04:17.2733333+00:00
Melvin Tran
40
Reputation points
0 answers
Cannot connect to SQL Database. Please contact SQL server team for further support.
Hello support from time to time i get an error that username/password is invalid for database. However database is up and running and in use. below are the infromation activityid 701ef079-7056-4d6b-b941-c2e2f011e57a
asked 2024-10-01T08:13:02.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 83%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENZ%3C/text%3E%3C/svg%3E)
Ninoslav Zekic
10
Reputation points
commented 2024-10-02T02:07:35.69+00:00
LiHongMSFT-4306
27,016
Reputation points
1 answer
How to set ADF linked services and triggers parameterized for respective environment when we create pull request
How to Set ADF linked services and triggers parameterized for respective environment when we create pull request like dev-parameter.json
asked 2024-10-01T11:22:51.0433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 38%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMR%3C/text%3E%3C/svg%3E)
Madhu Ram
0
Reputation points
commented 2024-10-01T20:24:58.4233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
1,730
Reputation points Microsoft Vendor