9,338 questions with Azure Data Factory tags

Sort by: Updated
1 answer

How to skip error and continue unzip process using Azure Data Factory copying activities

I have large zip file that I transferred from SFTP to blob storage. I have created a copy activity to unzip the files from one blob storage (source dataset) to another blob storage (sink dataset). The unzip process has the error : file operation is…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-28T02:20:15.25+00:00
Liu, Faye 0 Reputation points
commented 2024-03-28T14:55:52.33+00:00
Liu, Faye 0 Reputation points
0 answers

Authentication Type for High Sensitive data

Hi all, I've a new requirement where I've to load very high sensitive data within the organization to Azure SQL Database via Azure Data Factory. Only very few people have access to the database due to data sensitivity level. I'm looking into various…

Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
885 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-28T14:09:37.7866667+00:00
Nivaethitha Thangiah Marthandam 0 Reputation points
0 answers

unable to load data from salesforce to azure storage via ADF

Hi team, i built a adf pipeline using look up activity & copay activity, i am loading data from salesforce objects to azure storage adlsgen2 container. I have total 20 tables and 19 tables are successfully loaded into storage account and one table…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,608 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-28T12:20:49.2466667+00:00
Ram Babu 25 Reputation points
1 answer

Creating a pipeline for data transfer from SQL Server to blob storage

I'm looking to create a single pipeline that can transfer data from a SQL Server database to BLOB storage in a Parquet format. This pipeline should go through a table that lists the connection string and SQL commands needed for each table or Parquet file…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-28T11:01:28.26+00:00
Vikranth-2626 80 Reputation points
answered 2024-03-28T11:25:27.4066667+00:00
Smaran Thoomu 8,025 Reputation points Microsoft Vendor
0 answers

batch output of a lookup and then pass to a web action through for each loop

I do a lookup on a table to receive some ID's. Then I take these ID's and put them in the body of a http POST request. Right now I have it like this: The problem is that each row would be it's own request, and I don't want to send +100k requests at the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-27T13:30:16.76+00:00
Hamza Outa 41 Reputation points
commented 2024-03-28T10:57:36.1333333+00:00
Smaran Thoomu 8,025 Reputation points Microsoft Vendor
2 answers

private endpoint service endpoint private link and public endpoint

hi friends I am confused with private endpoint service endpoint private link and public endpoint why and when to use this what scenario.please give a practical example while explaining

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2023-01-12T08:42:25.7666667+00:00
Anshal 1,626 Reputation points
commented 2024-03-28T10:24:58.0833333+00:00
Sun Chen 20 Reputation points
0 answers

Azure Data Flow connection error to Dynamics 365 but pipeline can

I have a pipeline that has a connection to a table in Dynamics 365 which works fine. I need to create a dataflow as I need to lookup a value in the source that then replaces the value with a guid to load into Dynamics 365 but when I do a data preview in…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-04T05:21:30.9166667+00:00
Wayne Bartkowski 656 Reputation points
commented 2024-03-28T10:10:40.8266667+00:00
ShaikMaheer-MSFT 37,566 Reputation points Microsoft Employee
0 answers

In ADF using HDFS linked service my copy file activity throws the following error

Hi, I have an issue using ADF with HDFS linked service. I created a HDFS connection then a copy acitivity from HDFS to Azure Data Lake gen2. The source is a CSV file and the copy format is binary. When I run the pipeline I get the following error: …

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,315 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
6,978 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,251 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-28T10:07:15.1766667+00:00
Ács Dániel 0 Reputation points
2 answers

How to create the Data set, pipeline and run it, for excel format using C sharp for Data factory

I want to create and run my pipeline from the repo using C#. my code is string postgreSQLLinkedServiceName = "PostgreSQLLinkedService"; LinkedServiceResource postgreSqlLinkedService = new LinkedServiceResource( …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,097 questions
asked 2024-03-08T08:18:17.3666667+00:00
Ashish Sharma 0 Reputation points
answered 2024-03-28T10:06:28.0733333+00:00
ShaikMaheer-MSFT 37,566 Reputation points Microsoft Employee
2 answers

Unable to load data from Dynamics 365 CRM sales table to MS-SQL

I was trying to load data from Dynamics 365 Sales to MS-SQL using Using Azure Data Factory Copy data tool, Metadata-driven copy task. The type of load I am using is Delta Load/Incremental Load. Facing this issue: Failure happened on 'Source' side.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-10T10:46:15.3633333+00:00
Advait Bhujbal 0 Reputation points
answered 2024-03-28T10:03:10.77+00:00
ShaikMaheer-MSFT 37,566 Reputation points Microsoft Employee
0 answers

how to flatten the nested json array into csv file using ADF pipeline

how to flatten the nested json array file into csv file using ADF pipeline. using below example i need to get the each table details in the excel/csv file. need the file like these below using below ex json Ex: { "Table1":…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-27T18:40:49.3133333+00:00
commented 2024-03-28T09:58:52.8233333+00:00
phemanth 4,695 Reputation points Microsoft Vendor
1 answer

Data flow sink preview is sorted but not sorted in the outputted tsv sink file

I have a data flow in data factory that gets data from a database and transforms some columns and then outputs it on a tsv sink file. In this data flow the data is sorted by a certain column. The sink preview is sorted correctly but when I open the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-21T04:05:25.86+00:00
rbc_user 0 Reputation points
commented 2024-03-28T09:48:35.7433333+00:00
ShaikMaheer-MSFT 37,566 Reputation points Microsoft Employee
0 answers

Tumbling window trigger in copy pipeline seems to be copying more data than present in the window.

I used the 'Ingest' option in Azure data factory to enable continuous ingestion from one table to another table using Tumbling window trigger. I set the interval in the Tumbling window trigger to 15 mins and the window start and end time for each trigger…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-26T02:40:14.8633333+00:00
Swetha Seelam 0 Reputation points Microsoft Employee
commented 2024-03-28T09:48:09.3366667+00:00
Smaran Thoomu 8,025 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

what would be the reason where credential creation for linked service in ADF is taking time forever?

Hello There, I have an issue creating 'credential' in linked-service for 'azure blob storage' where after giving all the information including resource-group, resource-type, 'user-assigned-managed-identity', and hit the 'create' button, the creation…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2022-11-01T01:31:21.003+00:00
ArunRaaman 991 Reputation points
answered 2024-03-28T07:44:39.1833333+00:00
0 answers

Schedule Trigger not working even after week creation?

Schedule trigger created for pipeline to run every 15 min on daily basis. And trigger created week back but still trigger not happening. What could be the problem?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-28T02:39:30.0033333+00:00
Anuganti Suresh 140 Reputation points
commented 2024-03-28T06:56:32.3133333+00:00
Smaran Thoomu 8,025 Reputation points Microsoft Vendor
1 answer

ERROR: {"code": "InvalidTemplate", "message": "Deployment template validation failed}

Hi Team, We are trying to deploy to ADF instance using Jenkins as CI/CD platform. We are following the documentation for reference: …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-28T03:51:43.8+00:00
ShaikNisharunnisa-3795 0 Reputation points
commented 2024-03-28T06:10:32.5866667+00:00
ShaikNisharunnisa-3795 0 Reputation points
2 answers

How to trigger an adf pipeline of one adf resource in another adf resource pipeline

We required to call a pipeline from one ADF resource to another ADF resource. I tried to export and import json pipeline format but could not import same in other ADF resource due to asking Git configuration. I tried through WEB activity but failed by…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-26T13:21:49.15+00:00
Bhaskar Dindi 0 Reputation points
commented 2024-03-28T06:05:54.84+00:00
phemanth 4,695 Reputation points Microsoft Vendor
1 answer

Copy data activity from synapse serverless sql pool to ADLS gen 2

Hi, I have synapse serverless sql pool configured to ADLS Gen 2 account where the container is filled with files from D365 using synapse link. I need to build datawarehouse in Azure. We have very tight restrictions on budget and don't want to go…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,315 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,251 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-06T17:28:48.3066667+00:00
rajanisqldev-42 206 Reputation points
commented 2024-03-28T05:42:35.0133333+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
1 answer

Json object transform

hi, I wanted to form a json object in a derived Column as following: "categories": [ { "id": 9 }, { "id": 14 } ] the id's are stored in diff columns. ex: Can you please guide me Thanks, Pratisti…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-13T18:24:06.35+00:00
Pratisti Satardekar 90 Reputation points
commented 2024-03-28T05:38:23.94+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
2 answers

Copy Activity error in Synapse Pipelines

I am encountering an error when attempting to copydata to SQL Server from a JSON file located in the Datalake. Specifically, we've encountered the following error: {     "errorCode": "2200",     "message":…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,315 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,251 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,890 questions
SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
12,482 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,338 questions
asked 2024-03-14T14:37:37.3333333+00:00
Abhra Mitra 0 Reputation points
commented 2024-03-28T05:36:32.2433333+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee