9,267 questions with Azure Data Factory tags

Sort by: Updated
0 answers

ADF data flow json sometimes array and sometimes object

I think it relates to this issue [http://michaelcummings.net/mathoms/using-a-custom-jsonconverter-to-fix-bad-json-results/] Our procedure was like below: Create a Copy Activity in ADF pipeline: calling workday API, and sink response in Azure Data Lake…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-13T00:55:18.3933333+00:00
Zhenqi Liu 0 Reputation points
commented 2024-03-19T01:46:20.4566667+00:00
Zhenqi Liu 0 Reputation points
0 answers

How to execute java command on Azure Virtual Machine from Azure Data Factory

I have executable fat jar deployed on Azure Virtual Machine (LINUX). I want to run this jar using java command from Azure Data Factory. Basically, I want to create a pipeline to call several java commands, one at a time in pipeline. I tried to create…

Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
6,931 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-19T01:32:30.7833333+00:00
satya 0 Reputation points
0 answers

Filtering data for last 24 months in Mapping Data Flows

How can I filter data using Mapping Data Flows for the last 24 months, starting from the max date and going back? The date column is in the mm/dd/yyyy format.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,305 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,581 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,347 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,212 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-19T01:29:58.9966667+00:00
ADITYA SINGH 0 Reputation points
0 answers

How to execute java command on Azure Virtual Machine from Azure Data Factory

I have executable fat jar deployed on Azure Virtual Machine (LINUX). I want to run this jar using java command from Azure Data Factory. Basically, I want to create a pipeline to call several java commands, one at a time in pipeline. I tried to create…

Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
6,931 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-19T01:29:08.2466667+00:00
satya 0 Reputation points
1 answer

How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory ?

How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory 2.from one table in log analytics workspace 1 to another table in log analytics 2 workspace using Azure Data Factory ?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-14T00:37:19.5866667+00:00
Rishabh Srivastava 40 Reputation points Microsoft Employee
commented 2024-03-18T23:14:18.87+00:00
Rishabh Srivastava 40 Reputation points Microsoft Employee
0 answers

Copy Activity allows HTTP Post to get Excel but when debugging it fails

So I am trying to import an excel file from a website that uses a HTTP Post to pull the file. I have configured the linked service, dataset and then the Copy Activity. In the copy activity I am able to specifiy it use a Post method. When I click…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T22:51:22.9666667+00:00
Steve Dunker 20 Reputation points
1 answer

I get the following error while Sinking data in Azure: ErrorCode: 'EndpointUnsupportedAccountFeatures'. Message: 'This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.

When I try to select the file path under Linked service while sinking the data (ADLS Gen2), I get this error: "ErrorCode: 'EndpointUnsupportedAccountFeatures'. Message: 'This endpoint does not support BlobStorageEvents or SoftDelete. Please disable…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,305 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T18:50:49.3766667+00:00
Priyal 0 Reputation points
answered 2024-03-18T21:46:22.67+00:00
KarishmaTiwari-MSFT 17,752 Reputation points Microsoft Employee
0 answers

Moving 10TB of SharePoint data to Azure blob storage (Contains files with 5GB+ in size)

Greetings, I hope this message finds you well. I'm currently working on a solution for transferring over 10TB of data from a "Preservation Hold Library" to "Azure Blob Storage" for archiving SharePoint online deleted files. Upon…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
9,334 questions
Windows Server PowerShell
Windows Server PowerShell
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.PowerShell: A family of Microsoft task automation and configuration management frameworks consisting of a command-line shell and associated scripting language.
5,306 questions
asked 2024-03-12T07:06:19.6933333+00:00
Aman Varama 0 Reputation points
commented 2024-03-18T21:21:11.27+00:00
BhargavaGunnam-MSFT 24,726 Reputation points Microsoft Employee
0 answers

How to define pipeline return value in SetVariableActivity in .NET project?

Hi, I'm implementing SetVariableActivity in my .NET project. I wanted to define the return value pipeline but it is defined as DataFactoryElement<BinaryData> Value although it is an array of values. How can I define it in my project? I…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T10:35:28.33+00:00
OGINTZ Marina 85 Reputation points
commented 2024-03-18T20:41:13.8566667+00:00
BhargavaGunnam-MSFT 24,726 Reputation points Microsoft Employee
0 answers

In ADF Copy Activity- Synapse Sink giving SQL DW Error : The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction

When we are Copying data from ADLS to Azure Synapse Tables , we are getting following error since Friday . We are not able to write on Azure synapse from Azure pipeline Copy Activity. Below is the error.  Note : Earlier it was working fine , we are…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,212 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T07:52:06.1533333+00:00
Kapoor, Sipra (S) 0 Reputation points
commented 2024-03-18T20:03:29.56+00:00
BhargavaGunnam-MSFT 24,726 Reputation points Microsoft Employee
0 answers

How to post multi-GB data file to REST API using Azure Data Factory?

Hello, I am currently working on a data ingestion pipeline which needs to process some larger files (up to 5GB) and post them to a REST API. The files are stored in Azure Blob Storage. The data cannot be sent in chunks and so needs to be sent whole,…

Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
872 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T12:42:07.5466667+00:00
JegorsM 0 Reputation points
edited the question 2024-03-18T19:36:39.8166667+00:00
KarishmaTiwari-MSFT 17,752 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Row count of parquet files

Hello, How can we get the row count of a parquet file? I want do a conditional copy activity based on the row count of the source data file. I tried with the lookup condition and used a if condition, based on output of lookup activity the if condition…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2022-02-23T08:40:00.77+00:00
M. Adil 26 Reputation points
accepted 2024-03-18T18:13:19.1633333+00:00
M. Adil 26 Reputation points
0 answers

AdlsGen2OperationFailed - Error

Hi, I could migrate a single table from the MS SQL Server Database OnPremises to Azure Data Lake Storage in the data factory and the pipeline is running successfully. However, when I try to migrate multiple tables from MS SQL Server Database OnPremises…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T17:25:18.2033333+00:00
Sivakumar Rajabather 0 Reputation points
edited the question 2024-03-18T17:27:43.4066667+00:00
Sivakumar Rajabather 0 Reputation points
1 answer

MCT exception cases

Hi All, I have a colleague who is Microsoft Azure MVP and she conducts multiple free trainings and is a global speaker. So, can she apply directly for MCT certifications or is there a need for her to do the mandatory 3rd party vendor training?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T12:47:58.1833333+00:00
Priya Jha 866 Reputation points
answered 2024-03-18T16:38:21.63+00:00
Suba Balaji 11,021 Reputation points
3 answers

Can I connect directly to CosmosDB from ADF using the key vault?

I am trying to set up an azure key vault key to connect from Azure Data Factory directly to CosmosDB. I am following this link but it seems to be for a web app service. https://learn.microsoft.com/en-us/azure/cosmos-db/store-credentials-key-vault This…

Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,068 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,384 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-06T17:59:38.0433333+00:00
Andrew Hopkinson 6 Reputation points
commented 2024-03-18T16:34:04.8066667+00:00
Andrew Hopkinson 6 Reputation points
0 answers

ADF Dataflow - dervied column, Projection, Schema

Hi, I have observed that ADF dataflow derived column or any other transformations require projection to be available in previous transformations. For eg, I have a derived column after source. If I have to add a derived column 'xyz' as 'column1+5',…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T15:54:42.3366667+00:00
Santhi Dhanuskodi 125 Reputation points
commented 2024-03-18T16:28:16.9766667+00:00
Suba Balaji 11,021 Reputation points
2 answers

azure datalake compute engine

Hi friends,I have read that A data lake is simply storage—unlike a relational data warehouse, there is no compute engine associated with it, but in azure datalake documentation I read that it has hadoop and spark ccomute, so please help me to…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,305 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-02-20T17:25:37.5633333+00:00
Anshal 1,586 Reputation points
commented 2024-03-18T16:26:01.5066667+00:00
KarishmaTiwari-MSFT 17,752 Reputation points Microsoft Employee
1 answer

ADF storage event trigger pipeline is not triggered when file is SFTP to Azure Blob Storage.

Hi, I have a file coming from on-prem app through SFTP to Azure Blob Storage container. I set up an event storage trigger on my pipeline. However the pipeline is not getting started when the file (blob) created on Azure storage container through SFTP. On…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,347 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2023-09-25T13:31:34.3466667+00:00
Shrivastava, Ravi-ZB 0 Reputation points
commented 2024-03-18T15:06:31.34+00:00
Jesús Eduardo Álvarez Aguilar 0 Reputation points
2 answers

Data lake going to be replaced by delta lake

Hi friends, recently I have seen an architecture that uses Delta Lake for bronze, silver, and gold layers and ADF as a whole ingestion and movement service. Data validation is mostly done by using Azure data bricks in the silver layer. And each layer was…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,305 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-17T10:35:43.8333333+00:00
Anshal 1,586 Reputation points
commented 2024-03-18T15:04:59.0766667+00:00
Azar 16,315 Reputation points
1 answer

Failure happened on 'destination' side. ErrorCode=DeltaInvalidCharacterInColumnName.

This exception occurred when I use the pipeline of data factory to copy data from sql server to lakehouse. But I didn't find any problems with the raw data.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,305 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,267 questions
asked 2024-03-18T13:42:24.5+00:00
Xavier Yan 0 Reputation points
edited the question 2024-03-18T14:07:18.42+00:00
vipullag-MSFT 23,561 Reputation points Microsoft Employee