ADF data flow json sometimes array and sometimes object
I think it relates to this issue [http://michaelcummings.net/mathoms/using-a-custom-jsonconverter-to-fix-bad-json-results/] Our procedure was like below: Create a Copy Activity in ADF pipeline: calling workday API, and sink response in Azure Data Lake…
How to execute java command on Azure Virtual Machine from Azure Data Factory
I have executable fat jar deployed on Azure Virtual Machine (LINUX). I want to run this jar using java command from Azure Data Factory. Basically, I want to create a pipeline to call several java commands, one at a time in pipeline. I tried to create…
Filtering data for last 24 months in Mapping Data Flows
How can I filter data using Mapping Data Flows for the last 24 months, starting from the max date and going back? The date column is in the mm/dd/yyyy format.
How to execute java command on Azure Virtual Machine from Azure Data Factory
I have executable fat jar deployed on Azure Virtual Machine (LINUX). I want to run this jar using java command from Azure Data Factory. Basically, I want to create a pipeline to call several java commands, one at a time in pipeline. I tried to create…
How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory ?
How to move data from one table to another table within same workspace of LOG ANALYTICS using Azure Data Factory 2.from one table in log analytics workspace 1 to another table in log analytics 2 workspace using Azure Data Factory ?
Copy Activity allows HTTP Post to get Excel but when debugging it fails
So I am trying to import an excel file from a website that uses a HTTP Post to pull the file. I have configured the linked service, dataset and then the Copy Activity. In the copy activity I am able to specifiy it use a Post method. When I click…
I get the following error while Sinking data in Azure: ErrorCode: 'EndpointUnsupportedAccountFeatures'. Message: 'This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.
When I try to select the file path under Linked service while sinking the data (ADLS Gen2), I get this error: "ErrorCode: 'EndpointUnsupportedAccountFeatures'. Message: 'This endpoint does not support BlobStorageEvents or SoftDelete. Please disable…
Moving 10TB of SharePoint data to Azure blob storage (Contains files with 5GB+ in size)
Greetings, I hope this message finds you well. I'm currently working on a solution for transferring over 10TB of data from a "Preservation Hold Library" to "Azure Blob Storage" for archiving SharePoint online deleted files. Upon…
How to define pipeline return value in SetVariableActivity in .NET project?
Hi, I'm implementing SetVariableActivity in my .NET project. I wanted to define the return value pipeline but it is defined as DataFactoryElement<BinaryData> Value although it is an array of values. How can I define it in my project? I…
In ADF Copy Activity- Synapse Sink giving SQL DW Error : The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction
When we are Copying data from ADLS to Azure Synapse Tables , we are getting following error since Friday . We are not able to write on Azure synapse from Azure pipeline Copy Activity. Below is the error. Note : Earlier it was working fine , we are…
How to post multi-GB data file to REST API using Azure Data Factory?
Hello, I am currently working on a data ingestion pipeline which needs to process some larger files (up to 5GB) and post them to a REST API. The files are stored in Azure Blob Storage. The data cannot be sent in chunks and so needs to be sent whole,…
Row count of parquet files
Hello, How can we get the row count of a parquet file? I want do a conditional copy activity based on the row count of the source data file. I tried with the lookup condition and used a if condition, based on output of lookup activity the if condition…
AdlsGen2OperationFailed - Error
Hi, I could migrate a single table from the MS SQL Server Database OnPremises to Azure Data Lake Storage in the data factory and the pipeline is running successfully. However, when I try to migrate multiple tables from MS SQL Server Database OnPremises…
MCT exception cases
Hi All, I have a colleague who is Microsoft Azure MVP and she conducts multiple free trainings and is a global speaker. So, can she apply directly for MCT certifications or is there a need for her to do the mandatory 3rd party vendor training?
Can I connect directly to CosmosDB from ADF using the key vault?
I am trying to set up an azure key vault key to connect from Azure Data Factory directly to CosmosDB. I am following this link but it seems to be for a web app service. https://learn.microsoft.com/en-us/azure/cosmos-db/store-credentials-key-vault This…
ADF Dataflow - dervied column, Projection, Schema
Hi, I have observed that ADF dataflow derived column or any other transformations require projection to be available in previous transformations. For eg, I have a derived column after source. If I have to add a derived column 'xyz' as 'column1+5',…
azure datalake compute engine
Hi friends,I have read that A data lake is simply storage—unlike a relational data warehouse, there is no compute engine associated with it, but in azure datalake documentation I read that it has hadoop and spark ccomute, so please help me to…
ADF storage event trigger pipeline is not triggered when file is SFTP to Azure Blob Storage.
Hi, I have a file coming from on-prem app through SFTP to Azure Blob Storage container. I set up an event storage trigger on my pipeline. However the pipeline is not getting started when the file (blob) created on Azure storage container through SFTP. On…
Data lake going to be replaced by delta lake
Hi friends, recently I have seen an architecture that uses Delta Lake for bronze, silver, and gold layers and ADF as a whole ingestion and movement service. Data validation is mostly done by using Azure data bricks in the silver layer. And each layer was…
Failure happened on 'destination' side. ErrorCode=DeltaInvalidCharacterInColumnName.
This exception occurred when I use the pipeline of data factory to copy data from sql server to lakehouse. But I didn't find any problems with the raw data.