Azure Data Factory

How can I connect Azure Data Factory to SFTP server using ssh authentication

We are slowly moving all our ETL to ADF. We have few SFTP servers and we would like to connect those SFTP servers from ADF. Can someone shed some light on SSH key authentication between ADF and SFTP. Thanks in Advance Nikhil

Processing Azure Data Factory Event Trigger Properties

I have a data factory which triggers based on storage blob event. In the triggered event, I see two properties TriggerTime and EventPayload. As I have need to read the Storage Blob related information I am trying to process the EventPayload in the Data…

Can we use Azure Data Factory to do the following?

Hi, there: We are Microsoft Skype for Business(SFB) and Teams data platform team, currently focus on SFB/Teams Audio/video/application sharing quality back end data collection and analysis. For every SFB and Teams Audio/video/application sharing…

Azure Data Factory: Missing data in the Monitor Dashboard for Data Factory Contributor role

Observed a disparity in the Monitor Dashboard view for the Azure Data Factory instance between: user with Owner role at subscription scope, and, user with Data Factory Contributor role at resource group scope From the account with Owner…

Data factory monitoring activity runs runtime consolidation report

Hello team, I have a master pipeline which has For Each and inside the loop, it has more than 10 activities. How/where to find the time taken of each activity run in consolidation. For ex:- Master For Each Iteration 1 Activity 1 Activity 2 …

Data Factory ForEach performance questions with REST API GET requests

I have a pipeline that has a Lookup activity that is querying a single column of data from an Azure SQL table, fetching the first 1000 rows of data. This input is then being used in a ForEach activity in order to use the values in a string…

Azure Data Factory: Link connection Sharepoint

Hello, i'm trying to link in Data Factory a Sharepoint and i have some problems, when i try to copy data i have this warning: The access token generated failed, status code: BadRequest, error message:…

How to send data from one Salesforce to another one?

I have followed documentation, but still get error: Operation on target Copy data1 failed: Failure happened on 'Sink' side.…

Is there any way to store json Array or json object in SQL as a string into Azure SQL datawarehouse or anything to convert the Array into string(in Azure Data Factory).

Is there any way to store json Array or json object in SQL as a string or anything to convert the Array into string. Currently I am using the Data factory to fetch the Azure API data and try to store the same into Azure SQL data warehouse, but some of…

Azure Data factory V2 pipeline trigger stuck with In-progress status?

Actually, we have one pipeline with 30 different schedules based on different pipeline parameter. Some of schedule executed every 10 min and some of executed once in a day. After setup the pipeline it running fine for one day. But after that suddenly…

azure data factory - java.nio.BufferOverflowException

Hi, I'm using copy activity to copy ORC files from DL Gen 2 to Synapse. Some files are running fine but other files fail with this error: What is the reason. I'm using : AutoResolveIntegrationRuntime "errorCode": "2200", …

Can't process ORC files in Data Factory : ErrorCode=ParquetJavaInvocationException

Hi, Our organisation uses ORC formatted files for our central file storage, in our data factory I am unable to process most of the ORC files, I am only able to process really small files (< 1mb) for all the other ORC files the pipeline fails to run. …

What will be the required value in the additional expression? It returns an error saying "conversion failed when converting date or time from character string.

ADF V2 failed activity

On June17th when an stored procedure activity failed, anything that was written to output (aka Print) in the stored procedure was included in the error in ADF. Since July 1st this no longer is included in the output. Is there a way we can get this…

Data Factory Salesforce objects not the same as Objects seen in Salesforce URL

I have created a LinkedService to connect to the company supplied Salesforce URL using the customised url, the supplied user id, password and have acquired the correct security token. This connection tests successfully. It is also the same parameters…

Spark Connector in ADF

Hi, I have created a spark connector to connect to azure data bricks. In copy activity source is spark connector and sink is Azure SQL DB. In spark Connector query, CreatedDate is being converted to String and throwing error where as it is timestamp…

Copy Activity data socurce preview error

Hi Team, Currently i have setup some tables move to dataLake using copy Activity pipeline, However i have realized few of tales have a issue when i connect it to copy activity source, According to error pop up as a below, Could you please check and…

How to checke pipeline cpu usage

Hi team , I have some pipelines. Whenever some pipelines run at that time cpu usage goes beyond 80. I have checked for cpu usage in task manager it is showing self-hosted ir process is running there which is taking too much cpu.. How can I determine…

Error while creating data factory

I have free subscription account and i m trying to create data factory using sandbox and getting error 'deny' Policy action.

Data Factory Variable Comparison not working as expected.

Hi All, I'm running a simple pipeline that takes an input variable, adds 1 day to it, and compares it to another variable. However, the comparison isn't working as I'm expecting it to. Variable boolTest is set to false after the update of the…