Is there a way to restrict multiple instances of an ADF pipeline on Same path of event based trigger?
I'm running Spark notebooks in Synapse using an event-based ADF pipeline. If any notebook gets triggered twice, I encounter a conflict error. I understand that I can avoid concurrent execution of the pipeline by setting the concurrency to 1. However, I…
Unable to resolve Azure database for mysql name from Data Factory
I am creating a linked service from my Azure Data Factory environment, to access my Azure Database for Mysql database. I select "Azure database for Mysql", and I can see all my mysql servers. I select "From Azure subscription" as the…
Obvious behavior of dynamic mapping in ADF
I have a single copy activity that moves data from a source parquet file to an azure sql database. When I used auto mapping and tried to change the name of source or sink field it failed, however when I copy the json mapping generated and change one of…
How to forecast 12-month demand for medicines using Azure AutoML time-series?
I need to predict the monthly demand for medicines for the next 12 months. I have historical data from 2018 to 2023 for each medicine, including a unique drug identifier (Scode) and the date of consumption. Using this data, I aim to forecast the demand…
Azure Data Factory - Copy data activity failed: Failure happened on 'Source' side
Operation on target Copy data1 failed: Failure happened on 'Source' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path:…
Not able to connect to on premise sql server using self integration run time in azure data factory
I am able connect to on premise sql server using SSMS tool but while connecting through azure data factory using self hosted integration runtime and linked service getting below error. No mistakes in user name and password .Please help me on this. { …
Azure Data Factory: Fetch xml to return all the attributes not working properyl
Hi, Thanks in advance for your response. I am using copy activity to copy the data from one environment to another environment in dynamics 365. If I dint use any filter in the fetch xml query, it works fine. Otherwise i am getting missing key…
Salesforce Bulk API Connector
I am using SOQL instead of ObjectApiName. The SOQL is generated using parameter correctly. The older legacy connector works fine for the same pipeline but the new one is giving this error msg: …
i got stuck on deployment is in progress when creating data factory
this is the message that appear to me : status : InternalServerError { "status": "Failed", "error": { "code": "GatewayAuthenticationFailed", "message": "Gateway…
Copy activity throwing an error, when a auto incremented column is there on the Sink side
Hello Team, I am trying to copy data from Amazon S3 to Azure SQL database. On Azure SQL side I have set an autoincrement column(Row_Index) in a table, however the copy activity throws an error that "Column 'Row_Index' does not allow…
Except Unique Assert Transformation in Dataflow
I am trying to use Asset transformation in dataflow to check the dups in source. I am not able to get the correct result when using parameters that will have single or multiple columns as primary keys (example shared below) when using below expression. …
How to fix "Failure happened on 'Source' side A task was canceled" error
Data load from Salesforce object with 1million records fails with the below error in Azure Data Factory: Failure happened on 'Source' side. 'Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,' The data load…
Default TImezone in the Dataflow
Hi Team, Is there anyway to change the default timezone in the datafactory? Currently, when I am creating a column(loadtime) with currentTimestamp() in derived column transformation in the dataflow, then I get the time in UTC. But, I want the default…
Salesforce V2 Link Service doens't work. error message: a task was canceled Source=mscorlib
I am migrating the ADF salesforce connector to the new V2 connector and now testing it. When testing the connection for the link service, it is good. However, it failed in the copy activity. with message: Failure happened on 'Source' side.…
How can we run the different level of folders in a heirarchy in batches?
I have a doubt if we can approach something like this in ADF. I have created a for each activity where I'm iterating through an array of folders. Let's say I'm passing folder I'd of A and then I'm calling a pipeline in for each that will return sub…
Delete the file from SharePoint location
Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…
Creating new linked services returns SqlFailedToConnect
I'd like to start by first stating that this is a continuation of my efforts to connect an Azure SQL DB to Data Factory via the Linked services. Creating new linked services returns InvalidCredentialReference My new error is below in the azure portal in…
Invalid property name error in ADF Copy Data Activity
We have created pipelines for copying data from sharepoint. We have configured our copy data activity to store the file with some metadata also like author name, title, link, last modified, etc. Now for some files I'm getting below…
Hubspot connector doesn't work for deal properties: 414 Request-URI Too Large
I am getting this error message when trying to copy data from Deal properties from Hubspot: Failure happened on 'Source' side. ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR [HY000]…
Data factory Hubspot connector returns "414 Request-URI Too Large" on some endpoint(s)
Hey, we started getting an error with Data factory utilizing the Hubspot connector. " Operation on target Copy_7eo failed: Failure happened on 'Source' side.…