ADF pipeline stuck in "Queued" status while writing data in SFSC
I am writing to upsert / insert records in SFSC from sql server object . The connection to SFSC is working fine and successful. But when I run the pipeline the status is stuck in "Queued" . It never changes.
Copy Activity Sink to SFTP "useTempFileRename": false option not always working
Has anyone else experienced this issue, is it a bug? When using Copy Activity and ftp to a site which does not allow file rename I set "useTempFileRename": false. But even though ADF will say copy was successful the file sometimes is not…
protect JSON-file /ARM template Azure Data Factory
Hi, I have developed an ETL procedure for a specific ERP system in Azure Data Factory. I want to sell the entire procedure to customers so that they can use the ARM template in their own Azure tenant. However, I do not want them to make a copy of…
How to manually cancle a ADF pipline when an activity fails or completes?
Hi, I have a pipeline which has two concurrent list of activities running at the same time lets say the first activity has a flag which makes the until loop in the second list of activity to complete if flag is true. Now the problem is the flag is…
Error occurs at source transformation when trying to source csv file from folder in data lake storage gen2 while debugging the pipeline
While using mapping data flow activity, In source transformation I am trying to read a csv file from a folder in Azure data lake gen2. I am able to preview the data in "Data Preview" in source transformation but when trying to debug the…
Data Factory Pipeline to Azure SQL Server - fails on firewall IP
Hi Team, I'm not sure what I've done wrong here. I've got an Azure Data Factory pipeline that copies data from an on-prem SQL Server (using the Self-Hosted Integration Runtime) up into a database running on an Azure SQL Database. I can grab the…
ADF Copy activity output into multiple parquet snappy file?
In ADF copy activity output, pipeline is creating very large parquet file. Will it possible to split the output into multiple small parquet snappy files, so that synapse external table can use parallelism (polybase)? like Similar to DBrick data frame,…
[ADF] Azure Data Factory Copy Activity number of files
We've a simple ADF pipeline that copies data from an on premise sql server database (throught one IR node) to ADLS Gen 2 We are trying to optimize job duration/performance by writing multiple files sink side but the parallel copy parameters / DIU…
Alternative way to Deploy SSIS Package without Data factory
I need to deploy only few SSIS Package to Azure Database for that I cannot afford the Data factory, is there any Feasible way to deploy the package like using dtex option in Premises Server in Azure
data flow drops column in different data factory
I have a data flow which is not displaying a column on the last step the sink when I view the the data in Date Preview. The source has been pushed from for our repository. Checking the data flow JSON matches between the Dev data factory and the…
Please help me to fix this error 2200
"ErrorCode=UserErrorAzureMysqlNetBatchWriteTimeOut,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=batch write time out, more than…
Azure Data Factroy - select files from a folder based on a wildcard filename
I'm new to ADF and thought I'd start with something which I thought was easy and is turning into a nightmare! I'm trying to do the following. Copy files from a ftp folder based on a wildcard e.g. 'PN'.csv and sink into another ftp folder. (wildcard*…
ADF - Cosmos DB linked service - using managed identity
When creating a ADF linked service for cosmos db connection, I do not see a managed identity option. Only see Connection String / Azure Key Vault options to connect. Can we not connect from ADF to Cosmos db using managed identity. Thanks, Kothai.…
Azure Data Factory - Technical Bug when ingesting data from on-premise File System to Azure File Storage
Hi Team, I was working on a pipeline to move data from on-premise File System to Azure File Storage. I found a strange bug which is causing error when trying to create a Azure File Storage linked service using UI but it is working smoothly when…
Pipeline activities are executing twice
I've been noticing an issue over the past week where pipeline activities are being executed twice, with no apparent reason based on the pipeline design. For example, I have a pipeline that simply calls two sub-pipelines sequentially: For some…
Azure Data Factory - Failed to save dataflow. Error: Failed to fetch
Hi, I have the ADF data flow created in Git repository. After my modification in the data flow, when i click on save, getting the error message like 'Failed to save <data flow name>. Error: Failed to fetch'. Recently started facing this issue.…
ADF data Flow SQL Server as Sink
Hi I have one issue with the sql server database with the table as parameter on the data set: On the DataFlow the parameter not appear as option but of course gives a message that the parameter is not provided:
ADF web activity issue
seems web activity can't read HTTP header content type correctly. I am using WEB activity to refresh power BI dataset. it's working normal in ADF till 2020/08/30. I test the web API in local PostMam, it's working. so I think it's not Power…
Error Testing Connection from SSIS to ADLS Gen2
I'm trying to connect to ADLS Gen2 from SSIS using an Azure Storage Connection Manager using a service account & Service Principal authentication. I'm using the Key Vault client ID for the application ID and secret key for the Application key. I'm…
Azure SQL Database Port 1433 Workaround - Data Factory Enable Staging option not working
Hello, I have developed data pipelines and database to store data while I was working from home. However, I am not able to access the database in the corporate network. After some research I found that port 1433 Outbound should be open for Azure SQL…