Creating a pipeline for data transfer from SQL Server to blob storage
I'm looking to create a single pipeline that can transfer data from a SQL Server database to BLOB storage in a Parquet format. This pipeline should go through a table that lists the connection string and SQL commands needed for each table or Parquet file…
batch output of a lookup and then pass to a web action through for each loop
I do a lookup on a table to receive some ID's. Then I take these ID's and put them in the body of a http POST request. Right now I have it like this: The problem is that each row would be it's own request, and I don't want to send +100k requests at the…
private endpoint service endpoint private link and public endpoint
hi friends I am confused with private endpoint service endpoint private link and public endpoint why and when to use this what scenario.please give a practical example while explaining
Azure Data Flow connection error to Dynamics 365 but pipeline can
I have a pipeline that has a connection to a table in Dynamics 365 which works fine. I need to create a dataflow as I need to lookup a value in the source that then replaces the value with a guid to load into Dynamics 365 but when I do a data preview in…
In ADF using HDFS linked service my copy file activity throws the following error
Hi, I have an issue using ADF with HDFS linked service. I created a HDFS connection then a copy acitivity from HDFS to Azure Data Lake gen2. The source is a CSV file and the copy format is binary. When I run the pipeline I get the following error: …
How to create the Data set, pipeline and run it, for excel format using C sharp for Data factory
I want to create and run my pipeline from the repo using C#. my code is string postgreSQLLinkedServiceName = "PostgreSQLLinkedService"; LinkedServiceResource postgreSqlLinkedService = new LinkedServiceResource( …
Unable to load data from Dynamics 365 CRM sales table to MS-SQL
I was trying to load data from Dynamics 365 Sales to MS-SQL using Using Azure Data Factory Copy data tool, Metadata-driven copy task. The type of load I am using is Delta Load/Incremental Load. Facing this issue: Failure happened on 'Source' side.…
how to flatten the nested json array into csv file using ADF pipeline
how to flatten the nested json array file into csv file using ADF pipeline. using below example i need to get the each table details in the excel/csv file. need the file like these below using below ex json Ex: { "Table1":…
Data flow sink preview is sorted but not sorted in the outputted tsv sink file
I have a data flow in data factory that gets data from a database and transforms some columns and then outputs it on a tsv sink file. In this data flow the data is sorted by a certain column. The sink preview is sorted correctly but when I open the…
Tumbling window trigger in copy pipeline seems to be copying more data than present in the window.
I used the 'Ingest' option in Azure data factory to enable continuous ingestion from one table to another table using Tumbling window trigger. I set the interval in the Tumbling window trigger to 15 mins and the window start and end time for each trigger…
what would be the reason where credential creation for linked service in ADF is taking time forever?
Hello There, I have an issue creating 'credential' in linked-service for 'azure blob storage' where after giving all the information including resource-group, resource-type, 'user-assigned-managed-identity', and hit the 'create' button, the creation…
Schedule Trigger not working even after week creation?
Schedule trigger created for pipeline to run every 15 min on daily basis. And trigger created week back but still trigger not happening. What could be the problem?
ERROR: {"code": "InvalidTemplate", "message": "Deployment template validation failed}
Hi Team, We are trying to deploy to ADF instance using Jenkins as CI/CD platform. We are following the documentation for reference: …
How to trigger an adf pipeline of one adf resource in another adf resource pipeline
We required to call a pipeline from one ADF resource to another ADF resource. I tried to export and import json pipeline format but could not import same in other ADF resource due to asking Git configuration. I tried through WEB activity but failed by…
Copy data activity from synapse serverless sql pool to ADLS gen 2
Hi, I have synapse serverless sql pool configured to ADLS Gen 2 account where the container is filled with files from D365 using synapse link. I need to build datawarehouse in Azure. We have very tight restrictions on budget and don't want to go…
How to skip error and continue unzip process using Azure Data Factory copying activities
I have large zip file that I transferred from SFTP to blob storage. I have created a copy activity to unzip the files from one blob storage (source dataset) to another blob storage (sink dataset). The unzip process has the error : file operation is…
Json object transform
hi, I wanted to form a json object in a derived Column as following: "categories": [ { "id": 9 }, { "id": 14 } ] the id's are stored in diff columns. ex: Can you please guide me Thanks, Pratisti…
Copy Activity error in Synapse Pipelines
I am encountering an error when attempting to copydata to SQL Server from a JSON file located in the Datalake. Specifically, we've encountered the following error: { "errorCode": "2200", "message":…
How to access an Oracle database on a different server from ADF
Hi, I have Azure Data Factory (ADF) configured to work with Azure SQL Database. However, I also need to access data from a separate Oracle database located on another server. How to set up a connection in ADF to extract data from this Oracle database?…
Running a ForEach activity in parallel mode
Hi, I've implemented a pipeline having a Lookup activity to read a Synapse table to get the parameters to pass a stored procedure inside the next ForEach activity. Inside the ForEach activity I've use some Set variable activities to get the parameter…