10,180 questions with Azure Data Factory tags

Sort by: Updated
1 answer One of the answers was accepted by the question author.

Data factory pipeline shows success when copying csv to sql azure but data does not appear in the table

When running a data factory pipeline to copy a csv file to sql azure, the job shows as a success and confirms that the proper amount of rows were copied but when querying the table in the database, the data does not show.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-12T20:57:46.097+00:00
Robert Anderson 116 Reputation points
accepted 2020-11-17T14:58:58.18+00:00
Robert Anderson 116 Reputation points
2 answers

Azure Data Factory - Paypal - Lookup error

I am trying to execute a lookup to read records from Paypal: This is the query I am running Select Id,Transactions_Index,Sale_Create_Time,Sale_Parent_Payment,Sale_Amount_Details_Subtotal, Sale_Id from Payment_Transactions_Sale Where Sale_Create_Time…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-16T04:59:23.097+00:00
Ana Del Campo 121 Reputation points Microsoft Employee
answered 2020-11-17T01:22:19.28+00:00
Ana Del Campo 121 Reputation points Microsoft Employee
2 answers One of the answers was accepted by the question author.

Azure data factory - query pipeline by data annotations

Is there a way to programmatically query (using .net SDK) list of data factory pipelines by data annotations? I can set data annotation when I run pipeline as explained here, but not sure how to query pipelines or filter pipelines using the same?…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-10-23T21:37:28.577+00:00
JAY VORA 21 Reputation points
commented 2020-11-16T23:38:22.617+00:00
MartinJaffer-MSFT 26,061 Reputation points
1 answer

How we can get uptime for all the resources in azure.

Hi guys, I have to check the uptime for the resources in Azure. As for some resources, we can check directly from the Azure portal. I'm able to see uptime for the storage account and SQL database but how can I get the uptime for other resources such…

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,539 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-07T07:03:54.573+00:00
Ashima Gupta 66 Reputation points
commented 2020-11-16T22:45:46.163+00:00
HimanshuSinha-msft 19,386 Reputation points Microsoft Employee
0 answers

Expiry Datetime incorrect for Tumbling Window Trigger Catchup Run?

I have a pipeline that primarily has 2 copy activities . The sink for the copy activities is ADLS Gen 1, so it also sets an expiry datetime for each of the files copied over. It uses a tumbling window trigger with a self-dependency. Recently, the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-06T19:26:33.123+00:00
Rasika Athavale 1 Reputation point Microsoft Employee
commented 2020-11-16T22:40:37.95+00:00
HimanshuSinha-msft 19,386 Reputation points Microsoft Employee
1 answer

Logs are missing in the Log Analytics Workspace - Azure Data Factory starting 10/23/2020

Hello, Our ADF and the log analytic workspace are hosted under West US2, same as our ADF, and since 10/23, we’ve noticed several pipeline runs do not come with the Succeeded or Failed log, but just InProgress or Queued. It threw off our query since…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-05T18:59:56.11+00:00
Helen Ho 1 Reputation point Microsoft Employee
answered 2020-11-16T22:28:52.45+00:00
Helen Ho 1 Reputation point Microsoft Employee
2 answers

Data Factory Loading multiple csv files. Is there any way of dealing with column header differences?

I have csvs in a datalake. Im coping them into SQL using the copy activity the File Path type is a wildcard file path Very occasionally there are slight differences between the column headers. Is there any way that this can be dealt with rather…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-05T12:56:42.187+00:00
Debbie Edwards 521 Reputation points
commented 2020-11-16T22:16:54.58+00:00
HimanshuSinha-msft 19,386 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Getting TlsVersionNotPermitted error when SSIS package executing from ADF

Hi, I have created one SSIS package which is reading data from Storage account and putting that into Azure DB. This is working perfectly well when executing from Visual Studio. But when I am triggering it after deployment in ADF, getting below error. …

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,426 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,937 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-12T18:01:00.723+00:00
Vikas 21 Reputation points
commented 2020-11-16T18:10:09.6+00:00
MartinJaffer-MSFT 26,061 Reputation points
1 answer One of the answers was accepted by the question author.

Query Azure Function HTTP Get from Azure Data Factory

I have a function like: public static async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest req, ILogger log) { string dateFromString =…

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,662 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-13T12:22:03.513+00:00
Christian Hansen 21 Reputation points
accepted 2020-11-16T13:33:21.307+00:00
Christian Hansen 21 Reputation points
2 answers

Azure data factory web activity body

How do I use parameters in the body of Push method in data factory? It seems to allow static values or a parameter but not both. Say I want my message body as {"count" :@Anonymous } But this isn't working, any ideas

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-10T04:14:56.33+00:00
Accelerator 1 Reputation point
commented 2020-11-16T06:31:20.42+00:00
HarithaMaddi-MSFT 10,136 Reputation points
1 answer

Is it possible to rename file name with MD5 hash encoding in Azure data factory?

I have this scenario where I have a blob with multiple binary files and I would like to copy these files into another blob storage using Azure data factory. The issue is my destination file name should be MD5 encoded . Now I would like to know if it…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-10-05T12:46:22.937+00:00
ankit kumar 101 Reputation points
commented 2020-11-14T00:03:32.147+00:00
MartinJaffer-MSFT 26,061 Reputation points
1 answer

Azure data factory Cosmos Db delete record

hello After run an ETl process through ADF I need to run a bluk delete on cosmos DB . I know that ADf doesn't suppor this feature but it could be possible to do it through an azure function. I don't know how to deal with a bulk delete(based on a…

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,662 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,539 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-12T13:03:33.18+00:00
Diana Milena Coronado Granados 1 Reputation point
answered 2020-11-13T02:18:55.527+00:00
Navtej Singh Saini 4,221 Reputation points Microsoft Employee
1 answer

Linked Service with ADF Connection Failing

I am attempting to connect by Azure Data Lake Analytics account with Azure Data Factory via a Linked Service. I have added the required information in the Edit linked service blade, created an app and a secret for the Service Principal ID and key, set…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
Azure Data Lake Analytics
asked 2020-11-02T17:23:19.58+00:00
Supply Chain Glass 1 Reputation point
commented 2020-11-12T20:29:42.27+00:00
Saurabh Sharma 23,791 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

More than two time trigger duplicate records inserted in increment load.

Hi Team, I am follow below link create Pipeline i have haapy answer but I have problem when i am insert more two time for day its dupplicate records inserted. please anybody help how to avoid duplicate record in sink table my client didnt…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-09T16:01:42.427+00:00
Shanvitha 221 Reputation points
commented 2020-11-12T12:09:32.707+00:00
Shanvitha 221 Reputation points
1 answer

AZURE APP, AZURE POSTGRESQL, EMAIL

I have .net app deployed on azure web app, connected with azure postgres db. I push SAP BW data into azure postgres using azure data factory. Data processed in db and final processed/ not processed output is stored in db table. I want to trigger…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
Azure Stack Hub
Azure Stack Hub
An extension of Azure for running apps in an on-premises environment and delivering Azure services in a datacenter.
183 questions
Azure Database for PostgreSQL
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,399 questions
asked 2020-11-11T10:52:48.123+00:00
Charmi 1 Reputation point
answered 2020-11-12T11:38:33.98+00:00
KalyanChanumolu-MSFT 8,316 Reputation points
1 answer

Copy activity output entities and Query activity of ADW

HI All, We have a pipeline that contains copy activity which runs query on Azure SQL DB and generates a blob file. PFB screenshot of copy activity output 1) Wanted to know how Copy duration mentioned in the output is generated. Is it combination of…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,687 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-06T14:15:41.207+00:00
Priya Jha 871 Reputation points
commented 2020-11-12T06:51:07.097+00:00
HarithaMaddi-MSFT 10,136 Reputation points
0 answers

Azure Data Factory Issue SFTP Sink tasks fails after copying 19 files

Hi there, I have an Azure Data Factory Pipeline that has a task that is copying CSV files to a SFTP server. { "dataRead": 3570267, "dataWritten": 1491182, "filesRead": 31, …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-06T12:45:09.263+00:00
Nathan Dear 1 Reputation point
commented 2020-11-12T04:07:58.06+00:00
Nathan Dear 1 Reputation point
1 answer

Unable to query data factory pipeline runs

I am trying to query against azure pipelines runs for a given data factory pipeline resource. The account I am using has build in role Contributor role to the data factory resource. But I get a permission error when I try to query running pipelines…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-09T17:40:52.157+00:00
JAY VORA 21 Reputation points
commented 2020-11-12T01:56:44.467+00:00
Saurabh Sharma 23,791 Reputation points Microsoft Employee
2 answers One of the answers was accepted by the question author.

How to pass a parameter with more than one key field?

All I have a working pipeline, whereby the parameter LogicalKey, type string in pipeline as well as data flow, is filled with one value. This is the logical key of my Sink. Everything works fine. See ‘Old settings’ Now I changed the parameter. It…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-10-21T14:02:19.403+00:00
Poel van der, RE (Ron) 431 Reputation points
commented 2020-11-11T08:04:08.083+00:00
HarithaMaddi-MSFT 10,136 Reputation points
2 answers

Add transformation logic dynamically to Data Flow.

We have a Web App which the user uses to upload the file that need to be imported to Azure SQL Database via Azure Data Factory pipeline. Prior to uploading, the Web App provide a functionality where in the user can do the mapping of file columns to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
asked 2020-11-09T21:04:24.19+00:00
jigsm 236 Reputation points
commented 2020-11-11T00:22:23+00:00
Mark Kromer MSFT 1,146 Reputation points