Data factory pipeline shows success when copying csv to sql azure but data does not appear in the table
When running a data factory pipeline to copy a csv file to sql azure, the job shows as a success and confirms that the proper amount of rows were copied but when querying the table in the database, the data does not show.
Azure Data Factory - Paypal - Lookup error
I am trying to execute a lookup to read records from Paypal: This is the query I am running Select Id,Transactions_Index,Sale_Create_Time,Sale_Parent_Payment,Sale_Amount_Details_Subtotal, Sale_Id from Payment_Transactions_Sale Where Sale_Create_Time…
Azure data factory - query pipeline by data annotations
Is there a way to programmatically query (using .net SDK) list of data factory pipelines by data annotations? I can set data annotation when I run pipeline as explained here, but not sure how to query pipelines or filter pipelines using the same?…
How we can get uptime for all the resources in azure.
Hi guys, I have to check the uptime for the resources in Azure. As for some resources, we can check directly from the Azure portal. I'm able to see uptime for the storage account and SQL database but how can I get the uptime for other resources such…
Expiry Datetime incorrect for Tumbling Window Trigger Catchup Run?
I have a pipeline that primarily has 2 copy activities . The sink for the copy activities is ADLS Gen 1, so it also sets an expiry datetime for each of the files copied over. It uses a tumbling window trigger with a self-dependency. Recently, the…
Logs are missing in the Log Analytics Workspace - Azure Data Factory starting 10/23/2020
Hello, Our ADF and the log analytic workspace are hosted under West US2, same as our ADF, and since 10/23, we’ve noticed several pipeline runs do not come with the Succeeded or Failed log, but just InProgress or Queued. It threw off our query since…
Data Factory Loading multiple csv files. Is there any way of dealing with column header differences?
I have csvs in a datalake. Im coping them into SQL using the copy activity the File Path type is a wildcard file path Very occasionally there are slight differences between the column headers. Is there any way that this can be dealt with rather…
Getting TlsVersionNotPermitted error when SSIS package executing from ADF
Hi, I have created one SSIS package which is reading data from Storage account and putting that into Azure DB. This is working perfectly well when executing from Visual Studio. But when I am triggering it after deployment in ADF, getting below error. …
Query Azure Function HTTP Get from Azure Data Factory
I have a function like: public static async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest req, ILogger log) { string dateFromString =…
Azure data factory web activity body
How do I use parameters in the body of Push method in data factory? It seems to allow static values or a parameter but not both. Say I want my message body as {"count" :@Anonymous } But this isn't working, any ideas
Is it possible to rename file name with MD5 hash encoding in Azure data factory?
I have this scenario where I have a blob with multiple binary files and I would like to copy these files into another blob storage using Azure data factory. The issue is my destination file name should be MD5 encoded . Now I would like to know if it…
Azure data factory Cosmos Db delete record
hello After run an ETl process through ADF I need to run a bluk delete on cosmos DB . I know that ADf doesn't suppor this feature but it could be possible to do it through an azure function. I don't know how to deal with a bulk delete(based on a…
Linked Service with ADF Connection Failing
I am attempting to connect by Azure Data Lake Analytics account with Azure Data Factory via a Linked Service. I have added the required information in the Edit linked service blade, created an app and a secret for the Service Principal ID and key, set…
More than two time trigger duplicate records inserted in increment load.
Hi Team, I am follow below link create Pipeline i have haapy answer but I have problem when i am insert more two time for day its dupplicate records inserted. please anybody help how to avoid duplicate record in sink table my client didnt…
AZURE APP, AZURE POSTGRESQL, EMAIL
I have .net app deployed on azure web app, connected with azure postgres db. I push SAP BW data into azure postgres using azure data factory. Data processed in db and final processed/ not processed output is stored in db table. I want to trigger…
Copy activity output entities and Query activity of ADW
HI All, We have a pipeline that contains copy activity which runs query on Azure SQL DB and generates a blob file. PFB screenshot of copy activity output 1) Wanted to know how Copy duration mentioned in the output is generated. Is it combination of…
Azure Data Factory Issue SFTP Sink tasks fails after copying 19 files
Hi there, I have an Azure Data Factory Pipeline that has a task that is copying CSV files to a SFTP server. { "dataRead": 3570267, "dataWritten": 1491182, "filesRead": 31, …
Unable to query data factory pipeline runs
I am trying to query against azure pipelines runs for a given data factory pipeline resource. The account I am using has build in role Contributor role to the data factory resource. But I get a permission error when I try to query running pipelines…
How to pass a parameter with more than one key field?
All I have a working pipeline, whereby the parameter LogicalKey, type string in pipeline as well as data flow, is filled with one value. This is the logical key of my Sink. Everything works fine. See ‘Old settings’ Now I changed the parameter. It…
Add transformation logic dynamically to Data Flow.
We have a Web App which the user uses to upload the file that need to be imported to Azure SQL Database via Azure Data Factory pipeline. Prior to uploading, the Web App provide a functionality where in the user can do the mapping of file columns to…