Can't Copy file from SharePoint to Azure Store account
Hi, I want to copy file from SharePoint to Azure storage account. Follow the instruction here https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory#prerequisites But I continuous failed in the last step,…
Azure Data Factory V2 and Integration Runtime and Web Table proxy issue
I have a problem in Azure Data Factory with the Integration Runtime when using Web Table. This was working fine with version previous to v 5 av Integration Runtime. I was setting proxy configuration in Microsoft.Mashup.Container.NetFX40.exe.config…
data factory managed identity is not being identified as a trusted service by keyvault
We have an issue with an ADF pipeline, when attempting to reach a secret from a KV in the same RG, the connection fails with the following error: "Client address is not authorized, and caller is not a trusted service" The setup is made…
Why is FailureWebHook not executed in this case?
Please consider the following pipeline. The Execute Pipeline activity failed, so the ProcessTMONData notebook was skipped. Why was the failure webhook not triggered, as it's supposed to trigger when EITHER: ForEachEndpoint fails ProcessTMONData is…
Error during batchpoint invoke: Access to this resource is denied. Please check your ACL rules on the resource
Hello, I am getting a "Access to this resource is denied. Please check your ACL rules on the resource." when attempting to invoke a batch endpoint. I have tried both with Azure CLI and Azure Data Factory to invoke a batch endpoint, with both…
How to solve the error in ADF- Rest call failed with client error
I am trying to copy data from share point using graph API. I am able to see the data through postman (by providing the authorization through access token), but when I use the same URL inside the Rest API dataset (source) in a copy activity, I am getting…
ADLS connections with third party
Hi friends, we have ingested data from all the data sources and stored it in ADLS Gen2 but after that, there is a virtual layer in the form of Denodo. This virtualization tool only has views and once views are executed, it fetches the data from multiple…
Data flow Output metric Status as substatus instead of runstatus resulting error while capturing metrics for status using runstatus column for logging
Getting Status as Substatus instead of runstatus for DataFlow in Azure data factory, We are using this column to log the status of the pipelines for all data flows & pipelines, Few pipelines are getting failed for the same and getting fixed…
Staging of Source Data in Parquete vs SQL DB for DWH with ADF for Delta-Loads
Hi there, I have some "conceptual" question I would like to discuss. New DWH should is beeing build on Azure using ADF and SQL for Dimensions/Facts. Today, we have also running the landing on SQL DB. Also we are only loading delta datasets…
||Execution fail against sql server. Please contact SQL Server team if you need further support. Sql error number: 121. Error Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The sem
Hi, We have azure data factory pipeline and got error message below couple of times over week-end. ADF pipeline calls azure sql database stored procedure which failed with the message - stored procedure just removes rows from table. Intermittent and…
How to process this JSON structure
Hi, i get this structure back from a web activity. Showing only a part { "Response": "[\r\n {\r\n "CreationDate": "2024-01-07T00:00:00Z",\r\n "Operation": "ViewReport",\r\n "UserId":…
How to Handle Case Sensitivity in Dynamic JSON Input Keys ("A" vs "a") for Derived Columns in ADF Dataflow Pipeline?
I am working on a dataflow pipeline in ADF with an input dataset schema that includes a column 'A'. The dataflow processes various JSON files with dynamic schemas, where some JSON files use 'A' and others use 'a' as keys. Since the schema is case…
How to generate ETL pipeline for Python code which gives 2 JSON file in response
Requirement : I have python code in my local pc folder which returns 2 JSON file as a output (that code is using API and try to store the data of website in JSON file), I want to create automation for running that python code at particular time using…
Efficient Log Handling and Data Retention in Azure Data Factory and Databricks
I need to create a solution to send logs from Azure Data Factory to the Databricks Unity Catalog. I'm considering the following structure: Whenever an activity run results in either failure or success, the corresponding log will be sent to Azure Logic…
Google drive on Azure without GCP
Hello, I'm wondering what options I have to transfer Google Drive data to Azure without GCP access. I found that the Google Drive API and Google Storage API require a Google Cloud account. What other options do I have to securely connect to and transfer…
Inserting Error Logs from Data Factory into Databricks Delta Table (Catalog) - Streaming vs. Batch
Hi everyone, I'm working on a solution to capture and store error logs generated by my Azure Data Factory pipelines. My goal is to insert these logs into a Databricks Delta Table (catalog) for further analysis and troubleshooting. I'm considering two…
Xero connection error using oAuth2
I'm trying to establish the connection to Xero using the Xero connector via oAuth2 with the clientId and clientSecret. In the UI form, the required fields are: ClientID ClientSecret TenantID RefreshToken After filling all fields with the correct…
How to build ETL Method for CDC Data Load for Multiple tables
Hi, I am trying to build an ETL Pipeline using data flows, so that multiple table data loads can be initiated (as a sequential activity), So far I have tried to achive this using an etl table and parameterizing pipeline and dataflows so that I can call…
How to change my tenant on azure data factory
I log in on my azure portal using my personal account. I have been able to create all resources required for my data factory. When launching the azure data factory studio, It prompter me to sign in again with 2fa enable etc, then redirect me to a wrong…
Error Data Flow connection vs CosmosDb
I'm creating a ADF data flow to import data from CosmosDb. I have created a linked service and a dataset to connect with CosmosDb; these connections run successfully: Linked Service connection Dataset connection When i try to do a data preview…