Azure Data Factory

Best resource class for an ADF managed identity database user in a Synapse SQL pool

Hi, for a my Azure data platform solution I'm using Data Factory to implement the pipelines and Synapse Analytics to store the data read from the sources. I've a dedicated SQL pool DW100c. Inside the SQL pool I've specified the ADF system managed…

Read XML text from a xml file and pass it to web activity

Hi, I have a xml file in storage account in a container. I want to read this xml file and get full xml text present in the file and pass it to web activity(Calling an API with body text as this XML). I am using lookup activity to read the file, but it…

Looking for guidance on integrating third-party data into our domain

Hi, My company's transactional system is MS Dynamics 365. We need to import various third-party data into our domain. I'm a Power BI developer but have been assigned this project. I'm looking for guidance on a suitable database and integration tool. …

How to ingest CDC data from SQL DATA SYNC to Event hub?

Hi we have a scenario to implement real-time CDC from SQL DATA SYNC to Azure SQL DB. Which tools are good for near real-time processes and prices as well. By using data bricks we can merge stream data into delta lake. But How to ingest real-time…

ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'All Weekly 18.02.2024.zip' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type

While Files copying from SFTP to ADLS using ADF Copy activity Pipeline fails when doing unzip from SFTP to ADLS.

ADF Power Queries Suddenly not completing - Region Australia East

Hi Team We suddenly have ADF power Query steps not completing and in a hung state. This started happening On Sunday the 21st of April 2024. I have deployed all these pipelines in the Australia East region. It is across all Pipelines where a Power Query…

Azure Data Factory SQL join output of stored procedure with another SQL table

I need to be able to SQL join the output of one Stored procedure with another SQL table. Currently, I am running one lookup activity (name - CustomSP) to run the Stored procedure and get the output. Output of Stored Procedure is as follows - { …

How to fix "LeaseIdMissing" or "LeaseNotPresent" Error in Synapse pipeline

Hello, In past couple of weeks we are facing issues with data lake loads. As a part of the process we delete historical data from the data lake and reload the data into the data lake through synapse pipelines. When trying to delete the historical data…

Express route

Hi, We have newly setup express route with on-prem. I have some queries. How do we test that we have express route connectivity between Azure to On-prem destination servers ? We want to connect on-prem oracle db over express route and TLS using ADF…

Deployment of a simple bicep takes too long

I have a bunch of biceps that deploy various parts of Azure infrastructure. All of them complete within reasonable timeframes except this guy. It runs forever, though, in fact, approves the private managed connection as supposed. Any thoughts? param…

ADF dataflow is creating a 0 Byte file only when Source is configured to move file after completion

ADF provides an option to move the source file after dataflow completion by setting up options in 'Source Settings'. We're trying to use this option to move our source file from SFTP server to an archive location (another folder on same SFTP…

SAP latency data

Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…

ADF dataflows are creating temporary folders / files and fails while trying to rename the file at SFTP

Hi, We're facing the issue while using SFTP sink via a Dataflow. We wish to generate output files using Dataflow at the SFTP location but dataflow is creating temp folders / files and in the end, it tries to rename the file but fails due to reason: Spark…

I've completed the AZ-204 Challenge, and I haven't received the discount coupon.

Hello, what's up? I've completed the training, and I haven't received my certification discount. I recently updated my email for warranty notification, but I haven't found any way to resend the emails to this new sender.

Copying complex JSON data dynamically using Mapping Data Flows in Azure

In my current data engineering project, I am tasked with building a pipeline that can efficiently handle various JSON structures. One of the key requirements is the ability to dynamically flatten nested arrays within the JSON data using parameters, and…

How to fix web activity error "Error calling the endpoint" for Azure API management service from Azure data factory

I have an azure data factory in VNet and API management service with virtual network type "Internal". I am trying to invoke one of the Api endpoints from data factory using web activity "POST" method but getting the below mentioned…

How to convert XML files to Parquet without losing any data. In my scenario the xml file is in complex nested format and I am unable to capture all fields?

Retrieving a single Value from a data flow for pipeline use

I have a problem where I need to use a value calculated inside a Data Flow in my pipeline, but I can't use the usual method of setting a variable because of how the Data Flow is set up. Usually, I'd use a 'Set Variable' activity, but that won't work here…

Snowflake date function to use in ADF

Hi All, We are trying to use below snowflake queries but facing issues. Source table field datatype is varchar and target table date datatype. DD/MM/YYYY (Source format) to MM/DD/YYYY (Target format we want) Can anyone tell me which function I should…

Copy LOB based data from DB2 to Azure Data Lake Gen2 storage using ADF

Hi team, I was trying to work on a scenario to copy data from a table in db2 database having a column with datatype as LOB(CLOB) to Azure Data Lake Gen2 storage container using Azure Data Factory. However, its taking a considerable amount of time( a day)…