Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
0 answers
ADF Copy Activity - Source Schema not fixed
Hi Experts, I am still in learning phase of ADF. I have a scenario wherein I use Copy activity with source dataset connecting to SFTP server and fetch csv files from there and storing it in Gen2 containers. Inside the Copy activity, schema mapping has…
Azure Data Factory
asked 2021-10-08T18:16:49.257+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 12%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kesavan R
1
Reputation point
commented 2021-10-19T15:43:33.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 63%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
svijay-MSFT
5,256
Reputation points Microsoft Employee
Moderator
1 answer
Issue with Lookup activity in Data Factory
I am facing issue while updating the Error message of a previous activity into a sql table using Look up activity . Below is the error message Error:Failure happened on 'Source' side.…
Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,671 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-07T17:47:39.733+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 42%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aman Sharma
1
Reputation point
commented 2021-10-19T15:41:13.477+00:00
svijay-MSFT
5,256
Reputation points Microsoft Employee
Moderator
2 answers
partition delta files based on a particular column, as we write them in to a sink.
I am extracting my changed data every night as parquet and then merging (upsert) them into my delta lake (delta file). In the optimization tab I am setting dynamic range and passing the column name as a parameter. When I check the input json, I see…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-07T12:20:04.387+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 45%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
shadman44717
1
Reputation point
commented 2021-10-19T15:38:37.997+00:00
svijay-MSFT
5,256
Reputation points Microsoft Employee
Moderator
1 answer
How to implement logic if one of any five different boolean variable are true...
I have a daily ELT process that picks up some file from blob storage. This process has five required files that need to be present for it to proceed. This is the part of the logic I am trying to implement. I already have the logic that checks to see…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-09-30T20:11:22.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 56.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ5%3C/text%3E%3C/svg%3E)
JasonW-5564
161
Reputation points
commented 2021-10-19T15:36:56.61+00:00
svijay-MSFT
5,256
Reputation points Microsoft Employee
Moderator
0 answers
Copy Activity erroring 'received an invalid column length'
copy activity failed when encounter one record with unexpected length, is there a way to log the record instead of failing the activity? configured as 'Skip incompatible rows' already and it only for mismatch with data type. if can log data truncation…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-06T19:06:36.213+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 90%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JW_6688
1
Reputation point
commented 2021-10-19T14:53:19.167+00:00
svijay-MSFT
5,256
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
update issue
Can you please see what I am doing wrong to cause insert as well as update? I am simply trying to update the Delta sink parquet file by setting IsDeleted field = 1 when the source row is deleted. The pipeline calls the dataflow and the source…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
2 answers
How to merge files by picking them dynamically
There are a lot of files with timestamp suffixed in their filename. I keep getting few of those files each day, and some of those few files have multiple copies of them with different timestamp (hh or mm or ss component of timestamp differ on each day). …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-17T10:50:11.217+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 30%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERV%3C/text%3E%3C/svg%3E)
Ranjan, Vineet
1
Reputation point
answered 2021-10-19T03:57:01.387+00:00
Ranjan, Vineet
1
Reputation point
2 answers
One of the answers was accepted by the question author.
ADF SHIR; MySQL; and connectivity issues
tl;dr; The self-hosted integration runtime, SHIR, reports connection timeout errors to MySQL instances whereas a query from HeidiSQL/MySQL Workbench to the same instance works fine. 2021-10-18 update It appears to be a timeout issue. The SHIR drops the…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-05T21:15:14.123+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 63%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBF%3C/text%3E%3C/svg%3E)
Bill Fellows
96
Reputation points
accepted 2021-10-18T19:22:26.417+00:00
Bill Fellows
96
Reputation points
1 answer
One of the answers was accepted by the question author.
Copy Data Import Schemas not working for API call in ADF
In the last 2 weeks, the "Import Schemas" button in the Mapping tab of the Copy Data activity has quit working when the source dataset is an API. This obviously affects new development, but to test it, I attempted to import the schema of an…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-12T21:35:07.247+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 0%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELO%3C/text%3E%3C/svg%3E)
L-O-L-A-Lola
26
Reputation points
commented 2021-10-18T19:15:37.893+00:00
L-O-L-A-Lola
26
Reputation points
1 answer
Dbase DBF File to Azure SQL
Problem Statement : Dbase .dbf files to be exported to Azure SQL using Azure Data Factory. Looking for a solution : How we can export .dbf files to Azure SQL using ADF. Is there any connectors and transformations for storing the data from Dbase files…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-01T04:25:20.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 84%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Praveen kumar raju Chamarthi
6
Reputation points
commented 2021-10-18T19:13:52.51+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,851
Reputation points Microsoft Employee
Moderator
0 answers
can't parse json files stored as octet-stream
i am storing data in azure by using the data reader getstream method from sql server. This allows me to upload the stream to azure blob via c# without incurring memory issues. I can read the file which is a gz json file formatted as a list of json…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-09-20T13:52:59.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 21%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
wayne
1
Reputation point
commented 2021-10-18T18:55:33.193+00:00
Saurabh Sharma
23,851
Reputation points Microsoft Employee
Moderator
0 answers
Copy Activty Issue
Hello Team, I am using copy activity Source :Blob storage Target : Snowflake Table(And using procedure to run the copy command to transfer the data).The data is getting copied to the target table from blob sotrage file .But the…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-11T16:43:21.943+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rohit Kulkarni
731
Reputation points
commented 2021-10-18T15:28:07.02+00:00
svijay-MSFT
5,256
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
Write semi structured csv from Azure Data Factory
I've been tasked with writing a CSV file with a certain format - NEM12 (see link below) MDFF-Specification-NEM12--NEM13-v106.pdf I'm stumped as to how I could do this in ADF. The general format has a different number of columns per row, depending…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-07-12T07:46:33.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 19%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Peter Sharp
181
Reputation points
accepted 2021-10-18T08:34:30.983+00:00
Peter Sharp
181
Reputation points
1 answer
One of the answers was accepted by the question author.
Parametrize Snowflake LinkedService
I am trying to parametrise snowflake linkedservice credentials. I am trying to read from keyvault. Based on dbname,i need fetch different secrets from keyvault .I am getting error. I am not sure how to do it.
Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,455 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-14T08:32:32.163+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 48%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Aditya Raj
286
Reputation points
accepted 2021-10-18T06:42:48.74+00:00
Aditya Raj
286
Reputation points
0 answers
ADF Complex json file load
Hi Team, I am having a really complex json file as shown in below sample. Kindly provide me input to deal with this kind of file to load into SQL table by using ADF. [ { "TBL": "Aisha_abc", "Row": [ …
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,385 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-09-30T12:29:46.587+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 85%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Aayush Ku Soni
1
Reputation point
commented 2021-10-15T20:03:29.273+00:00
Aayush Ku Soni
1
Reputation point
2 answers
Data Factory using Key vault for linked Services Information
Im using the following information https://learn.microsoft.com/en-us/azure/data-factory/store-credentials-in-key-vault I go through. get the Managed Identity Object ID from data Factory In Key vault I do an Access Policy with GET. and I used…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-02-01T12:31:35.397+00:00
Debbie Edwards
526
Reputation points
answered 2021-10-15T19:50:38.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 59%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERR%3C/text%3E%3C/svg%3E)
Raphael Ramos Da Silva
1
Reputation point
1 answer
One of the answers was accepted by the question author.
SFTP/FTP source location and Azure Fuctions
Hi All, My source sits on FTP/SFTP which are quite a few Excel data sets in a Zip folder, what I want to do is use Azure Functions and convert them in to CSV format and store in Blob Storage and ultimately copy in to sink ADLS. My question is can I…
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,943 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,203 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-09-11T09:26:59.577+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 12%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Samy Abdul
3,376
Reputation points
commented 2021-10-15T10:19:33.883+00:00
Samy Abdul
3,376
Reputation points
1 answer
Mapping data flow expression for converting simpledate time format to epoch timestamp
There are existing functions where in we can convert datetime from epoch to other formats. What could be the possible function combination to convert a particular datetime as below to epoch timestamp. Ex: lastconnected:…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-11T12:13:48.183+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 78%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECP%3C/text%3E%3C/svg%3E)
Chetan Prakash Saraswat
31
Reputation points
commented 2021-10-15T05:49:39.303+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
4 answers
Restrict ADF converting Null data from source date field returning 1900-01-01 in target column
Hello All, I am not sure, whether this was already asked, I did a search and didn't find any suitable replies related to this. ADF copy data is returning '1900-01-01' by default to a NULL DateTime field when moving the data from Source to target table…
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,385 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-03-09T23:14:10.59+00:00
KarthikShanth
26
Reputation points
answered 2021-10-15T01:53:25.08+00:00
Darpan Patel
6
Reputation points
1 answer
One of the answers was accepted by the question author.
How to Automate path location of notebook file of ADB using azure data factory.
Hi, I am trying to automate the file location of databricks csv file using ADF notebook pipeline activity, so we can avoid the hardcode the path while running the code I have followed this MS doc : …
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,551 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,665 questions
asked 2021-10-13T14:57:27.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 38%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kumari, Shweta
41
Reputation points
accepted 2021-10-14T05:21:48.947+00:00
Kumari, Shweta
41
Reputation points