Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,810 questions
0 answers
ADF unable to process large JSON file
How can I fix the issue of ADF data flow failing to transform a 700MB JSON file with the error message "Operation on target Execute LIA failed: Operation on target LIA failed:…
asked 2024-06-03T14:41:24.3766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 51%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Sathisha Bannihall
0
Reputation points
asked 2024-06-03T14:41:24.3766667+00:00
Sathisha Bannihall
0
Reputation points
0 answers
Azure Data Factory simple copy file and rename
Hello: I have a simple requirement. I have a couple of files in an Azure Blob folder (receiving folder) and I need to copy them, to another fold (Azure Blob folder) and rename them. This posting is really close to what I need to do: …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
0 answers
How to add a GUEST user to my AZURE Portal with my Microsoft Azure Sponsorship Member account
Hello, I have a Microsoft Azure Sponsorship membership that expires in October 2024, I have just begun working on my concept with developers and an AI team. I need help getting GUEST users inside of my AZURE portal instead of inviting them to be a full…
asked 2024-06-03T14:34:02.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 31%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
BOBBY SHEPPARD
0
Reputation points
asked 2024-06-03T14:34:02.28+00:00
BOBBY SHEPPARD
0
Reputation points
0 answers
azure data factory's databricks runtime
Hello, i am trying to ingest data from SAP to fabric lakehouse table using dataflow and copy activity, however because my sink table has change data feed enabled, the pipeline throws the following error: Operation on target Data flow1 failed:…
asked 2024-06-03T14:24:46.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 56.00000000000001%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOT%3C/text%3E%3C/svg%3E)
Osama Tarek
0
Reputation points
asked 2024-06-03T14:24:46.9933333+00:00
Osama Tarek
0
Reputation points
0 answers
ADF Data Flows Flatten nested json array values are being populated as null
Hi All I am building a data transamination with ADF data flow using a nested json array of objects , but after parse and flatten the json node itOffer.item.LeadOfer.zdeal.item[].dealNumber I am seeing that the column values are populated as null . I…
asked 2024-05-30T11:06:56.13+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
45
Reputation points
commented 2024-06-03T14:14:40.8266667+00:00
venkat rao
45
Reputation points
1 answer
insert record from azure datafactory into a delta table generated in databricks
Hi team, i'm trying to insert value into a delta table from datafactory, the table has been created in databricks and in the table properties i setted "delta.enableChangeDataFeed=true", for this reason the protocol minWriterVersion of my delta…
asked 2022-03-02T12:08:37.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 61%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAN%3C/text%3E%3C/svg%3E)
Antonio Nuzzo
11
Reputation points
commented 2024-06-03T14:13:46.6433333+00:00
Osama Tarek
0
Reputation points
1 answer
Issue with Copy Dataverse data into Azure SQL using Synapse Link Template
I am experiencing issues using ADF I am trying to use the template "Copy Dataverse data into Azure SQL using Synapse Link" to move f&o data from datalake to Azure…
asked 2024-05-30T07:08:10.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 20%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDO%3C/text%3E%3C/svg%3E)
Dipo Onibile
0
Reputation points
answered 2024-06-03T14:13:41.7366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 25%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
rn
0
Reputation points
0 answers
Parameterized ADF DataFlow - Using parameter in Parse JSON
Hi, I am using Azure Data Factory to process data stored in ADLS. The data is stored in the following format -> raw/measurements/{measure_id}/year/month/{site_id}.json {measure_id} is an unique ID for the type of measurement from the upstream…
asked 2024-06-03T13:58:56.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 65%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
RosinaNorton
0
Reputation points
asked 2024-06-03T13:58:56.1866667+00:00
RosinaNorton
0
Reputation points
3 answers
Connecting to a Azure MySQL instance from Data Factory Linked Service
I have an Azure Mysql Instance in a Virtual Network created as part of the instance setup. I want to be able to copy data to MYSQL as a Copy Data sink. When I try to set up a connection to the MySQL instance using the AutoResolveIntegrationRuntime it…
asked 2023-10-10T15:33:47.4133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 12%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENE%3C/text%3E%3C/svg%3E)
Navid Erfani-Ghadimi
0
Reputation points
commented 2024-06-03T13:57:20.13+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 79%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Silva Sergio (CI/BTM-HR3)
0
Reputation points
4 answers
Script Activity not supporting multi-instruction with new Snowflake connector
I have a pipeline, in Azure Data Factory that has an "Activity Script" that executes 3 instructions against Snowflake. The script was working fine until I have updated the Snowflake connection in order to remove the legacy Snowflake connection…
asked 2024-03-21T08:23:45.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 40%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DavidR
20
Reputation points
commented 2024-06-03T13:17:56.9533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 90%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Laan, Niels
0
Reputation points
2 answers
How to convert type of any block blob to append blob inside adls gen2? Is there any process or activity which will convert the type of my entire blob without changing the content?
I have a json file inside a data lake gen2 storage and that json file blob type is a block blob. I want to convert that blob type to append blob through any services of azure. Need to know the process in detail.
asked 2024-05-31T06:30:48.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 36%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECN%3C/text%3E%3C/svg%3E)
Chakraborty, Nabarun (External)
0
Reputation points
answered 2024-06-03T12:09:39.6833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
3,121
Reputation points Microsoft Vendor
1 answer
Mapping to polymorphic field in dynamics 365
Hello Team - I am loading data into the Incident table of a Dynamics 365 environment from an Azure SQL DB using the Azure Data Factory. the Incident table has lookups to the Account table on AccountId and Contacts via the ContactId. I have populated…
asked 2024-06-03T05:37:20.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 10%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
Binway
676
Reputation points
answered 2024-06-03T11:56:26.6866667+00:00
Amira Bedhiafi
17,201
Reputation points
3 answers
Looking for guidance on integrating third-party data into our domain
Hi, My company's transactional system is MS Dynamics 365. We need to import various third-party data into our domain. I'm a Power BI developer but have been assigned this project. I'm looking for guidance on a suitable database and integration tool. …
asked 2024-04-18T16:54:37.4433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 21%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anon101
51
Reputation points
answered 2024-06-03T10:47:20.6633333+00:00
Pinaki Ghatak
2,405
Reputation points Microsoft Employee
1 answer
Running specific pipelines from an Orchestration pipeline
How can I run individual pipelines from an Orchestration pipeline that has multiple pipelines running in parallel? I want to be able to choose which pipeline to run separately from the rest.
asked 2024-05-30T21:38:39.0933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 41%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
ADITYA SINGH
0
Reputation points
edited the question 2024-06-03T10:45:18.8166667+00:00
Nehruji R
3,121
Reputation points Microsoft Vendor
1 answer
Updating Oracle on-premise table from a Synapse table
Hi, I need to update an Oracle on-premise table from a Synapse table by joining the Oracle table with the Synapse table avoiding to build a staging table on Oracle platform to copy the Synapse data. It is a problem to obtain the authorization from my…
asked 2024-06-01T12:00:58.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 41%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
pmscorca
837
Reputation points
commented 2024-06-03T09:26:51.3466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
11,045
Reputation points Microsoft Vendor
1 answer
How can i monitor azure data factory pipelines runtime logs in realtime with no latency?
Hi team, I'm trying to to retrieve Azure Data Factory pipeline execution logs in Realtime without any latency. If tried using log analytics, I see there is a latency in logs to get available. How can I get the Realtime pipeline details including IN…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 3%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
3 answers
How to update and delete a row using Azure Data Factory Change Data Capture
I am exploring Azure Data Factory CDC feature, and I am trying to perform the CDC from one SQL table to another SQL table, also the SQL table in the source has a primary key. Whenever a new row is added to the source table, it gets added to the…
asked 2024-04-28T17:40:36.5966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
jigsm
236
Reputation points
answered 2024-06-03T09:01:35.7833333+00:00
Pinaki Ghatak
2,405
Reputation points Microsoft Employee
3 answers
Lookup activity output into a CSV file in ADLS
I'm using lookup activity to read all tables from a Oracle SQL Developer application for a particular schema. The query is SELECT TABLE_NAME FROM all_tables WHERE owner = 'Schema_Name' Here the scheme name is dynamic taking value from a variable. Now, my…
asked 2024-04-26T16:14:09.8366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 60%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Bhajanthri Bala Sujay
0
Reputation points
answered 2024-06-03T08:56:32.6633333+00:00
Pinaki Ghatak
2,405
Reputation points Microsoft Employee
3 answers
Trying to write an expression for a filter activity in Azure Data Factory
I currently am using 3 filter activities where I am using the following condition @contains(item().name,'example') to filter out a certain group of files from the array obtained by get metadata This works fine but is still bringing back 5 files with the…
asked 2024-04-25T15:31:35.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 2%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA0%3C/text%3E%3C/svg%3E)
azure-000
1
Reputation point
answered 2024-06-03T08:45:30.17+00:00
Pinaki Ghatak
2,405
Reputation points Microsoft Employee
4 answers
How do I copy data from a public sharepoint URL to Microsoft Blob Storage?
Hi everyone! I am new to using Azure services. My co-author at Stanford has uploaded several files (in csv format) to a public link. This link starts as: office365stanford-my.sharepoint.com/personal It has 40 files, ranging from a few 100 MB to 100 GB. I…
asked 2024-04-08T16:28:15.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 12%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENN%3C/text%3E%3C/svg%3E)
Namrata Narain
0
Reputation points
answered 2024-06-03T08:38:37.6966667+00:00
Pinaki Ghatak
2,405
Reputation points Microsoft Employee