Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,892 questions
0 answers
Get jsonlines response from Rest API and load into SQL DB using ADF pipeline
Hi Guys, Requirement is to call Rest APIs which will response data in jsonlines format (not Json array) and load data into table. Tried with Web activity and Set Variable to capture the response but not getting the response in expected format. Can you…
asked 2024-04-15T10:19:39.51+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 45%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
SarmisthaS
0
Reputation points
commented 2024-06-14T08:45:01.2766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
7,345
Reputation points Microsoft Vendor
0 answers
Synapse Dedicated pool index stats
I try to get the fragmentation of my indexes with code SELECT IPS.Index_type_desc, IPS.avg_fragmentation_in_percent, IPS.avg_fragment_size_in_pages, IPS.avg_page_space_used_in_percent, IPS.record_count, …
asked 2024-06-14T08:39:27.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 7.000000000000001%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJH%3C/text%3E%3C/svg%3E)
Jani Hämäläinen
20
Reputation points
asked 2024-06-14T08:39:27.7+00:00
Jani Hämäläinen
20
Reputation points
1 answer
What are the best ways to utilize Microsoft 365 to become a Microsoft Innovative Educator Expert?
how can I use Microsoft in best way to become MIEE
Microsoft 365
Microsoft 365
Formerly Office 365, is a line of subscription services offered by Microsoft which adds to and includes the Microsoft Office product line.
4,086 questions
Microsoft Teams
Microsoft Teams
A Microsoft customizable chat-based workspace.
9,371 questions
Microsoft Authenticator
Microsoft Authenticator
A Microsoft app for iOS and Android devices that enables authentication with two-factor verification, phone sign-in, and code generation.
5,823 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,892 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
10,011 questions
asked 2024-06-14T08:10:14.2533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 26%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVA%3C/text%3E%3C/svg%3E)
Verdha Abid
0
Reputation points
answered 2024-06-14T08:37:20.6033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 11%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJZ%3C/text%3E%3C/svg%3E)
Jing Zhou
3,435
Reputation points Microsoft Vendor
0 answers
Non-breaking space in source file name causing Copy Data activity to fail in ADF
Use case: Copy files from the SFTP source location to Azure blob storage using ADF (Copy Data Activity) Implementation: I have a DataSet pointing to the SFTP location acting as a source to Get MetaData Activity. The Get MetaData Activity returns the list…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 27%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
ADF pipeline/dataflow creates temporary GUID folders and files and not delete them
Hi, I have a problem that my processes, from time to time, create temporary GUID folders and/or files in them but, in the end, not deleting them. Where is the problem/reason of that?
asked 2024-06-10T11:16:40.6933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 13%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGE%3C/text%3E%3C/svg%3E)
Gużewski, Jacek
95
Reputation points
edited the question 2024-06-14T07:59:58.3766667+00:00
PRADEEPCHEEKATLA-MSFT
82,191
Reputation points Microsoft Employee
1 answer
ADF linked service for Snowflake is timing out
Hi, In my production account we would be doing the data processing on ADF and we would be using Snowflake as a data warehouse. Also, for some ADF pipelines same Snowflake DWH will also be a source. I first create a role, granted it permissions created a…
asked 2024-06-13T09:36:27.5366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 90%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Shashwat Tiwary
40
Reputation points
commented 2024-06-14T07:23:32.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
0 answers
Error de Conexión REST: TrustFailure y Certificado Inválido en Azure Data Factory
Description: Hello everyone, I'm facing an issue while trying to connect a REST service in Azure Data Factory. The configuration is as follows: Base URL: https://yam.3psi.com.pe/wsextdatsee/api/operacioncliente?datpet=20383929050_1_Lgo2023$1 …
asked 2024-06-12T16:34:44.1533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 97%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EY%3C/text%3E%3C/svg%3E)
Yamaha_motors
0
Reputation points
commented 2024-06-14T07:11:12.21+00:00
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
1 answer
ADF deployment -linked services and IRs
Hi, I want to use CI/CD pipeline to deploy by ADF dev instance code to other environments(UAT, Prod). Can I deploy the code without integration runtime,linked services? I want to deploy only the rest of the code. As these IRs and integration runtimes…
asked 2024-06-07T05:54:43.7066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Santhi Dhanuskodi
205
Reputation points
commented 2024-06-14T06:33:45.1566667+00:00
PRADEEPCHEEKATLA-MSFT
82,191
Reputation points Microsoft Employee
1 answer
How to write a SQL query in ADF lookup activity for cosmos gremlin DB(json data)?
I have the gremlin cosmos db and i loaded the data as vertices and edges. when i am trying to read the data using adf look up activity(select * from grem_db_test), I am getting data in the json format which is fine. But why it's not supporting when i run…
asked 2024-06-13T01:31:15.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 73%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Siddartha Reddy Jammula
20
Reputation points
edited a comment 2024-06-14T05:52:23.5766667+00:00
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
1 answer
Dynamic Content Option for Password Field in SFTP Connector in Azure Synapse Analytics
I am working on a feature in Azure Synapse Analytics that connects to different SFTP servers dynamically based on a meta-data table. As I proceeded through it, I found out that in SFTP Linked Service, the PASSWORD field does not allow dynamic content.…
asked 2024-06-12T17:19:45.9533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 5%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Khushank Madaan
0
Reputation points
commented 2024-06-14T05:48:26.0466667+00:00
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
1 answer
Concatenate 2 files with different schema in Data Factory
I have a CSV file that needs to be that need to be concatenated with the results of a pipeline. Essentially the first CSV file acts like the metadata for the pipeline file, so they do not have the same schema. I can't seem to merge them together using…
asked 2024-06-13T21:58:57.7033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 94%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Michael Lawless
0
Reputation points
commented 2024-06-14T05:43:44.9366667+00:00
Vinodh247-1375
12,146
Reputation points
1 answer
ADF Linked Service connection to Databrciks SQL
Hi Team, I need to read table from the Azure Databricks SQL end point, can you please let me know how to setup the linked service. i know this option was not there around one year before, just checking if anything new came up. please do not suggest other…
asked 2024-06-13T15:53:58.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 8%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Sharukh Kundagol
105
Reputation points
answered 2024-06-14T05:32:31.9533333+00:00
PRADEEPCHEEKATLA-MSFT
82,191
Reputation points Microsoft Employee
0 answers
ErrorCode=ODataFailedClientCreation
we would like transfer data about odata to sql. Our linked service is working. Only the ADF pipeline doesn't work with folling error... …
asked 2024-06-12T05:44:04.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 15%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETA%3C/text%3E%3C/svg%3E)
T1 ADM Sven ***
0
Reputation points
commented 2024-06-14T05:32:13.34+00:00
phemanth
7,345
Reputation points Microsoft Vendor
0 answers
Static Code Analysis for ADF or Synapse Pipelines in Azure
Hi team, I would like to know is there any use case possible to perform static code analysis of our ADF or Synapse pipelines residing in Azure DevOps repository? Thanks, Debajit Mitra
asked 2024-06-14T03:42:29.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 48%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDM%3C/text%3E%3C/svg%3E)
Debajit Mitra
0
Reputation points
commented 2024-06-14T05:25:29+00:00
Nandan Hegde
29,996
Reputation points MVP
1 answer
File movement from sharepoint Nested Subfolders using Microsoft Graph API to blob using ADF
I am trying to copy files from sharepoint to Azure Blob Storage using Microsoft Graph API in ADF. I am able to do it when the sharepoint folder structure is flat. I am facing issue when the sharepoint has nested subfolders. Is there a way to get…
asked 2024-06-05T02:34:02.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 95%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jagdev Sinder
0
Reputation points
commented 2024-06-14T05:14:24.47+00:00
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
1 answer
How to pass a parameter from one dataflow to another dataflow in azure data factory?
I'm stuck in one of the azure data factory related problem int which i want to pass the parameter of one dataflow's sink to another dataflow's source. Can anyone help me out with this problem?
asked 2024-06-13T07:07:50.5266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 34%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERP%3C/text%3E%3C/svg%3E)
Rachit Pandya
0
Reputation points
commented 2024-06-14T04:57:44.83+00:00
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
1 answer
ADF copy-activity from Microsoft 365 to storage account fails
Hello, I’ve a continuous problem with my ADF pipeline - I’m trying to run a "copy-data" pipeline and encounter access and permission errors with my specific resources. My source is a Microsoft 365 Table connector (I’m retrieving some columns…
asked 2024-06-09T22:26:35.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 6%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYS%3C/text%3E%3C/svg%3E)
Yonatan Shlain
0
Reputation points
edited the question 2024-06-14T04:51:08.2133333+00:00
KapilAnanth-MSFT
38,056
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Creating ADF linked service to connect to a storage account on private endpoint using service principal
HI, Currently my storage account is open and allows public access. I disabled public network access to storage and followed this article for creating a secured SFTP connection through a firewall -…
asked 2024-06-10T10:28:27.29+00:00
Shashwat Tiwary
40
Reputation points
edited the question 2024-06-14T04:50:14.6866667+00:00
KapilAnanth-MSFT
38,056
Reputation points Microsoft Employee
1 answer
Is it possible to fetch in-progress pipeline details via an API call in Azure Data Factory, and if so, how can it be done?
I attempted to use the following API: POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01 However, I was…
asked 2024-06-13T06:02:22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 37%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Parthkumar Pavra
0
Reputation points
answered 2024-06-14T04:38:08.6533333+00:00
Smaran Thoomu
11,370
Reputation points Microsoft Vendor
2 answers
Copy LOB based data from DB2 to Azure Data Lake Gen2 storage using ADF
Hi team, I was trying to work on a scenario to copy data from a table in db2 database having a column with datatype as LOB(CLOB) to Azure Data Lake Gen2 storage container using Azure Data Factory. However, its taking a considerable amount of time( a day)…
asked 2024-04-21T03:40:26.3+00:00
Debajit Mitra
0
Reputation points
commented 2024-06-14T03:39:35.9133333+00:00
Debajit Mitra
0
Reputation points