Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,342 questions
1 answer
One of the answers was accepted by the question author.
Filter Query error
Hi, I have requirement to filter date folders from the container like shown below: 2022-01-15 2022-02-04 sample_data test_data out of these folder i want to filter only date folders (2022-01-15,2022-02-04) what query should i use in If…
asked 2022-02-14T04:35:48.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 20%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Prapul Kumar Dongari
116
Reputation points
commented 2022-02-21T10:53:47.71+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee
2 answers
Copy csv files from multiple sub folders and sink to adls
My source is adls, having different folders and folders contains csv files. I need to look in to each folder and copy data dynamically. storage account having multiple folders. i used get metadata to get folders and foreach to iterate each folder .…
asked 2022-02-10T12:20:44.063+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 42%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
shanucnu8888
1
Reputation point
answered 2022-02-17T01:18:07.563+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Databricks mount (with SAS, using WASB driver) storage account gives update error on delta tables: Blob operation is not supported
Hi, We have an azure storage account mounted on our databricks workspace. Using databricks, we upsert delta tables located in the azure storage account. The azure storage account is mounted using a SAS token (WASB driver). We get sometimes (10-20% of…
asked 2022-02-11T08:38:28.037+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 49%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFV%3C/text%3E%3C/svg%3E)
Freia Vercruysse
31
Reputation points
commented 2022-02-16T20:04:23.177+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
2 answers
Data Factory: Error Code 24200: Unable to establish a connection from link service to Data lake v2
Hello everyone, Apologies if this has already been asked, but I cannot find any similar answers on this forum. I am following an online course for Azure Data Factory and I'm setting up a pipeline. Right now this pipeline only has one activity (a copy…
asked 2021-11-29T14:00:56.793+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 81%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJH%3C/text%3E%3C/svg%3E)
Justin Hughes
1
Reputation point
answered 2022-02-16T19:50:41.227+00:00
Tamur Aziz
1
Reputation point
0 answers
How to execute Hive query in Databricks?
We are calling a ".jar" file from Azure Data Factory using Databricks JAR activity. In the JAR activity we are specifying the Cluster Id in Databricks Linked Service. In Databricks cluster we are adding below Spark config: …
asked 2022-02-11T12:21:07.377+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 34%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Suman Dutta
1
Reputation point
commented 2022-02-16T19:47:50.707+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
0 answers
SFTP Storage Algorithms issue
When connecting, we are getting an error that the server does not support FIPS compliant algorithms. Will this change/improve in the future? Is there a workaround?
asked 2022-02-07T06:35:26.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 35%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Mike P
21
Reputation points
commented 2022-02-16T17:00:11.247+00:00
Sumarigo-MSFT
43,641
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Job failed due to reason: at Source... ...Job aborted due to stage failure
I have a pipeline in Synapse which calls a data flow. That data flow has started to fail and return the below error. {"message":"Job failed due to reason: at Source 'RawTransaction': org.apache.spark.SparkException: Job aborted due to…
asked 2021-04-15T20:01:24.453+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 87%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Steven Howe
106
Reputation points
answered 2022-02-16T15:36:36.977+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 84%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
praveen sharma
1
Reputation point
2 answers
dataflow taking toomuch time for executions need help on priority.
Hi In my production pipeline i am using dataflow activity to falttern json file into csv . it taking too much time that is 45 minutes and still either fail or success. same pipeline in dev environment same input data taking 4 to 5 min to complete.…
asked 2022-02-09T09:09:17.397+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 35%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKV%3C/text%3E%3C/svg%3E)
Karnati,Venkata Suchendra Reddy,IN-Bangalore
6
Reputation points
commented 2022-02-16T06:47:50.037+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee
0 answers
Problem while uploading specific files from sftp (ADF) : ErrorCode=UserErrorFailedToReadStream
Hello, I use Azure DataFactory for upload files from an SFTP to an Azure BlobStorage V2. Most of the time, everything works fine. But on specific files, i get the following error : ErrorCode=UserErrorFailedToReadStream I tried to…
asked 2022-02-11T11:17:13.8+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 6%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pichon, Simon
6
Reputation points
commented 2022-02-15T17:42:44.943+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,026
Reputation points
1 answer
One of the answers was accepted by the question author.
Error: 'Key columns' expression should return string[], when passing array of parameters in DataFlow with "Delta" as Sink
Hello, I have set of parquet files that I need to Upsert (Update else Insert) into "Delta" (sink) using Data Flow with mainly following 3 transformations: SourceParquetFile==>AlterRowUpsert:true()==>SinkDelta (AllowUpsert) I…
asked 2022-02-04T15:27:43.013+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 72%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
ManiK
36
Reputation points
commented 2022-02-15T17:17:07.337+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
ADF Release Pipeline- Deployment failed CD
Hi I am using the DeployAzureResourceGroup task and have a parameter in my deployment template of type securestring for a ls_connectionString. I have set this as a variable with the 'secret' box checked, and other way I'm done with Azure Keyvault…
asked 2022-02-10T09:15:13.547+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 17%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
surender
46
Reputation points
commented 2022-02-15T11:09:19.977+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee
1 answer
Error in copying data to ADLS Gen2
I am trying to copy data from on-prem dataset to ADLS and Getting this error here: Operation on target CopyActivity_XXX failed: 'Type=System.TimeoutException,Message=This request operation sent to…
asked 2022-01-25T17:46:07.217+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 14.000000000000002%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
AzureLearner
1
Reputation point
commented 2022-02-15T10:48:10.463+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee
0 answers
Source SQL Table Have 1 column with tab spaces, unable to copy data to ADLS it shows below.
Source is SQL: One of the table contains column as below: Basic document flow "Hi Team We are separated by our margins. Because of margin collapsing, we are separated by the width of one of our margins, not both. Charles…
asked 2022-02-04T11:59:01.733+00:00
shanucnu8888
1
Reputation point
commented 2022-02-14T22:57:12.88+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
2 answers
The requested name is valid, but no data of the requested type was found error when renaming datalake file in C#
HI I am facing below issue when renaming a file in azure data lake using C# .net core 3.1 can any one show me a solution error The requested name is valid, but no data of the requested type was found. here is the sample code …
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,342 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,247 questions
asked 2022-01-23T13:33:49.367+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 86%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Santhosh Kumar S
1
Reputation point
answered 2022-02-12T03:07:20.697+00:00
Santhosh Kumar S
1
Reputation point
3 answers
Need workaround to make event trigger on Datalake ADLS Gen1
I need to find out a workaround solution to trigger my ADF V2 pipeline when files are placed in data lake ADLS Gen1. So anyone tell me an alternative approach to make a workable solution. Any reference solutions or links are appreciated.
asked 2020-07-23T10:35:07.917+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 78%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Simhadri Saripalli
1
Reputation point
answered 2022-02-11T03:11:00.477+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 11%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Chand, Anupam SBOBNG-ITA/RX
451
Reputation points
2 answers
One of the answers was accepted by the question author.
Multiple Storage Accounts for a Data lake
All, We are thinking of implementing the Zones using Storage Accounts. Which means, each of the 3 Zones (RAW, Staged, Curated) will have one Storage Account each. Are there any best practice document? Is it normal to have more than one Storage…
asked 2022-02-08T02:56:18.287+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 24%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGR%3C/text%3E%3C/svg%3E)
Gopinath Rajee
646
Reputation points
accepted 2022-02-10T09:01:19.83+00:00
Gopinath Rajee
646
Reputation points
1 answer
One of the answers was accepted by the question author.
Returning result from azure databricks to azure datafacotry
I am new to using Azure Databricks and ADF. I have a table in azure databricks. I would like to run a select query on this table and return the result of my select query to ADF. When I searched on how this can be done, I see we can use…
asked 2022-02-08T00:48:57.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 43%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKG%3C/text%3E%3C/svg%3E)
Karpagam Gurumurthy
41
Reputation points
accepted 2022-02-09T00:04:15.157+00:00
Karpagam Gurumurthy
41
Reputation points
1 answer
Text Qualifier in ADF BLOB Source
Hi my blob file CSV has comma as row separator , In SSIS I specify text qualifier as " However in ADF source I cant find where can i define text qualifier File is like COLA, COLB 1222, 66666 1333,"Value with comma,…
asked 2022-02-02T06:13:55.953+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 44%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Mav
81
Reputation points
commented 2022-02-08T18:01:45.19+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
3 answers
One of the answers was accepted by the question author.
Latest file extraction (Azure Data factory)
Hi Team, I want to extract latest file from Azure data lake store based on last modified date and want to process in another folder.Could you please someone let me know what needs to be done here if anyone has idea or implemented the same. for…
asked 2020-10-23T10:27:06.567+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 98%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
pankaj chaturvedi
86
Reputation points
commented 2022-02-08T16:06:32.427+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 27%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJK%3C/text%3E%3C/svg%3E)
John Killeen
1
Reputation point
1 answer
SFTP Enabled Azure Storage: How to assign a static public IP?
How can we assign a static public IP to the SFTP enabled storage? We currently connect using a the blob storage name, but we have clients that need to allow/add the public IP to their firewall. Thanks.
asked 2022-02-03T00:41:55.197+00:00
Mike P
21
Reputation points
commented 2022-02-07T17:36:17.297+00:00
Sumarigo-MSFT
43,641
Reputation points Microsoft Employee