Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,403 questions
1 answer
One of the answers was accepted by the question author.
org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename temp file
[Repeat Question due to old thread] We have built a streaming pipeline with spark autoloader. Source Folder is a azure blob container. We've encountered a rare issue (could not replicate it). Below is the exception…
asked 2024-06-07T01:40:56.47+00:00
Hiran Amarathunga
25
Reputation points
accepted 2024-06-14T02:26:30.36+00:00
Hiran Amarathunga
25
Reputation points
1 answer
Run Databricks notebook from ADF - error to find azure module to save the data in blob storage
Hi Guys, The requirement is - Call Rest API, read the records in jsonlines format and load into table in Azure SQL server. I used Databricks to read the jsonlines from Open API using Python script. It can read and keep the data into a file in Azure blob…
asked 2024-06-12T12:39:10.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 35%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sarmistha Sarkar
0
Reputation points
answered 2024-06-13T05:37:40.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
7,670
Reputation points Microsoft Vendor
1 answer
unable to connect adls gen storage from Purview
unable to connect adls gen storage from Purview gettting following error ADLS Gen2 operation failed for: Storage operation '' on container 'bloblblob' get failed with 'Operation returned an invalid status code 'Forbidden''. Possible root causes: (1).…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How to assign a Static Public IP to Azure Storage Account
I have a storage account in Azure and I need to assign a static public IP because we use a dedicated internet link that requires the endpoint to have a public IP address. The dynamic IP addresses from Azure change over time, so I need a fixed IP. Is…
asked 2024-06-05T18:33:50.5833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 80%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Steven Axel Williams
20
Reputation points
accepted 2024-06-10T12:53:53.5933333+00:00
Steven Axel Williams
20
Reputation points
1 answer
One of the answers was accepted by the question author.
My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a
Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…
asked 2024-05-14T19:23:38.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 90%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Shashwat Tiwary
40
Reputation points
accepted 2024-06-10T09:53:29.01+00:00
Shashwat Tiwary
40
Reputation points
1 answer
Can i copy data from ADL to Neo4j graph database using ADF
Hello, I want to load data from ADL to neo4j database using ADF. Can I do that using ADF?? If yes, what are the different options I have? My first priority is I want to use ADF activities if that's not possible i can use python coding.
asked 2024-06-06T14:27:54.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 73%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Siddartha Reddy Jammula
20
Reputation points
commented 2024-06-10T07:33:17.0466667+00:00
phemanth
7,670
Reputation points Microsoft Vendor
0 answers
Calling Power BI reports from ADF
Hello, I have one report with multiple client folders this is the ADLS gen 2 folder location for one client and data source connection: AzureStorageDataLake for different files: …
asked 2024-06-06T11:57:54.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 1%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBA%3C/text%3E%3C/svg%3E)
Bodke, Anil
0
Reputation points
commented 2024-06-10T03:40:58.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
5,270
Reputation points Microsoft Vendor
3 answers
How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF
Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…
asked 2024-01-04T06:25:53.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 17%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETY%3C/text%3E%3C/svg%3E)
Tom Young
0
Reputation points
edited an answer 2024-06-07T13:17:10.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 50%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEL%3C/text%3E%3C/svg%3E)
Edward Loughran
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to Create Delta Table in Azure Synapse Analytics with Id Auto Increment Identity Column ?
I have created the Delta Lake Delta tables In ADLS using Synapse Notebook and in that table, I want to add an identity column (Auto increment 1,1) but I am not able to create the same, Below is my Create table script and error which i am facing. Table…
asked 2024-06-04T06:11:14.4766667+00:00
Vedant Desai
651
Reputation points
accepted 2024-06-07T09:52:14.62+00:00
Vedant Desai
651
Reputation points
1 answer
Building a chatbot via Azure
Hi Azure community! I want to build a chatbot for a company. It will be able to answer questions using the company data sources, including csv and pdf files. For example, if I ask the chatbot to summarize some csv financial reports, it will return an…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 92%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 61%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
2 answers
I am unable to mount containers using databricks and storage gen 2 ?
what is the issue?
asked 2024-06-05T15:57:33.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 14.000000000000002%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
smriti das
0
Reputation points
answered 2024-06-06T12:19:54.16+00:00
Luis Arias
5,676
Reputation points
2 answers
The synapse link is not able to write the data in Azure data lake gen 2 storage account
Hi Team, The synapse link is not able to write the data in Azure data lake gen 2 storage account. I am getting the following error message. {"code":"AuthorizationFailed","message":"The client…
asked 2024-05-24T08:27:24.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 60%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Abhishek Sajjan
0
Reputation points
commented 2024-06-06T02:01:47.0166667+00:00
AnnuKumari-MSFT
31,806
Reputation points Microsoft Employee
2 answers
Parameters for switch between linked services
I have a pipeline in Azure Synapse which has multiple dataflows and I have 2 linked service how can I use the pipeline parameters to switch between linked services without changing the linked services in each data manually.
asked 2024-05-27T17:49:37.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 41%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
ADITYA SINGH
0
Reputation points
commented 2024-06-06T01:40:32.1766667+00:00
AnnuKumari-MSFT
31,806
Reputation points Microsoft Employee
1 answer
Azure Data Lake integration with Business Central, API returns authorization error on signature string
Hello, I'm trying to integrate Business Central (SaaS) with Azure Data Lake. The Azure service uses Shared Access Key authorization. I am building my authorization string according to the API documentation: I decode my access key from base 64; I build my…
asked 2024-06-04T10:45:36.8066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 50%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Stefan Alexandru
0
Reputation points
edited an answer 2024-06-05T09:26:10.55+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
3,891
Reputation points Microsoft Vendor
1 answer
How to fix "the specify container does not exist"
Hi, When running the upgrade for my storage account to enable hierarchical namespace, I face an issue saying that “the specify container does not exist”. Can you help me to resolve this issue? Please find attached a screeshot of the issue I am following…
asked 2024-05-28T22:09:51.04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 61%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKW%3C/text%3E%3C/svg%3E)
Kiba, Wilfried
0
Reputation points
edited the question 2024-06-04T11:00:14.8166667+00:00
vipullag-MSFT
25,521
Reputation points
1 answer
Running specific pipelines from an Orchestration pipeline
How can I run individual pipelines from an Orchestration pipeline that has multiple pipelines running in parallel? I want to be able to choose which pipeline to run separately from the rest.
asked 2024-05-30T21:38:39.0933333+00:00
ADITYA SINGH
0
Reputation points
commented 2024-06-04T06:41:19.9466667+00:00
Harishga
5,270
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Call API with dynamic URL and store JSON results in DataLake Storage
I am trying to make implement the following scenario using Data Factory: I am making multiple API calls, relying on a dyanmic URL (e.g. "url.com/api/{ID}" for a list of IDs). The resulting JSON from each of the API calls should be stored as a…
asked 2024-05-30T11:55:42.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 13%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELK%3C/text%3E%3C/svg%3E)
Lucas Kluge
20
Reputation points
accepted 2024-06-04T06:12:25.94+00:00
Lucas Kluge
20
Reputation points
2 answers
How to convert type of any block blob to append blob inside adls gen2? Is there any process or activity which will convert the type of my entire blob without changing the content?
I have a json file inside a data lake gen2 storage and that json file blob type is a block blob. I want to convert that blob type to append blob through any services of azure. Need to know the process in detail.
asked 2024-05-31T06:30:48.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 36%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECN%3C/text%3E%3C/svg%3E)
Chakraborty, Nabarun (External)
0
Reputation points
answered 2024-06-03T12:09:39.6833333+00:00
Nehruji R
3,891
Reputation points Microsoft Vendor
2 answers
Azure to AWS
Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…
asked 2024-05-20T08:39:35.8366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sourav
80
Reputation points
edited an answer 2024-06-01T10:27:52.2833333+00:00
Sumarigo-MSFT
44,671
Reputation points Microsoft Employee
2 answers
Assert error output not writing to blob
Hello. I have built a pipeline which gathers data from a .csv file and then goes through a couple of assert activities to check the validity of the data. Valid rows are supposed to be input into the table, while assert-failure rows are setup to be…
asked 2024-05-28T10:17:13.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 92%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDL%3C/text%3E%3C/svg%3E)
Dejvis Leka
0
Reputation points
commented 2024-05-31T16:18:50.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,351
Reputation points Microsoft Employee