1,541 questions with Azure Data Lake Storage tags

Sort by: Updated
0 answers

I need to verify my storage account's total capacity limit and check if I am reaching my quota. How can I confirm the maximum storage allocation and expand it if necessary?

I need to verify my storage account's total capacity limit and check if I am reaching my quota. How can I confirm the maximum storage allocation and expand it if necessary?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
277 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
8,379 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,377 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,088 questions
asked 2025-02-18T21:06:50.03+00:00
Lars Seagren 0 Reputation points
1 answer

I want to restrict azure ADLS SFTP access to directory level.

I want to created sftp for 5 user and want to maintain all the sftp folder in one container.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,377 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,088 questions
asked 2025-02-13T13:50:41.1066667+00:00
Thaya 0 Reputation points
edited a comment 2025-02-18T11:11:41.9033333+00:00
Vinod Kumar Reddy Chilupuri 2,465 Reputation points Microsoft Vendor
1 answer

Pyspark dataframe is taking too long to save on ADLS from Databricks.

I'm running a notebook on Azure databricks using a multinode cluster with 1 driver and 1-8 workers(each with 16 cores and 56 gb ram). Reading the source data from Azure ADLS which has 30K records. Notebook is consist of few transformation steps, also…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,331 questions
asked 2022-11-29T10:09:42.24+00:00
Pratik Roy 1 Reputation point
commented 2025-02-18T04:26:33.81+00:00
Madhur Tyagi 0 Reputation points
2 answers One of the answers was accepted by the question author.

How to use linked service in Notebook with pyspark

I have pyspark script in Notebook to read and write data in ADLS Gen2. Below is an sample of the pyspark script. But in the Synapse I only have a linked service created with Service Principle could connect to the ADLS Gen2, so I need to specify in…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2023-03-31T04:24:40.7333333+00:00
Shao Peng Sun 91 Reputation points
commented 2025-02-13T11:38:00.3+00:00
Rajan Sethi 5 Reputation points
3 answers

How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF

Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,331 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2024-01-04T06:25:53.11+00:00
Tom Young 0 Reputation points
commented 2025-02-12T09:26:43.0333333+00:00
Milankumar Soni, Meetkumar 0 Reputation points
1 answer

How to control access to a folder in ADLS gen2 container while Storage account IAMs are in action

Hi, I have a synapse pipeline that saves an output file in a folder (ex: salary) in an ADLS container (ex: employee). Now Mr. X wants the data saved in the folder to be accessible only to him but storage account level IAMs have already given access to…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
Azure Role-based access control
Azure Role-based access control
An Azure service that provides fine-grained access management for Azure resources, enabling you to grant users only the rights they need to perform their jobs.
882 questions
asked 2025-02-03T19:28:59.77+00:00
SP 5 Reputation points
commented 2025-02-11T20:26:42.5333333+00:00
Keshavulu Dasari 3,370 Reputation points Microsoft Vendor
3 answers One of the answers was accepted by the question author.

Medallion architecture in ADLS

I am trying to find the most suitable storage architecture for the following use case. I have several clients and I need isolated storages so data cannot be mixed up I work with 3 different environments for each client: dev, pre, pro I need to…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,377 questions
asked 2025-02-08T11:20:54.93+00:00
Bravo Arribas, Miguel 25 Reputation points
commented 2025-02-11T01:56:24.1833333+00:00
Sathvika Reddy Dopathi 90 Reputation points Microsoft Vendor
0 answers

Connecting to Azure Data Lake Storage Gen2 from Tableau Desktop is throwing error

Error Receiving in Tableau Desktop: Tableau received an OAuth error from your request. Please see the error message for more information: User authorization failed (invalid_client). When checked the auth url it seems the url is the one like below -…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
asked 2025-01-31T21:06:38.6066667+00:00
Deep Saha 0 Reputation points
edited the question 2025-02-07T21:58:17.5+00:00
Sanoop M 765 Reputation points Microsoft Vendor
1 answer

Unable to create a directory named space character using ADLS Gen2 REST API

I am unable to create a directory in "Azure Data Lake Storage Gen2" that is named as just a single space character, despite reviewing the documentation and there being no indication this is a disallowed name. My primary access to "Azure…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
asked 2025-01-16T15:07:47.1866667+00:00
Phil R 0 Reputation points
commented 2025-02-05T17:57:10.7566667+00:00
Keshavulu Dasari 3,370 Reputation points Microsoft Vendor
1 answer

generate SAS tokens from serviceprincipal credentials

I am working on to create java client that generates sas tokens for the given service principal credentials. I am taking a reference from…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
asked 2025-01-08T12:00:29.36+00:00
SHYAMALA GOWRI 90 Reputation points
edited the question 2025-02-03T15:26:20.9866667+00:00
Givary-MSFT 35,296 Reputation points Microsoft Employee
1 answer

Need to extract zreports from SAP HANA

Hi, I have a use-case where I want to extract data from SAP Hana. The use-case is as follows: I have SAP Hana deployment from where I need to extract data. The data is stored in zreports which are extracted using T-Codes. Now, I want to extract the data…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-01-30T05:24:40.5533333+00:00
Om Bhimani 0 Reputation points
commented 2025-02-03T13:07:16.4666667+00:00
Smaran Thoomu 20,295 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

We want to know details about azure local , can we use databricks , adlsg2, datafactory, machine learning on azure local ?

We want to know details about azure local , can we use databricks , adlsg2, datafactory, machine learning on azure local?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,136 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,331 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
Azure Local
asked 2025-02-03T08:34:20.4466667+00:00
Mohammad Mahamudul Hasan 20 Reputation points
accepted 2025-02-03T10:09:22.4133333+00:00
Mohammad Mahamudul Hasan 20 Reputation points
3 answers

ADF Copy Data JSON Source dynamic schema mapping

Hi I am working on ADF Copy data activity. HTTP Dataset is returning a JSON with the following sample JSON output { "totalRowCount": 1, "data": [ { "ProductCode": "P - 1", …

Azure SQL Database
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,331 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2022-01-27T08:33:23.507+00:00
Mav 81 Reputation points
answered 2025-02-03T07:57:28.8533333+00:00
vishnu 0 Reputation points
1 answer

How to fix "Failed to upload block This request is not authorized to perform this operation." error while writing a csv to ADLS Gen2 using synapse notebook

I am trying to write a dataframe into a csv file on ADLS Gen2 using Synapse Notebook. I am using pandas to_csv as below. The 'linkedService': 'xxxxx' uses system assigned managed identity which has the "Storage Blob Data Contributor" role…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-01-30T21:07:29.7933333+00:00
Pranav Babu - O365 Admin Account 0 Reputation points
commented 2025-02-03T00:36:57.5033333+00:00
Chandra Boorla 8,470 Reputation points Microsoft Vendor
1 answer

I am unable to load a table to fabric lakehouse

I am unable to load tables to fabric lakehouse. I was having no issues until recently but now when I publish a table from gen 2 dataflow the table in the lakehouse has all blank rows. In the SQL endpoint they are all null values.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
asked 2025-01-29T14:54:03.4433333+00:00
John Larson 0 Reputation points
answered 2025-01-29T15:33:46.29+00:00
Hari Babu Vattepally 1,720 Reputation points Microsoft Vendor
1 answer

How to write the query on Azure to get the information of particular SPN linked with all the folders present in containers..

Hi, As there are multiple folders are preset into Azure storage containers, each folders of containers linked with some SPN. I am looking for the script to create to find, which SPN associate with which all folders?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Container Apps
Azure Container Apps
An Azure service that provides a general-purpose, serverless container platform.
541 questions
asked 2025-01-13T14:36:48.55+00:00
Saurabh Parmar 0 Reputation points
edited a comment 2025-01-29T06:42:59.95+00:00
Nandamuri Pranay Teja 805 Reputation points Microsoft Vendor
1 answer

Monitor AutoResolve Integration Runtime in Azure Data Factory

Hi all I need to monitor the Integration Runtime performance for a pipeline that uses stored procedure activity to load data from ADLS to Snowflake. The stored procedure in this case resides in snowflake itself and it’s called from Snowflake Tenant to…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-01-24T06:12:53.87+00:00
AyusmanBasu-6332 6 Reputation points
commented 2025-01-29T03:46:23.0266667+00:00
Smaran Thoomu 20,295 Reputation points Microsoft Vendor
1 answer

ErrorCode=AdlsGen2OperationFailed...Failure happened on 'Sink' side.

Getting the below error while copying the data to ADLS from SAP Hana, however this error is random to tables failing daily. When I try load each of the failed tables from the same source and with the same file name then there is no issue and it works as…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-01-25T11:59:38.9966667+00:00
Tyagi, Amit 5 Reputation points
commented 2025-01-28T06:39:40.6466667+00:00
phemanth 13,800 Reputation points Microsoft Vendor
1 answer

I am integrating the dataflow activity which removes duplicates from the csv file to the existing pipeline i have. I need a help to configure the parameters.

I created a dataflow activity which removes duplicates from the source files( csv formatted files). So now i integrated this dataflow activity to existing pipeline i have. but i am not able to parametrize the source files. It would be helpful if i get…

Azure SQL Database
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,377 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,088 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-01-24T08:02:14.5133333+00:00
Lakshmi Moulya Nerella 0 Reputation points
commented 2025-01-28T03:42:01.08+00:00
phemanth 13,800 Reputation points Microsoft Vendor
1 answer

DP-203 Labs

Is there any way to do all the labs for DP-203 without costing alot of money AND without taking like 10 minutes to install something? For each lab, the ./setup.ps1 file takes like 10 or 15 minutes to load. I'd like to not have to delete the resources…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,331 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2024-07-21T00:03:12.49+00:00
Ben Cohn 10 Reputation points
edited the question 2025-01-27T16:17:38.78+00:00
Rakesh Gurram 12,200 Reputation points Microsoft Vendor