Microsoft Q&A

Azure Data Lake Storage

929 questions

An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.

Browse all Azure tags

929 questions with Azure Data Lake Storage tags

Sort by: Updated
1 answer

I'm attempting to use Data Factory to transfer a CSV file from my computer to Azure Data Lake. I'm unable to select the path to the file I want to migrate from my local computer when I try to link the file system in the data factory's source tab.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,663 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,843 questions
asked 2023-03-09T19:43:27.6566667+00:00
Abhijith Yamsani 0 Reputation points
edited a comment 2023-03-23T07:23:43.6233333+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
1 answer

DataLakeFileSystemClient listPath, NumberFormatException when using String

Hello, I am new to this and trying to list all files in a container recursively. ListPathsOptions options = new ListPathsOptions(); options.setRecursive(true); for (PathItem pathItem :fileSystemClient.listPaths(options, null)){ ... } Doing so result…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-07T13:24:41.86+00:00
AUDIER Maxime 0 Reputation points
answered 2023-03-23T07:21:27.9633333+00:00
AUDIER Maxime 0 Reputation points
1 answer One of the answers was accepted by the question author.

How to fix MSSQL_101 "INSERTABLE columns list or UPDATABLE column list or key column list is not properly passed to the stored procedure." in Data Flow Sink?

I am trying to load CSV rows from Data Lake into an Azure SQL Database. I use the upsert functionality of the data flow and parameterize it to be able to load the same data into different tables. this is how i designed my data flow: In the alter row…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,663 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-15T09:32:16.0933333+00:00
Martin 50 Reputation points
commented 2023-03-23T07:08:02.6266667+00:00
AnnuKumari-MSFT 21,086 Reputation points Microsoft Employee
2 answers

how setup dfs insted blob endpoint for BlobContainerClient

Hi, In general I need to create app with java which will perform some operations on azure storage like upload file, append to file, rename, check if exist and so on. And IMPORTANT It has to communicate with DFS endpoint https://xxxx.dfs.core.windows.. …

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2021-10-25T06:14:34.32+00:00
Miko 1 Reputation point
edited the question 2023-03-23T07:04:38.1666667+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
1 answer

how config azure adls gen2 to migrate from hdfs

Could you please helpme to migrate data from on-premise hadoop to adls gen2, how can i config mi azure tenant, security netowork...

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-21T23:37:25.77+00:00
Edwin Vargas Antolinez 0 Reputation points
answered 2023-03-23T06:49:50.7733333+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee
0 answers

Can't query partitioned tables - Azure Synapse Link for Dataverse

Hi, we have a problem with accessing partitioned tables in Azure Synapse Workspace that were previously created by ingesting tables from Power Platform via Azure Synapse Link for Dataverse. Snapshot folder and .csv files are normally created but can't be…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,843 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-20T16:05:06.8433333+00:00
Leo Milos 5 Reputation points
commented 2023-03-22T18:27:40.7166667+00:00
BhargavaGunnam-MSFT 10,156 Reputation points Microsoft Employee
0 answers

Unable to mount ADLS

Hi All, I am trying to mount ADLS to Azure Databricks but getting this error - "shaded.databricks.org.apache.hadoop.fs.azure.AzureException: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container scd in account…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,287 questions
Azure Active Directory
Azure Active Directory
An Azure enterprise identity service that provides single sign-on and multi-factor authentication.
13,507 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-17T16:22:46.6333333+00:00
Sarvesh Pandey 71 Reputation points
commented 2023-03-21T22:03:29.21+00:00
BhargavaGunnam-MSFT 10,156 Reputation points Microsoft Employee
0 answers

How to mount dataset or datalake with read/write permission in Notebook of Azure Machine Learning Studio?

I've successfully mounted a dataset with read permission to notebook of my machine learning studio account. But when I try to write back to the datalake/dataset, it throughs "[Errno 30] Read-only file system". How can I mount the dataset with…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
1,708 questions
asked 2023-03-15T22:26:15.2333333+00:00
Milad Resketi 0 Reputation points
commented 2023-03-21T17:18:59.0766667+00:00
romungi-MSFT 29,711 Reputation points Microsoft Employee
0 answers

ADF Event based trigger not working for ADLS Gen 2 : when a file is manually uploaded or moved via adf pipeline or logic app trigger is working fine but when file is being moved via java code trigger is not working

I am using azure data factory where I have configured an event based trigger pointing to ADLS Gen 2 storage. The trigger expects a file to end with Completed-DS001_01.csv ("Blob Path Ends With") in the file location specified in "Blob…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,663 questions
asked 2021-03-17T09:10:07.887+00:00
Deepanshi Singh 1 Reputation point
commented 2023-03-21T13:09:07.22+00:00
Shilpi Prasad 0 Reputation points
1 answer

Copying data from synapses serverless SQL pool view to tables in the pool

Hi , I have created views for all D365 F&O tables in synapses serverless SQL pool using the CDMUtil solution . I am looking to transform the data from views to table in the same or a different DB , i.e. cache mechanism, to reduce the cost of call the…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,843 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
2,789 questions
Azure SQL Database
asked 2023-03-16T10:32:21.6366667+00:00
Athira Reghuraman 20 Reputation points
commented 2023-03-21T10:32:05.3433333+00:00
ShaikMaheer-MSFT 29,886 Reputation points Microsoft Employee
0 answers

How to distribute the contents of a spark dataframe over different azure storage containers

I want to build a multi-tenant deltalake based on a database which contains all data of all tenants. I am using Azure Synapse Pipelines and Spark Notebooks. In the database there is one table which contains all tenants. Besides that we have several other…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-03T07:54:52.65+00:00
Dirk Vrancken 21 Reputation points
commented 2023-03-21T07:24:12.7833333+00:00
Dirk Vrancken 21 Reputation points
1 answer

Copy data from SFTP to ADLS Gen 2 via ADF

I have around 200GB of data in SFTP server in .gz format. I need to copy that into ADLS via ADF. Can it be done.? what would be dataset file format that needs to be taken. The file format should also be the same .gz format in adls. There is no need for…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,287 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,663 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-16T06:21:50.2266667+00:00
Anonymous 0 Reputation points
commented 2023-03-20T22:34:19.2266667+00:00
BhargavaGunnam-MSFT 10,156 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

About backup of azure data lake storage gen2

Hi There! Please tell me how to get a backup of azure data lake storage gen2 online (without interrupting the process on the target resource). We would like to restore the data in azure data lake storage gen2 in case we accidentally delete the…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,520 questions
asked 2022-07-12T08:56:13.01+00:00
Yuta Harada 91 Reputation points
edited a comment 2023-03-20T14:08:53.6233333+00:00
Rahel Admasu 0 Reputation points
2 answers One of the answers was accepted by the question author.

This XML file does not appear to have any style information associated with it

Hi All, I am trying to download a CSV file from Azure blob storage. I encounter this error while opening in google chrome. Here is the URL. Any pointers on why I am not able to download …

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
1,520 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
1,667 questions
asked 2022-12-23T03:40:46.163+00:00
edited the question 2023-03-18T13:58:52.1666667+00:00
Sumarigo-MSFT 32,001 Reputation points Microsoft Employee
1 answer

Stream RealTime data from an On Prem Microsoft Server to a Azure Data Lake.

There is a need for Real Time data to be collected from a On Prem SQL Server. This means ADF is not suitable because it only does batch. Azure Streaming Analytics only connects to Azure SQL Server and not on prem. Has anybody else encountered this and…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
235 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
380 questions
asked 2023-03-16T02:02:54.12+00:00
Sallie Worsley 5 Reputation points
commented 2023-03-18T02:41:00.72+00:00
BhargavaGunnam-MSFT 10,156 Reputation points Microsoft Employee
1 answer

Getting "put: Status code: -1 error code: null error message: InvalidAbfsRestOperationException" error

Team, Getting below error while ingesting data from our VM to ADLS using below command and find below details. Command: curl -vkS sftp://user@domain.com:/FilePath/ingestion/file.eot | hadoop fs…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2023-03-10T09:14:42.3266667+00:00
Vinod K 0 Reputation points
answered 2023-03-18T02:14:28.2533333+00:00
BhargavaGunnam-MSFT 10,156 Reputation points Microsoft Employee
1 answer

CDMUtilSolution Console App error

Hi , I am trying to run CDMUtil_ConsoleApp for converting D365 F&O manifest to synapses DLL format , and I am getting below error . Please help to fix this issue .

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
2,789 questions
asked 2023-03-10T13:15:30.4966667+00:00
Athira Reghuraman 20 Reputation points
commented 2023-03-16T17:00:05.43+00:00
BhargavaGunnam-MSFT 10,156 Reputation points Microsoft Employee
1 answer

ADF DataFlow : BadRequest error with any additional information

Hello, I have a strange error when I only want to use an Excel dataset hosted in ADSL2 as input to a Dataflow. The preview of the data at the Dataset level is not a problem : Unfortunately it becomes impossible at the Input stage of…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,663 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
asked 2022-04-20T10:56:05.217+00:00
Adrien VLR 16 Reputation points
commented 2023-03-16T09:45:42.0666667+00:00
HerryKarmito 0 Reputation points
1 answer

how to delete a folder in ADLS with different name

Hi, I have a pipeline who generate parquet file in ADLS gen2 with similar naming patternfor each folder every day. bust some times a bug cause an empty folder generate in all folders with different naming. I would like to use delete activity befor my…

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,287 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,843 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
2,789 questions
asked 2023-03-13T14:15:03.5433333+00:00
17285674 0 Reputation points
commented 2023-03-16T07:57:14.4733333+00:00
AnnuKumari-MSFT 21,086 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Unable to create View in Azure synapses using CDMUtil_ConsoleApp

I had tried to create a view for D365 F&O data in synapses using CDMUtil_ConsoleApplication.exe to convert the manifest data to DLL . I got stuck with the below error . It has mentioned to create credential in manage identities and CREATE DATABASE…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
2,843 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
929 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
2,789 questions
asked 2023-03-10T16:32:17.3666667+00:00
Athira Reghuraman 20 Reputation points
commented 2023-03-15T06:45:03.87+00:00
PRADEEPCHEEKATLA-MSFT 56,301 Reputation points Microsoft Employee