1,348 questions with Azure Data Lake Storage tags

Sort by: Updated
2 answers

azure file share map drive with AAD authentication without joining the domain?

through Storage key i am able to map the azure file share drive but instead of storage access key authentication how to enable azure file share mapping through AAD authentication and our laptops are workstation not AD joined? Also while creating the…

Azure Files
Azure Files
An Azure service that offers file shares in the cloud.
1,169 questions
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,714 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,436 questions
asked 2020-07-22T07:40:59.737+00:00
ever 1 Reputation point
answered 2020-07-27T04:35:45.227+00:00
HarithaMaddi-MSFT 10,131 Reputation points
1 answer

Error at create spark database in synapse workspace notebook using orchestrate pipeline.

I'm getting below error while running synapse notebook from orchestrate in synapse workbook. My notebook runs fine when ran individually from Develop. I gave Owner and Storage Blob Data Owner role to myself before running the pipeline as mentioned in…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,395 questions
asked 2020-07-21T07:01:54.63+00:00
Rishabh 11 Reputation points
commented 2020-07-23T22:04:55.32+00:00
MartinJaffer-MSFT 26,026 Reputation points
1 answer

Azure Data Lake Gen2 Disaster Recovery in case of failover , we dont find an option for Manual Failover?

I am planning for ADLS gen 2 storage with GA-GRS option to have replicated data from one to another region. DR is capable for Azure Data Lake Storage Gen 2, its just an automatic process that Microsoft kicks off in the event the storage account…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-07-10T09:34:02.02+00:00
Anusha Kurmindla 11 Reputation points
commented 2020-07-22T10:18:33.337+00:00
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee
1 answer

Azure Data Lake Gen2 Snapshots and point in time recovery isnt available?

I would like to enable snapshots or point in time recovery for azure Data Lake Store Gen2 but don't find any option for it. How do we restore data in case of accidental deletion by user and i would like to test it in my environment once.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-07-10T10:16:16.717+00:00
Anusha Kurmindla 11 Reputation points
commented 2020-07-22T10:17:03.72+00:00
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

$log container Name not in ARM template of azure Storage but it is deplyo on Target

HI All, i generate a ARM template for azure storage account- in JSON i didn't find any $log container, but after deployment i found a $log container. I Try to delete this container $log but not able to delete. message- i don't have permission. i…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,714 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,436 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,938 questions
asked 2020-07-03T09:52:24.953+00:00
manish verma 421 Reputation points
accepted 2020-07-20T14:34:33.553+00:00
manish verma 421 Reputation points
1 answer

Appending an empty file from local machine to ADLSGen2 Storage's file.

Does it make sense to append an empty file to an existing file in Azure Data Lake Storage Gen2? I have a csv file in Azure Data Lake Storage Gen2, I'm trying to append data to the same via REST API Call. In one of the scenario the data to be…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-07-08T14:49:44.147+00:00
J, Praveen Kumar 1 Reputation point
commented 2020-07-17T08:07:00.13+00:00
ChiragMishra-MSFT 951 Reputation points
0 answers

Data lake storage Gen 2 timeout issue

Facing timeout issue if we access list of paths in a selected file system. Using azure-storage-file-datalake 12.1.2 jdk 11. fileSystemClient.listPaths().forEach(path -> System.out.printf("Name: %s%n", path.getName())); Working fine…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-07-08T13:27:12.427+00:00
vidya sagar 1 Reputation point
commented 2020-07-17T08:05:58.503+00:00
ChiragMishra-MSFT 951 Reputation points
0 answers

Data migration to Azure data lake using sas failes suddenly

HI , i have a process which is copying data to azure data lake using sas key. from 27th june 2020 around 4:00 am ust it stopeed suddenly giving error message "java.util.NoSuchElementException: An error occurred while enumerating the result, check…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-06-30T12:45:27.587+00:00
Gigi Thomas 1 Reputation point
commented 2020-07-15T19:19:34.73+00:00
MartinJaffer-MSFT 26,026 Reputation points
1 answer One of the answers was accepted by the question author.

What is the best way to query data from Azure Data Lake ?

I am ingesting all variety of data in Azure Data Lake Gen 2(Azure Blob Storage with Hierarchical namespace enabled)? What is the best way to index data and to query data from Data Lake. Data Ingested can be in form of CSV, Json, XML files.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-07-15T11:30:01.487+00:00
Abhishek Gupta 26 Reputation points
commented 2020-07-15T12:07:40.967+00:00
Abhishek Gupta 26 Reputation points
1 answer One of the answers was accepted by the question author.

Uploading files from OneDrive to Azure Data Lake Gen2

This is the situation. A client of my has XML files coming from Peoplesoft. I would like the client to place them in a inbox on a OneDrive location. With Azure Logic Apps i want to create a trigger that sends an emial when there is a new file placed in…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,855 questions
asked 2020-07-08T05:58:47.92+00:00
Jacobus Wooning 41 Reputation points
commented 2020-07-14T10:23:13.77+00:00
Pramod Valavala 20,591 Reputation points Microsoft Employee
0 answers

java.io.IOException: Stream is closed! Error in HDInsight with ADLS Gen 2

I am currently using Hail for the pyspark library to perform varying operations on Genomic data in ADLS Gen 2 with an HDInsight 4.0, Spark 2.4 cluster. I have been in touch with the development team regarding this error I have been getting when running a…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
199 questions
Azure Data Lake Analytics
asked 2020-06-29T14:55:04.947+00:00
EagleByte 1 Reputation point
commented 2020-07-10T17:49:55.523+00:00
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
1 answer

Can't process ORC files in Data Factory : ErrorCode=ParquetJavaInvocationException

Hi, Our organisation uses ORC formatted files for our central file storage, in our data factory I am unable to process most of the ORC files, I am only able to process really small files (< 1mb) for all the other ORC files the pipeline fails to run. …

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,599 questions
asked 2020-07-07T09:04:54.987+00:00
Joris 6 Reputation points
commented 2020-07-08T21:31:08.813+00:00
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

How to pull the data from Dynamic 365 to Data lake

<p>Hi <a rel="user" nodeId="74270" href="/answers/users/74270/chiragmishramsft-1092.html">@ChiragMishraMSFT-1092</a> </p> <p>Can you pls help me with the below error:</p> <p>{…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,599 questions
asked 2020-06-30T06:16:02.907+00:00
Anonymous
commented 2020-07-01T20:59:37+00:00
KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee
1 answer

Azure Data Lake storage file directory listing Json Serialization Error

Hi, I am having troubles in listing the directories in in the Azure Data Lake storage. I am pretty much using the default template for listing the directories on a file system according to data-lake-storage-directory-file-acl-dotnet . I have wrapped…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-06-09T14:52:41.037+00:00
Cebiroglu Gökhan (MT-A) 1 Reputation point
commented 2020-07-01T10:54:31.88+00:00
ChiragMishra-MSFT 951 Reputation points
1 answer One of the answers was accepted by the question author.

how to replicate ongoing changes of on prem database to data lake?

Like AWS DMS handles ongoing replication and update bucket with respective files, what is available in azure to manage ongoing changes occurring at on-prem sql database? Initially it requires to do a full load in my data lake storage which can be…

Azure Database Migration service
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,297 questions
Azure Migrate
Azure Migrate
A central hub of Azure cloud migration services and tools to discover, assess, and migrate workloads to the cloud.
718 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,599 questions
asked 2020-06-19T17:52:50.293+00:00
Seuli Das Majumder 101 Reputation points
accepted 2020-06-25T13:31:24.097+00:00
Seuli Das Majumder 101 Reputation points
2 answers

Extract the date hierarchy from previous data in Data Lake Partition

Hi Team, Would be of great help to get some clear assistance on the below query. Data source is Dynamic365 & SAP Hana. My requirement is to create a data lake partition(ADLS Gen2) with date…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,599 questions
asked 2020-06-07T07:02:45.193+00:00
RahulK 101 Reputation points
commented 2020-06-25T07:39:37.837+00:00
ChiragMishra-MSFT 951 Reputation points
1 answer One of the answers was accepted by the question author.

Can a new user assign to Posix style ACL's without RBAC in Data Lake storge gen2

Hi all, i have a active directory user say A i want to assign him Posix style ACL's- folder permission without RABC permission. it is possible or not please confirm

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-06-15T10:09:26.967+00:00
manish verma 421 Reputation points
accepted 2020-06-24T09:45:18.06+00:00
manish verma 421 Reputation points
1 answer

how can we create a custom role in Data Lake Store Gen 2 has access on azure storage user can view folder and container but not read file data

HI All, in Data Lake Gen2. we have one requirement- we need to create a custom role-adding user in this role User- should be able to view folder , but not able to read data. how can we create a custom role to get this requirement

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
asked 2020-06-09T12:07:28.467+00:00
manish verma 421 Reputation points
commented 2020-06-18T11:48:42.797+00:00
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee
1 answer

Access Control Exception on setting data lake storage gen1 as output

Hi, I am receiving JSON data as input of stream analytics job from IOT hub and trying to write it in Data Lake Storage Gen-1. The stream analytics job has read, write and execute permissions and I have read permission to Data Lake storage. According to…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure IoT Hub
Azure IoT Hub
An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.
1,124 questions
Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
331 questions
asked 2020-04-19T12:33:39.407+00:00
Tanishq Singh 21 Reputation points
commented 2020-06-18T11:46:28.45+00:00
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee
1 answer

More convenient service to read avro files from Azure Data Lake Gen2

Hi, I have to read lots of avro files created by an Event Hub Capture in a Data Lake Gen2. Data must be filtered, processed and then applied to train a machine learning model. I'm considering Azure Databricks and the Azure Machine Learning service…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,348 questions
Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,571 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
559 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,938 questions
asked 2020-06-08T22:37:56.897+00:00
Ariel Cedola 21 Reputation points
commented 2020-06-15T20:20:36.227+00:00
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee