1,371 questions with Azure Data Lake Storage tags

Sort by: Updated
1 answer

Delete the file from SharePoint location

Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,972 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
asked 2024-05-22T10:16:27.3866667+00:00
ADF_Coder 0 Reputation points
commented 2024-05-27T17:05:23.9833333+00:00
ADF_Coder 0 Reputation points
1 answer

Have previously been able to map a received dataset to data store. But have been unable to do so today - OK button does not become enabled after selecting a data store folder. Has something changed?

I received some (Azure Data Lake Storage Gen2 Folder) datasets from an outside (partner) organization through a data share invite. Previously was able to map the dataset to a folder in one of my (Azure Blob Storage) data stores, but today the OK button…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
asked 2024-05-23T23:29:39.0033333+00:00
Richard Krueger 0 Reputation points
answered 2024-05-27T14:08:01.6733333+00:00
Nehruji R 3,041 Reputation points Microsoft Vendor
1 answer

How to use a user-delegation SAS token to load parquet table from ADLS gen2?

Now I have a parquet table stored in ADLS gen2: adlss://mycontainer@mystorage.dfs.core.windows.net/folder1/table1. This is a read-only table and I want to restrict my service principal to only have read access to this table only. So I use ACL to grant…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,763 questions
asked 2024-05-24T07:28:30.62+00:00
Shunlei Tang 20 Reputation points Microsoft Employee
edited a comment 2024-05-27T08:59:30.2966667+00:00
Shunlei Tang 20 Reputation points Microsoft Employee
1 answer

Azure to AWS

Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,972 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
asked 2024-05-20T08:39:35.8366667+00:00
Sourav 80 Reputation points
edited a comment 2024-05-27T06:31:00.0666667+00:00
Sumarigo-MSFT 44,081 Reputation points Microsoft Employee
1 answer

How Can I read csv files which are in nested folders and copy them preserving the hierarchy?

I have csv.gz files which are partitioned this way /2024/01/01/xyz/x.csv /2024/01/01/yza/y.csv /2024/01/01/zab/z.csv and there are files for several years and i want to copy all those files using adf while maintaining the folder structure and hierarchy…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
asked 2024-05-24T16:33:37.1766667+00:00
Jay Remally 0 Reputation points
commented 2024-05-27T06:17:56.7533333+00:00
Harishga 4,250 Reputation points Microsoft Vendor
1 answer

The synapse link is not able to write the data in Azure data lake gen 2 storage account

Hi Team, The synapse link is not able to write the data in Azure data lake gen 2 storage account. I am getting the following error message.  {"code":"AuthorizationFailed","message":"The client…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,474 questions
asked 2024-05-24T08:27:24.22+00:00
Abhishek Sajjan 0 Reputation points
answered 2024-05-25T13:25:32.4366667+00:00
Vinodh247-1375 11,476 Reputation points
0 answers

Azure Synapse Link for dataverse to bring Dynamics 365 data to ADLS and Synapse for analytical purpose - Facing Issues and inconsistencies in the feature

We are using Synapse Link for Dataverse feature to bring dynamics 365 data to ADLS and Synapse Analytics. We have below open issues due to which we are unable to finalize the solution: In F&O linked dataverse environment, we have created a synapse…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,474 questions
Dynamics 365 Training
Dynamics 365 Training
Dynamics 365: A Microsoft cloud-based business platform that provides customer relationship management and enterprise resource planning solutions.Training: Instruction to develop new skills.
83 questions
Microsoft Power Platform Training
Microsoft Power Platform Training
Microsoft Power Platform: An integrated set of Microsoft business intelligence services.Training: Instruction to develop new skills.
118 questions
Microsoft Dataverse Training
Microsoft Dataverse Training
Microsoft Dataverse: A Microsoft service that enables secure storage and management of data used by business apps. Previously known as Common Data Service.Training: Instruction to develop new skills.
10 questions
asked 2024-05-21T04:28:25.19+00:00
Sriram M 0 Reputation points
edited a comment 2024-05-25T05:18:30.6533333+00:00
Sriram M 0 Reputation points
0 answers

How to setup modern Arcitechure for Small/Medium Business?

Currently we're using the following setup which is slow to process the data and is slow on the power bi side: Azure VM for third parties to upload via sftp C# script to ETL data to azure sql server and move files to ADLS Gen2 Power BI report pulling…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,289 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,972 questions
asked 2024-05-23T20:55:59.0633333+00:00
Jordan 5 Reputation points
2 answers One of the answers was accepted by the question author.

VPN and networking config for secure upload to ADLS Gen2 storage account?

I'm looking into network architectures and security for implementing secure access from user laptops into Azure to upload files to ADLS Gen2 data lake blob containers. We have no on-prem network or AD - just individual user laptops. We do have MS Entra…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
asked 2024-05-09T14:59:36.4666667+00:00
Peter Daniels 20 Reputation points
accepted 2024-05-23T12:32:13.1733333+00:00
Peter Daniels 20 Reputation points
1 answer One of the answers was accepted by the question author.

How can i create a linked service in ADF for Sharepoint online?

I want to extract files from sharepoint to ADL by only using ADF. I followed few steps Step1: Azure Active Directory -> Registered new app -> created new secret key I have the Tenant ID, ClientID(App ID), Secret Key Step2: Sharepoint online ->…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
9,875 questions
asked 2024-04-27T21:58:38.2233333+00:00
Siddartha Reddy Jammula 20 Reputation points
commented 2024-05-22T05:08:28.6733333+00:00
Nehruji R 3,041 Reputation points Microsoft Vendor
1 answer

How to sync Azure data lake storage with sharepoint drive?

We have copied files from sharepoint site multiple drives having nested folders in Azure data lake storage container maintaining same folder structure. Now I want to create a pipeline in Azure data factory to delete file from ADLS container which is not…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
9,875 questions
asked 2024-05-12T15:00:05.1166667+00:00
Rayne 20 Reputation points
commented 2024-05-21T11:12:43.21+00:00
Nehruji R 3,041 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

How to copy files from sharepoint drive that has huge heirarcy of folders and subfolders

We want to get all files from the sharepoint into our ADLS container. In sharepoint drive there are arround 60k files but through our existing pipeline we are getting only 600 files. We have a web activity with below…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
asked 2024-05-12T03:16:23.7+00:00
Rayne 20 Reputation points
accepted 2024-05-21T10:26:36.3033333+00:00
Rayne 20 Reputation points
0 answers

How can I get all files inside a drive irrespective of folder structure in ADF?

I want to copy files files from sharepoint drive which has lots of nested folders. Maximum hierarchy of folders is for 12 levels. Currently I'm using below endpoint in ADF's Web activity as it was mentioned in some of the articles that it provides every…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
asked 2024-05-14T00:43:50.9066667+00:00
Mansi Yadav 0 Reputation points
edited a comment 2024-05-20T09:47:49.2033333+00:00
phemanth 6,710 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Having problem with Azure sandbox storage account

Hi, I am having issue to access to storage account in Azure sandbox environment. I have login to Azure sandbox environment, but not able to access to Azure storage account even though my subscription is selected as sandbox. Capture.PNG

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Cloud Services
Azure Cloud Services
An Azure platform as a service offer that is used to deploy web and cloud applications.
652 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,763 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,469 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,047 questions
asked 2024-05-11T05:36:05.7433333+00:00
Weiy 20 Reputation points
edited a comment 2024-05-18T13:41:51.58+00:00
Weiy 20 Reputation points
1 answer

Unable to copy share point files through ADF

I am facing issue now! I followed all the steps https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory and tried to read pdf files from sharepoint through ADF. source: Sharepoint(PDF files)(I selected…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
9,875 questions
asked 2024-05-11T12:02:41.84+00:00
Siddartha Reddy Jammula 20 Reputation points
commented 2024-05-17T08:37:19.2666667+00:00
Haoyan Xue_MSFT 20,906 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Special character handling for file processing

Hello, I have some CSV files as feeds into datalake storage, this file will contain data /records with some special characters eg '/' We need to process the file from one container to another and we will need to remove some of the these special…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,972 questions
asked 2024-05-15T02:02:23.0533333+00:00
Sourav 80 Reputation points
accepted 2024-05-16T07:24:24.5+00:00
Sourav 80 Reputation points
1 answer

How to send the history of 3 months of files from one directory to another according to the date in its name

Hello, I'm having problems because I want to move the history of some files stored in a data lake according to the date in its name, but in the end what I manage to do is only move either one specific day or all of them. example: I have files October -…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
Azure Data Lake Analytics
asked 2024-05-14T22:35:22.23+00:00
ALER 0 Reputation points
answered 2024-05-15T09:53:51.7833333+00:00
phemanth 6,710 Reputation points Microsoft Vendor
1 answer

How to mount dataset or datalake with read/write permission in Notebook of Azure Machine Learning Studio?

I've successfully mounted a dataset with read permission to notebook of my machine learning studio account. But when I try to write back to the datalake/dataset, it throughs "[Errno 30] Read-only file system". How can I mount the dataset with…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,614 questions
asked 2023-03-15T22:26:15.2333333+00:00
Milad Resketi 5 Reputation points
answered 2024-05-15T09:41:00.78+00:00
OPERTI Felipe 0 Reputation points
1 answer

My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a

Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,972 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,762 questions
asked 2024-05-14T19:23:38.4366667+00:00
Shashwat Tiwary 0 Reputation points
edited an answer 2024-05-15T04:39:12.8666667+00:00
Sumarigo-MSFT 44,081 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Data Mesh architecture implementation on Azure data lake

Hi friends, Data mesh architecture is a decentralized approach that organizes data based on business domains (e.g., marketing, sales, HR), I have the following questions Is it required to build a separate data lake for each department? When data…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,371 questions
asked 2024-05-07T09:29:55.1333333+00:00
Anshal 1,986 Reputation points
accepted 2024-05-13T13:59:26.5033333+00:00
Anshal 1,986 Reputation points