Azure Data Lake Storage

1 answer

How can I find help validating a certificate?

Indeed, I am a student in cloud computing in Morocco and I do not have enough means to obtain the AZ 900 certification and my family also does not have enough financial means. Could someone help me validate this certificate please?

asked

HAMIDOU YOUSSOUF 20

accepted

HAMIDOU YOUSSOUF 20

0 answers

Possible bug or issue in Synapse dedicated SQL pool when exporting parquet files

I'm not sure if this is really a bug, but its definitly a frustration for me at least ;-). When trying to write data from Synapse dedicated SQL pools to data lake storage as parquet files (using a CETAS statement) it produces files with non-standard…

asked

demolition porg 11

commented

Martin B 21

1 answer

Delete the file from SharePoint location

Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…

asked

ADF_Coder 0

commented

ADF_Coder 0

1 answer

How to sync Azure data lake storage with sharepoint drive?

We have copied files from sharepoint site multiple drives having nested folders in Azure data lake storage container maintaining same folder structure. Now I want to create a pipeline in Azure data factory to delete file from ADLS container which is not…

asked

Rayne 20

commented

Haoyan Xue_MSFT 21,491 Microsoft Vendor

0 answers

Changing Synapse Notebook variable values during deployment in Azure DevOps

How can I change the variable values in a Synapse Notebook, such as storage account name, container name, and file location, during deployment from dev to prod in Azure DevOps? The notebook is already being used by other notebooks in my dev environment.

asked

ADITYA SINGH 0

commented

Harishga 5,270 Microsoft Vendor

1 answer

How to use a user-delegation SAS token to load parquet table from ADLS gen2?

Now I have a parquet table stored in ADLS gen2: adlss://mycontainer@mystorage.dfs.core.windows.net/folder1/table1. This is a read-only table and I want to restrict my service principal to only have read access to this table only. So I use ACL to grant…

asked

Shunlei Tang 40 Microsoft Employee

commented

Amrinder Singh 3,795 Microsoft Employee

1 answer

Aritechture best practices to store data from websraping and use it in analytics

I am scrapping data from websites and I want to export what I scrapped as csv files then store them in Azure Data Lake, then apply an ETL and the final data output will be used for Power BI reports and for machine learning do I need to use Azure synapse?…

asked

WeirdMan 220

accepted

WeirdMan 220

1 answer

How Can I read csv files which are in nested folders and copy them preserving the hierarchy?

I have csv.gz files which are partitioned this way /2024/01/01/xyz/x.csv /2024/01/01/yza/y.csv /2024/01/01/zab/z.csv and there are files for several years and i want to copy all those files using adf while maintaining the folder structure and hierarchy…

asked

Jay Remally 0

commented

Harishga 5,270 Microsoft Vendor

2 answers

VPN and networking config for secure upload to ADLS Gen2 storage account?

I'm looking into network architectures and security for implementing secure access from user laptops into Azure to upload files to ADLS Gen2 data lake blob containers. We have no on-prem network or AD - just individual user laptops. We do have MS Entra…

asked

Peter Daniels 20

accepted

Peter Daniels 20

1 answer

How can i create a linked service in ADF for Sharepoint online?

I want to extract files from sharepoint to ADL by only using ADF. I followed few steps Step1: Azure Active Directory -> Registered new app -> created new secret key I have the Tenant ID, ClientID(App ID), Secret Key Step2: Sharepoint online ->…

asked

Siddartha Reddy Jammula 20

commented

Nehruji R 3,891 Microsoft Vendor

0 answers

How can I get all files inside a drive irrespective of folder structure in ADF?

I want to copy files files from sharepoint drive which has lots of nested folders. Maximum hierarchy of folders is for 12 levels. Currently I'm using below endpoint in ADF's Web activity as it was mentioned in some of the articles that it provides every…

asked

Mansi Yadav 40

edited a comment

phemanth 7,670 Microsoft Vendor

1 answer

Having problem with Azure sandbox storage account

Hi, I am having issue to access to storage account in Azure sandbox environment. I have login to Azure sandbox environment, but not able to access to Azure storage account even though my subscription is selected as sandbox. Capture.PNG

asked

Weiy 20

edited a comment

Weiy 20

1 answer

Unable to copy share point files through ADF

I am facing issue now! I followed all the steps https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list?tabs=data-factory and tried to read pdf files from sharepoint through ADF. source: Sharepoint(PDF files)(I selected…

asked

Siddartha Reddy Jammula 20

commented

Haoyan Xue_MSFT 21,491 Microsoft Vendor

1 answer

Special character handling for file processing

Hello, I have some CSV files as feeds into datalake storage, this file will contain data /records with some special characters eg '/' We need to process the file from one container to another and we will need to remove some of the these special…

asked

Sourav 80

accepted

Sourav 80

1 answer

How to send the history of 3 months of files from one directory to another according to the date in its name

Hello, I'm having problems because I want to move the history of some files stored in a data lake according to the date in its name, but in the end what I manage to do is only move either one specific day or all of them. example: I have files October -…

asked

ALER 0

answered

phemanth 7,670 Microsoft Vendor

1 answer

How to mount dataset or datalake with read/write permission in Notebook of Azure Machine Learning Studio?

I've successfully mounted a dataset with read permission to notebook of my machine learning studio account. But when I try to write back to the datalake/dataset, it throughs "[Errno 30] Read-only file system". How can I mount the dataset with…

asked

Milad Resketi 10

answered

OPERTI Felipe 0

1 answer

Data Mesh architecture implementation on Azure data lake

Hi friends, Data mesh architecture is a decentralized approach that organizes data based on business domains (e.g., marketing, sales, HR), I have the following questions Is it required to build a separate data lake for each department? When data…

asked

Anshal 2,006

accepted

Anshal 2,006

1 answer

Why Data asset is not supported when try to create a AutoML job?

Hi, I successuffly created a data asset (folder_URI type) with uploaded imges in an Azure Blob storage (registered storage source), but when I try to create a job in Azure AutoML the Data asset shows (not supported). Any idea what is the issue? Thanks

asked

Gonzalez Barba, Jorge 0

commented

Brian Van Steen 20

2 answers

While fetching data from cosmos db conatiner and persisting the json file in ADLS Gen 2 through Synapse Pipeline, some objects in my json file are appearing as blank string causing data loss. This is happening in PROD only not in UAT and DEV.

Hi, The issue i am facing is with the Synapse Analytics Service - Pipeline. I have created a dataflow which is pulling data from cosmos db container in json format and storing that json file in ADLS Gen 2. When I check the json file in ADLS I see that…

asked

Vity Pandita 0

commented

ShaikMaheer-MSFT 38,316 Microsoft Employee

1 answer

Cannot read excel file which is in using adls using load_workbook of openpyxl in databricks

Cannot read excel file which is in using load_workbook of openpyxl but can read if copied to dbfs

asked

Alpha 20

commented

phemanth 7,670 Microsoft Vendor

Filter

Content

1,403 questions with Azure Data Lake Storage tags

How can I find help validating a certificate?

Possible bug or issue in Synapse dedicated SQL pool when exporting parquet files

Delete the file from SharePoint location

How to sync Azure data lake storage with sharepoint drive?

Changing Synapse Notebook variable values during deployment in Azure DevOps

How to use a user-delegation SAS token to load parquet table from ADLS gen2?

Aritechture best practices to store data from websraping and use it in analytics

How Can I read csv files which are in nested folders and copy them preserving the hierarchy?

VPN and networking config for secure upload to ADLS Gen2 storage account?

How can i create a linked service in ADF for Sharepoint online?

How can I get all files inside a drive irrespective of folder structure in ADF?

Having problem with Azure sandbox storage account

Unable to copy share point files through ADF

Special character handling for file processing

How to send the history of 3 months of files from one directory to another according to the date in its name

How to mount dataset or datalake with read/write permission in Notebook of Azure Machine Learning Studio?

Data Mesh architecture implementation on Azure data lake

Why Data asset is not supported when try to create a AutoML job?

While fetching data from cosmos db conatiner and persisting the json file in ADLS Gen 2 through Synapse Pipeline, some objects in my json file are appearing as blank string causing data loss. This is happening in PROD only not in UAT and DEV.

Cannot read excel file which is in using adls using load_workbook of openpyxl in databricks