1,395 questions with Azure Data Lake Storage tags

Sort by: Updated
2 answers

Parameters for switch between linked services

I have a pipeline in Azure Synapse which has multiple dataflows and I have 2 linked service how can I use the pipeline parameters to switch between linked services without changing the linked services in each data manually.

Azure SQL Database
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,821 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,546 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
asked 2024-05-27T17:49:37.5766667+00:00
ADITYA SINGH 0 Reputation points
commented 2024-06-06T01:40:32.1766667+00:00
AnnuKumari-MSFT 31,731 Reputation points Microsoft Employee
1 answer

Azure Data Lake integration with Business Central, API returns authorization error on signature string

Hello, I'm trying to integrate Business Central (SaaS) with Azure Data Lake. The Azure service uses Shared Access Key authorization. I am building my authorization string according to the API documentation: I decode my access key from base 64; I build my…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
asked 2024-06-04T10:45:36.8066667+00:00
Stefan Alexandru 0 Reputation points
edited an answer 2024-06-05T09:26:10.55+00:00
Nehruji R 3,726 Reputation points Microsoft Vendor
1 answer

How to fix "the specify container does not exist"

Hi, When running the upgrade for my storage account to enable hierarchical namespace, I face an issue saying that “the specify container does not exist”. Can you help me to resolve this issue? Please find attached a screeshot of the issue I am following…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
asked 2024-05-28T22:09:51.04+00:00
Kiba, Wilfried 0 Reputation points
edited the question 2024-06-04T11:00:14.8166667+00:00
vipullag-MSFT 25,441 Reputation points
1 answer

Running specific pipelines from an Orchestration pipeline

How can I run individual pipelines from an Orchestration pipeline that has multiple pipelines running in parallel? I want to be able to choose which pipeline to run separately from the rest.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-30T21:38:39.0933333+00:00
ADITYA SINGH 0 Reputation points
commented 2024-06-04T06:41:19.9466667+00:00
Harishga 4,880 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Call API with dynamic URL and store JSON results in DataLake Storage

I am trying to make implement the following scenario using Data Factory: I am making multiple API calls, relying on a dyanmic URL (e.g. "url.com/api/{ID}" for a list of IDs). The resulting JSON from each of the API calls should be stored as a…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-30T11:55:42.17+00:00
Lucas Kluge 20 Reputation points
accepted 2024-06-04T06:12:25.94+00:00
Lucas Kluge 20 Reputation points
2 answers

How to convert type of any block blob to append blob inside adls gen2? Is there any process or activity which will convert the type of my entire blob without changing the content?

I have a json file inside a data lake gen2 storage and that json file blob type is a block blob. I want to convert that blob type to append blob through any services of azure. Need to know the process in detail.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,496 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,546 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,017 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-31T06:30:48.3333333+00:00
Chakraborty, Nabarun (External) 0 Reputation points
answered 2024-06-03T12:09:39.6833333+00:00
Nehruji R 3,726 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

How to copy files from sharepoint drive that has huge heirarcy of folders and subfolders

We want to get all files from the sharepoint into our ADLS container. In sharepoint drive there are arround 60k files but through our existing pipeline we are getting only 600 files. We have a web activity with below…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
10,029 questions
asked 2024-05-12T03:16:23.7+00:00
Rayne 40 Reputation points
edited a comment 2024-06-01T14:30:36.69+00:00
Mansi Yadav 40 Reputation points
2 answers

Azure to AWS

Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,017 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-20T08:39:35.8366667+00:00
Sourav 80 Reputation points
edited an answer 2024-06-01T10:27:52.2833333+00:00
Sumarigo-MSFT 44,501 Reputation points Microsoft Employee
2 answers

Assert error output not writing to blob

Hello. I have built a pipeline which gathers data from a .csv file and then goes through a couple of assert activities to check the validity of the data. Valid rows are supposed to be input into the table, while assert-failure rows are setup to be…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
asked 2024-05-28T10:17:13.85+00:00
Dejvis Leka 0 Reputation points
commented 2024-05-31T16:18:50.0033333+00:00
BhargavaGunnam-MSFT 28,111 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

How can I find help validating a certificate?

Indeed, I am a student in cloud computing in Morocco and I do not have enough means to obtain the AZ 900 certification and my family also does not have enough financial means. Could someone help me validate this certificate please?

Azure SQL Database
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,500 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-30T15:54:26.2366667+00:00
HAMIDOU YOUSSOUF 20 Reputation points
accepted 2024-05-31T14:55:42.77+00:00
HAMIDOU YOUSSOUF 20 Reputation points
0 answers

Possible bug or issue in Synapse dedicated SQL pool when exporting parquet files

I'm not sure if this is really a bug, but its definitly a frustration for me at least ;-). When trying to write data from Synapse dedicated SQL pools to data lake storage as parquet files (using a CETAS statement) it produces files with non-standard…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
asked 2023-04-06T14:05:05.62+00:00
demolition porg 11 Reputation points
commented 2024-05-30T14:07:55.3666667+00:00
Martin B 21 Reputation points
1 answer

Delete the file from SharePoint location

Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,017 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-22T10:16:27.3866667+00:00
ADF_Coder 0 Reputation points
commented 2024-05-30T06:38:40.1+00:00
ADF_Coder 0 Reputation points
1 answer

How to sync Azure data lake storage with sharepoint drive?

We have copied files from sharepoint site multiple drives having nested folders in Azure data lake storage container maintaining same folder structure. Now I want to create a pipeline in Azure data factory to delete file from ADLS container which is not…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
10,029 questions
asked 2024-05-12T15:00:05.1166667+00:00
Rayne 40 Reputation points
commented 2024-05-30T05:58:53.8666667+00:00
Haoyan Xue_MSFT 21,241 Reputation points Microsoft Vendor
0 answers

Changing Synapse Notebook variable values during deployment in Azure DevOps

How can I change the variable values in a Synapse Notebook, such as storage account name, container name, and file location, during deployment from dev to prod in Azure DevOps? The notebook is already being used by other notebooks in my dev environment.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,821 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,546 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
asked 2024-05-29T04:11:27.8566667+00:00
ADITYA SINGH 0 Reputation points
commented 2024-05-29T09:07:42.47+00:00
Harishga 4,880 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

How to use a user-delegation SAS token to load parquet table from ADLS gen2?

Now I have a parquet table stored in ADLS gen2: adlss://mycontainer@mystorage.dfs.core.windows.net/folder1/table1. This is a read-only table and I want to restrict my service principal to only have read access to this table only. So I use ACL to grant…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,821 questions
asked 2024-05-24T07:28:30.62+00:00
Shunlei Tang 40 Reputation points Microsoft Employee
commented 2024-05-28T12:55:46.9466667+00:00
Amrinder Singh 3,795 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Aritechture best practices to store data from websraping and use it in analytics

I am scrapping data from websites and I want to export what I scrapped as csv files then store them in Azure Data Lake, then apply an ETL and the final data output will be used for Power BI reports and for machine learning do I need to use Azure synapse?…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,661 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-28T08:27:28.46+00:00
WeirdMan 220 Reputation points
accepted 2024-05-28T08:35:56.1466667+00:00
WeirdMan 220 Reputation points
2 answers

How to setup modern Arcitechure for Small/Medium Business?

Currently we're using the following setup which is slow to process the data and is slow on the power bi side: Azure VM for third parties to upload via sftp C# script to ETL data to azure sql server and move files to ADLS Gen2 Power BI report pulling…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
7,381 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,017 questions
asked 2024-05-23T20:55:59.0633333+00:00
Jordan 5 Reputation points
answered 2024-05-28T04:54:01.1433333+00:00
Sumarigo-MSFT 44,501 Reputation points Microsoft Employee
1 answer

How Can I read csv files which are in nested folders and copy them preserving the hierarchy?

I have csv.gz files which are partitioned this way /2024/01/01/xyz/x.csv /2024/01/01/yza/y.csv /2024/01/01/zab/z.csv and there are files for several years and i want to copy all those files using adf while maintaining the folder structure and hierarchy…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,899 questions
asked 2024-05-24T16:33:37.1766667+00:00
Jay Remally 0 Reputation points
commented 2024-05-27T06:17:56.7533333+00:00
Harishga 4,880 Reputation points Microsoft Vendor
0 answers

Azure Synapse Link for dataverse to bring Dynamics 365 data to ADLS and Synapse for analytical purpose - Facing Issues and inconsistencies in the feature

We are using Synapse Link for Dataverse feature to bring dynamics 365 data to ADLS and Synapse Analytics. We have below open issues due to which we are unable to finalize the solution: In F&O linked dataverse environment, we have created a synapse…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,555 questions
Dynamics 365 Training
Dynamics 365 Training
Dynamics 365: A Microsoft cloud-based business platform that provides customer relationship management and enterprise resource planning solutions.Training: Instruction to develop new skills.
58 questions
Microsoft Power Platform Training
Microsoft Power Platform Training
Microsoft Power Platform: An integrated set of Microsoft business intelligence services.Training: Instruction to develop new skills.
154 questions
Microsoft Dataverse Training
Microsoft Dataverse Training
Microsoft Dataverse: A Microsoft service that enables secure storage and management of data used by business apps. Previously known as Common Data Service.Training: Instruction to develop new skills.
10 questions
asked 2024-05-21T04:28:25.19+00:00
Sriram M 0 Reputation points
edited a comment 2024-05-25T05:18:30.6533333+00:00
Sriram M 0 Reputation points
2 answers One of the answers was accepted by the question author.

VPN and networking config for secure upload to ADLS Gen2 storage account?

I'm looking into network architectures and security for implementing secure access from user laptops into Azure to upload files to ADLS Gen2 data lake blob containers. We have no on-prem network or AD - just individual user laptops. We do have MS Entra…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,395 questions
asked 2024-05-09T14:59:36.4666667+00:00
Peter Daniels 20 Reputation points
accepted 2024-05-23T12:32:13.1733333+00:00
Peter Daniels 20 Reputation points