1,360 questions with Azure Data Lake Storage tags

Sort by: Updated
0 answers

How to store set variable json array output into csv file in data factory

I used the Lookup activity under for each loop and used Store Procedure which gave me some record as an output if the condition is fails So that output I have stored into the Set Variable and then those values I need to store into csv file whose file…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-04-26T18:01:30.62+00:00
Mohit Bokade 0 Reputation points
commented 2024-05-02T13:44:22.6766667+00:00
Smaran Thoomu 10,245 Reputation points Microsoft Vendor
1 answer

Azure Synapse Link with F&O missing tables

Hello, I'm facing this problem with synapse link, I did: Connect F&O environment with power apps. Established an incremental link with data lake. I can see D365 Finance and Operation in the list of tables in manage tables of the link. My problem…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,432 questions
asked 2024-03-19T08:57:53.9533333+00:00
Ameer Jaber 0 Reputation points
edited a comment 2024-05-02T10:10:16.0533333+00:00
Rupa Kamble (Accenture Solutions Private Li) 0 Reputation points Microsoft Vendor
2 answers

Unable to create Storage Event trigger in Synapse Data Factory

I want to create a pipeline triggered by an event in my storage account : when a blod is created inside.nWhen I want to publish that event trigger in synapse, I have this error : The client 'd4d9f262-75fa-4138-845c-019afa12cf7a' with object id…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,432 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-04-30T12:22:49.8033333+00:00
Junior Steve KAMDEM DJOKO 0 Reputation points
answered 2024-05-02T09:11:16.7+00:00
Anand Prakash Yadav 6,235 Reputation points Microsoft Vendor
2 answers

Copy LOB based data from DB2 to Azure Data Lake Gen2 storage using ADF

Hi team, I was trying to work on a scenario to copy data from a table in db2 database having a column with datatype as LOB(CLOB) to Azure Data Lake Gen2 storage container using Azure Data Factory. However, its taking a considerable amount of time( a day)…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-04-21T03:40:26.3+00:00
Debajit Mitra 0 Reputation points
commented 2024-05-01T10:02:15.8666667+00:00
Amira Bedhiafi 15,991 Reputation points
3 answers

Filtering data for last 24 months in Mapping Data Flows

How can I filter data using Mapping Data Flows for the last 24 months, starting from the max date and going back? The date column is in the mm/dd/yyyy format.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,432 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-03-19T01:29:58.9966667+00:00
ADITYA SINGH 0 Reputation points
edited an answer 2024-05-01T08:15:23.97+00:00
Pinaki Ghatak 2,400 Reputation points Microsoft Employee
0 answers

How to fix error Job failed due to reason: com.microsoft.dataflow.broker.InvalidPayload

Operation on target Country failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: com.microsoft.dataflow.broker.InvalidPayloadException: Fail to validate with reason: Retry Request, fail to…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,729 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-04-29T13:20:24.6633333+00:00
Dhruva B 50 Reputation points
commented 2024-04-30T15:54:52.89+00:00
BhargavaGunnam-MSFT 26,501 Reputation points Microsoft Employee
3 answers

How to send a mail notification for a failed pipeline in Azure Synapse Analytics?

How can I send a notification email to a specific email address without using a logic app when one of my Synapse Analytics pipelines fails? I would like to include the error message in the email notification.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,457 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,432 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-03-20T15:48:58.53+00:00
Aditya Singh 105 Reputation points
commented 2024-04-30T15:30:48.9533333+00:00
Roberto Cuevas 5 Reputation points
3 answers One of the answers was accepted by the question author.

Azure blob storage gen2

I need to restrict uploading files to Blob storage at 18 MB size and allow only .txt extension

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
asked 2024-04-26T10:54:48.57+00:00
Chandru Sati 20 Reputation points
accepted 2024-04-29T01:54:10.3433333+00:00
Chandru Sati 20 Reputation points
1 answer

Azure logic app , extract file from sharepoint to ADLS

Hi, I am working on a file extraction from sharepint to ADLS and the file size is 1 GB. I need to copy the file to ADLS. I am currently using standard logic app as the file size is more than the default size. After that i tried adding the file size in…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,877 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
9,762 questions
asked 2024-04-26T20:06:43.52+00:00
Ritu Harivansh Prajapati 0 Reputation points
answered 2024-04-27T04:20:28.6433333+00:00
Abhishek Tripathi 0 Reputation points
0 answers

Download data from cosmos by v-studio

When I download data from cosmos from azure data lake storage of visual studio, it shows an error: 流 URL 无效

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
asked 2024-04-16T12:07:24.16+00:00
Xinlong Wang 0 Reputation points Microsoft Employee
commented 2024-04-26T21:33:55.55+00:00
KarishmaTiwari-MSFT 18,602 Reputation points Microsoft Employee
1 answer

Copy Data activity fails when copying from ADLS to ADLS

I'm currently attempting to use a Copy Data activity to copy a .json file over from what we treat as a "staging area" of our data over to our Datalake all within Azure Datalake Storage. The files start out in .json format but are copied over…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-04-24T14:31:33.2366667+00:00
Christopher Ackerman 11 Reputation points
answered 2024-04-25T06:42:19.9933333+00:00
Nehruji R 2,201 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Can't deploy python function with the module azure-identity or azure-storage-filedatalake

I'm trying to deploy my function that I've been working on locally, whenever I add the module "azure-storage-filedatalake" or the module "azure-identity" my function doesn't appear on the function app anymore, but when I remove this…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,336 questions
asked 2024-04-18T09:55:12.23+00:00
Ayoub Ben Yamoune 20 Reputation points
accepted 2024-04-24T08:27:54.5833333+00:00
Ayoub Ben Yamoune 20 Reputation points
2 answers

ErrorCode=UserErrorUnzipInvalidFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file 'All Weekly 18.02.2024.zip' is not a valid Zip file with Deflate compression method.,Source=Microsoft.DataTransfer.ClientLibrary,''Type

While Files copying from SFTP to ADLS using ADF Copy activity Pipeline fails when doing unzip from SFTP to ADLS.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2024-04-09T06:13:16.2066667+00:00
Osadhis Nanda 0 Reputation points
answered 2024-04-24T05:21:07.2066667+00:00
KarishmaTiwari-MSFT 18,602 Reputation points Microsoft Employee
1 answer

How to write to datalakegen2 storage using databricks in delta format when connected using SAS tocken

I converted the data from parquet form to data format. Now I want to write the data to blob storage - datalakegen2. But facing below error while writing. I useed the below command to write my data: output_path =…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,729 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,950 questions
asked 2024-04-09T23:29:12.6666667+00:00
Sai Sunny Kothuri 0 Reputation points
commented 2024-04-24T05:00:35.0166667+00:00
KarishmaTiwari-MSFT 18,602 Reputation points Microsoft Employee
1 answer

How to fix "LeaseIdMissing" or "LeaseNotPresent" Error in Synapse pipeline

Hello, In past couple of weeks we are facing issues with data lake loads. As a part of the process we delete historical data from the data lake and reload the data into the data lake through synapse pipelines. When trying to delete the historical data…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,432 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,667 questions
asked 2023-10-30T06:46:56.86+00:00
Kiran 26 Reputation points
answered 2024-04-24T04:10:25.4633333+00:00
Vijayan, Uvaneswaran 20 Reputation points
1 answer

proper cleanup of spark _temporary directory after an exception

We have a data collection app that periodically queries some data source, then writes the result to ADLS:                 df.write.mode('append').option('compression', 'gzip').parquet(rawDbPath) Every other week, the data source does some maintanence…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
asked 2024-04-19T21:54:08.9333333+00:00
Pan, John 120 Reputation points
commented 2024-04-23T06:04:56.27+00:00
Vinodh247-1375 11,301 Reputation points
3 answers

Urgent: Reuse Azure Storage Account Name After Deletion

Hello, I created and deleted a Azure Data Lake Storage Gen2 account in tenant #1 more than 14 days ago. Now I am trying to create Azure Data Lake Storage Gen2 account in tenant #2 with the same name but I keep getting below error message: The storage…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,729 questions
asked 2024-04-17T17:05:04.94+00:00
azurearchitectvs 0 Reputation points
commented 2024-04-23T05:17:08.2933333+00:00
KarishmaTiwari-MSFT 18,602 Reputation points Microsoft Employee
1 answer

Understanding the Structure of Incremental D365 FO data in Data Lake Gen2

I am a data engineer new to working with Azure and I have set up an ETL process to read incremental data out of Data Lake Gen2 storage and push to Azure SQL Database. I am using Azure Synapse Link to expose Dynamics 365 FO tables to the data lake. I'm…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,432 questions
asked 2024-04-16T12:19:26.2366667+00:00
Arbuckle, Grant 0 Reputation points
commented 2024-04-19T05:20:49.0666667+00:00
Smaran Thoomu 10,245 Reputation points Microsoft Vendor
1 answer

I am using the Student version of Microsoft azure. Can I create Data lake and Data warehouse with this version?

I am student in CIT , Coimbatore.I want to explore the concepts of Data warehouse so I choosed Microsoft azure student version.Can I create warehouse in this version if not please suggest another platforms to create effective warehouses for free.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Database for MySQL
Azure Database for MySQL
An Azure managed MySQL database service for app development and deployment.
721 questions
asked 2024-04-18T05:01:48.5133333+00:00
Harish Kumar 0 Reputation points
edited an answer 2024-04-18T07:38:57.85+00:00
Sumarigo-MSFT 43,816 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Migrate Storage Account (Gen2 Datalake) from GRS to ZRS

Trying to move a datalake from GRS to ZRS. The documentation says to convert to LRS and then request a migration. When I try and request a migration the options are not present at documented. It does not seem possible to request this any longer. Can…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,360 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,729 questions
asked 2024-02-14T08:51:13.1833333+00:00
Ian Henry 226 Reputation points
commented 2024-04-17T13:01:52.22+00:00
Guilherme Augusto Faccini 0 Reputation points