Writing parquet file throws…An HTTP header that's mandatory for this request is not specified
I have two ADLSv2 storage accounts, both are hierarchical namespace enabled. In my Python Notebook, I'm reading a CSV file from one storage account and writing as parquet file in another storage, after some enrichment. I am getting below error when…
Where to store delta-files
Hey guys! I have been wondering about the below question for a while and I hope you can help me get a good night sleep again. Question: When you are working in Azure DataBricks to do transformations, you can save the results using a delta format.…
Error Setting Access Rights on ADL (Gen1) User Folders During HDInsight Cluster Creation
Hello All The issue occurs when the cluster is created. One of the last operations that Ambari performs are 'Post user creation hook for 1 users'. This fail with the attached errors. Here is some more information: It's an ESP cluster -- we…
Can't connect to our ADLS gen2 in Powerbi Online (scheduled refresh)
Hi all, We have recently set up a ADLS gen 2 and we want to update our reports automatically. In the powerBI desktop version everything works, but online I get the following error: Failed to update data source credentials: The credentials provided…
Mounting entire ADLS on azure databricks
Hi, I want to mount entire ADLS storage account on databricks. I've checked in documents I can mount a single filesystem at a time but I want to mount entire ADLS on databricks. I've around 70 containers in my ADLS and I want to mount all of them…
Adls Availability Alerts
Hi Team, I have set alerts on my adls gen2 such as if availability is less than 80 percent it should send alert notification. My question is this availability is Microsoft responsibility, right ? how to find out root cause of why this availability…
Copy Different type of file from Gen1 Azure lake to Azure Gen2 lake with attribute( like last updated)
I need to migrate all my data from Azur data lake Gen1 to Lake Gen2. In my lake we have different types of file mixed (.txt, .zip,.json and many other). We want to move them as-it-is to GEN2 lake. Along with that we also want to maintain last updated…
How to delete specific data from files stored in Azure Data Lake
I'm migrating data sets to be stored in ADLS. After certain period I want to delete data within this data set that is more than 3yrs old. How can I do that? I do not want to delete the file, but specific data within the files.
Data Factory V2 Storage Account not Refreshing - Because it thinks its blob Storage
I have created a Storage Account set to be Azure Data Lake V2 by setting the hierarchical namepace. And I used Azure Storage Explorer to add some files into the blob Container (Which is an Azure datalake gen2) I'm now creating a Data Lake V2 Linked…
I'm trying to source data in a comma-delimited file from an Azure Gen2 Storage account using Flexible File Source task. The connection succeeds but Preview of data is blank.
I'm trying to pull data from ADLS Gen2 using the SSIS Flexible File Source task. I'm getting blank results when trying to view the data in the Columns tab of the dialog.
Error occurs at source transformation when trying to source csv file from folder in data lake storage gen2 while debugging the pipeline
While using mapping data flow activity, In source transformation I am trying to read a csv file from a folder in Azure data lake gen2. I am able to preview the data in "Data Preview" in source transformation but when trying to debug the…
Write Access for Azure Data Lake Gen 2 Container
Hi, I have an Azure Data Lake Gen 2 Storage Account. I have created multiple containers inside it say A , B & C. Now , I want to control access for files & folders for respective containers. I have tried giving the following access - …
Can't access txt file with json format from Data Lake Gen2 inside a Data Flow
I have stored a txt file with json format (coming from a MongoDB) in my Data Lake Storage Gen2 and inside my Data Flow I would like to access that file, convert it into a column structure, transform the data and shift it to my Azure SQL DB. Currently I…
Common data service is not able to connect to Azure data lake store Gen 2 when "Selected Networks" is configured in firewall.
Issue: Common data service is not able to connect to Azure data lake store Gen 2 when "Selected Networks" is configured in firewall. Description: Both environment and storage account are in west europe region. However, the reason connection…
Data labeling and classification solution in ADLS gen2
Hi, I need to classify and the label with the classification the data stored in Data Lake Storage e.g. assign Sensitive label on some folder/file and then have a way to present in some report, provide retention based on the classification etc. One…
USQL Datalake Question
Is there anyway to backup the partition files of usql table? Or the only way is to import the data to csv file.
Import incremental data from Azure Data Lake
Good afternoon, I have the following scenario !! I have a table in SQL that always receives changes daily, and I need to send this data to Azure data lake So, today I take this data and send it in the datalake file: Historicos.csv …
Unable to load data into Azure Synapse from ADLS Gen 2 .csv file
Hi @ChiragMishraMSFT-1092 Upon trying to load the data from ADLS(csv files) into Azure DW, am encountering a error in loading the data in DW. Here is the error message: "message": "Failure happened on 'Sink' side.…
Express route and Azure Data Lake gen 2
Hi, I am doing some testing around accessing Azure Data Lake Gen 2 from an On Premise hadhoop I need to access the service via Express Route What is the recommended setup if anyone has been through a configuration? thanks
How do I create an exploratory zone on top of my Azure Data Lake
Hi, My sincere apologies if this question would have been already answered. However, I searched hard but couldn't find the relevant information. Here lies my question: I have a Data Lake setup in Azure already. Further on, I have a requirement…