How to read append blob in (Azure Blob or ADLS) from Synapse or Databricks

Bloody Data 0 Reputation points
2023-05-13T16:01:41.36+00:00

Hi All I am trying to read a telemetry log , which is stored currently in blob storage without the hierarchical namespace enabled , when I try to read , I am getting the following error.

All my files in storage account are currently saved as append blob, when I try to read the append blob in data lake gen 2 with hierarchical namespace enabled also I am getting similar kind of error.

Please help , thanks in advance.

Exception: Incorrect Blob type, please use the correct Blob type to access a blob on the server. Expected BLOCK_BLOB, actual APPEND_BLOB.
Azure Storage
Azure Storage
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,537 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,378 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,526 questions
{count} votes

3 answers

Sort by: Most helpful
  1. Vinodh247 34,666 Reputation points MVP Volunteer Moderator
    2023-05-13T16:51:38.9+00:00

    Looks like Read append blob is not supported in azure Databricks and synapse, pls see the link below where this has been answered and accepted earlier.

    
    

  2. Sumarigo-MSFT 47,471 Reputation points Microsoft Employee Moderator
    2023-05-15T06:42:53.0733333+00:00

    @Bloody Data Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

    An append blob is composed of blocks and is optimized for append operations. When you modify an append blob, blocks are added to the end of the blob only, via the Append Block operation. Updating or deleting of existing blocks is not supported. Unlike a block blob, an append blob does not expose its block IDs.

    https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs

    There is similar thread discussion, please refer to the suggestion mentioned here: How to Read Append Blobs as DataFrames in Azure DataBricks

    You can also use the Azure Storage explorer tool : https://azure.microsoft.com/en-us/products/storage/storage-explorer

    Please let us know if you have any further queries. I’m happy to assist you further.


    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments No comments

  3. Paul Grardel 0 Reputation points
    2023-07-06T13:42:31.1433333+00:00

    Encountered exactly the same issue using mounts in Databricks.

    However it seems it uses a different driver, I managed to read append blobs with a Databricks notebook using abfss.

    You can create an external table with an abfss location and then load it in any dataframe.

    Simply follow the basic guide from Databricks to access Blob storage : https://docs.databricks.com/storage/azure-storage.html#language-Account%C2%A0key

    Hope this helps

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.