Reading a Delta table from ADLS Gen2 in Python

Johan á Rogvi-Hansen 106 Reputation points
2022-09-21T09:04:34.477+00:00

Hi,

I've created a Delta table in an ADLS Gen2 and need to load the locally using Python.

I've tried using this approach unsuccessfully: https://pypi.org/project/delta-lake-reader/ with the error "Could not find transaction logs in path <> Make sure you point to the root of a delta table" though it is the root that I am pointing to.

Do you know of any other methods? Note that I tried reading the .parquet file with pd.parquet_reader without issues.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,363 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,442 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 78,986 Reputation points Microsoft Employee
    2022-09-22T06:31:37.457+00:00

    Hello @Johan á Rogvi-Hansen ,

    Thanks for the question and using MS Q&A platform.

    From the error message it clearly says as the could not find the transaction logs in the path. To resolve the issue - make sure you point to the root of a delta table" though it is the root that I am pointing to.

    There are three common causes for this error message.

    Cause 1: You start the Delta streaming job, but before the streaming job starts processing, the underlying data is deleted.
    Cause 2: You perform updates to the Delta table, but the transaction files are not updated with the latest details.
    Cause 3: You attempt multi-cluster read or update operations on the same Delta table, resulting in a cluster referring to files on a cluster that was deleted and recreated.

    For more details, refer to A file referenced in the transaction log cannot be found.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is jhow you can be part of Q&A Volunteer Moderators