Spark pool - tabula.read_pdf error - No such file or directory

Question

Hi Team,

I am trying to fetch tabular data from pdf using the tabula in Spark Pool. The pdf file is kept in a datalake and I have mounted the same. The mounted path is provided as input to the read_pdf function. But, I am getting the error - "No such file or directory". The thing is, its not working only in Azure Synapse Spark Pool. The same code works fine in azure databricks workspace. I have tried with giving the abfs url (Data lake url) instead of mounted path and still the same error.

There is no problem with the path here. The same path works if used in any other different function (Read the same pdf as binary file) as below: -

To sum it up, I have couple of questions as below: -

What is the issue with tabula.read_pdf?

Is there a way to convert the binary data (In second screenshot) to human readable tabular data?

Thanks in advance.

Regards,

Nagesh CL

Answer 1

If you're on a Unix-like system, you can use the ls -l command to check file permissions.

You can use this code to verify the path :

import os
if os.path.exists(file_path):
    # Open and read the file
else:
    print(f"{file_path} does not exist.")

The binary content you have represents the raw content of the PDF file. To convert it into a human-readable tabular format directly in Spark without using tabula, you'd need a library or method that can parse PDF content in a distributed manner, which can be challenging.

However, you can still work around this. One way is by using Azure Databricks (since you mentioned it works there). Read the PDF using tabula in Databricks, then convert the resultant table to a format like Parquet or CSV and save it to the data lake. Afterward, you can easily read this Parquet/CSV file in Azure Synapse Spark Pool.