Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,559 questions
I have an error when calling Synapse Analytics laptop script via azure data factory to concurrently read a parquet table, the parquet file is stored in azure data lake store gen2, some report errors and some don't, there is no problem with manual executio
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 54%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENI%3C/text%3E%3C/svg%3E)
Niu, Ivan
20
Reputation points
Hello.
Calling Synapse Analytics laptop script via azure data factory concurrently reads parquet table with error, some tables don't report error, some tables report error, parquet file is stored in azure data lake store gen2, logs show that parquet file can't exist in cache/has been read to the end of the file but is still being read.
Strangely enough, the manual execution is fine and the parquet is fine and can be read normally.
When reading 10 parquet tables at a time, both concurrent and non-concurrent reads report errors.
Is there any way to solve this problem? This issue hasn't come up before, but has been around since the end of September.
Azure Data Lake Storage
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,373 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,623 questions
{count} votes
-
Amira Bedhiafi 33,071 Reputation points Volunteer Moderator2023-10-21T20:37:51.4166667+00:00 I think the main issue appears to be related to caching when trying to fetch the footer for a Parquet file. The relevant lines from your logs are:
Failed to fetch footer for file abfs:...
java.util.concurrent.ExecutionException: java.io.IOException ... at org.apache.spark.sql.execution.datasources.parquet.ParquetMetadataCacheReader$.getFooterFromCache(ParquetMetadataCacheReader.scala:114) ...there's a problem fetching metadata in the footer of the Parquet file from the cache.
-
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator2023-10-23T06:17:30.29+00:00 Hi Niu, Ivan,
Thank you for posting query in Microsoft Q&A Platform.
kindly help with below details.
- Are you running Synapse notebook here? If yes, could you please share high level script of where exactly you are facing issue.
- Are you running synapse notebook from ADF pipeline? using synapse notebook activity?
- What you mean, when you say parquet table? parquet if a file format. There is no parquet tables. Kindly elaborate more.
- Kindly share the screenshot of your ADF activity details and concurrent settings details.
However, between from the error message, it seems, it might be due to corrupted parquet file or due to cache issues. could you please try below and see if that helps.
- Increase Spark Pool configurations which are running your notebook from ADF.
- Try to reduce the concurrent reads value and see if that helps.
- Try to recreate the parquet file from source. This may help to have good parquet file without any corruption.
Hope this helps. Please let me know how it goes. Thank you.
-
Niu, Ivan 20 Reputation points
2023-10-24T00:41:47.85+00:00 Thank you both very much for your feedback.
I've tried reducing the amount of concurrency or not using the concurrency feature before, but neither solved the problem.
Another attempt was as follows
spark.conf.set('spark.sql.parquet.cacheMetadata','false') to disable metadata caching for Parquet files, it worked on the first day after executing the job, but on the second day the original problem reappeared and the command seems to be invalid again for the currently started session.
I am referring to external tables of type parquet
-
ShaikMaheer-MSFT 38,546 Reputation points Microsoft Employee Moderator
2023-10-24T13:14:44.9366667+00:00 Hi Niu, Ivan, I am doubting more about the parquet file here. It seems the parquet is corrupted and hence ending with footer error. Are we sure our parquet files are good? Could you please try to create
integrated datasetfor the parquet file and see if you are able to preview data from the specific parquet file correctly? -
Niu, Ivan 20 Reputation points
2023-10-25T16:27:14.7066667+00:00 I tested it and there is no problem, I also read the parquet file directly through the code and it can be read.
I also tested removing concurrency by reading them one by one in a for loop, and the same problem occurs.
Of course, I also tested not letting the code read the parquet table and reading the file directly and still had the same problem.
I also configured spark.conf.set('spark.sql.parquet.cacheMetadata','false') in my laptop's pyspark script to disable metadata caching of parquet files. Strangely enough the first execution via the trigger call worked, but the second and third times still have the same problem and are still fetching metadata from the cache.
Is there any other command that can be used in the currently started session to disable reading from the cache so that data can be read directly from the file?
Sign in to comment
Sign in to answer