Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,847 questions
Using spark.sql.execution.datasources.csv.VegasCSVFileFormat and encountering NullPointerException
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 78%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
David Beavon
976
Reputation points
I use Synapse Spark for some workloads that consume data from a CSV.
I keep encountering a java NullPointerException (NPE) when reading CSV files out of the storage account. This is in a synapse pool with default "intelligent cache" settings, and I am reading data from adls gen2 storage.
Below is a stack trace from the executor. The "WARN" severity level at the top is deceptive. It eventually causes the driver to crash.
2023-11-03 12:35:51,155 WARN
BlockManager [Executor task launch worker for task 0.0 in stage 43.0 (TID 52)]: Putting block rdd_132_0 failed due to exception java.io.IOException: NullPointerExceptionjava.lang.NullPointerException
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:121) at org.apache.spark.sql.execution.datasources.HadoopFileLinesReader.<init>(HadoopFileLinesReader.scala:65) at org.apache.spark.sql.execution.datasources.csv.TextInputCSVDataSource$.readFile(CSVDataSource.scala:97) at
>>>>>>>>> org.apache.spark.sql.execution.datasources.csv.VegasCSVFileFormat.$anonfun$buildReader$1(VegasCSVFileFormat.scala:423)
at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:150) at org.apache.spark.sql.execution.datasources.FileFormat$$anon$1.apply(FileFormat.scala:135) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile (FileScanRDD.scala:162) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:224) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:137) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:763) at org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer$$anon$1.hasNext(InMemoryRelation.scala:118) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460) at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:239) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:318) at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1482) at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1409) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1473) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1296) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:385) at org.apache.spark.rdd.RDD.iterator(RDD.scala:336) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374) at org.apache.spark.rdd.RDD.iterator(RDD.scala:338) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374) at org.apache.spark.rdd.RDD.iterator(RDD.scala:338) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374) at org.apache.spark.rdd.RDD.iterator(RDD.scala:338) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374) at org.apache.spark.rdd.RDD.iterator(RDD.scala:338) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:57) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:374) at org.apache.spark.rdd.RDD.iterator(RDD.scala:338) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750)
... I'm not really a java or scala programmer, but based on the stack it looks to me like Microsoft has replaced the default handling of CSV read operations. There is something called a "VegasCSVFileFormat" in this stack trace and I'm not sure how it got there but it is suspicious to me. The reason I distrust it is because (1) it is not found in google, and (2) this "vegas" code-word has Microsoft's name written all over it. I think it is related to their home-grown "intelligent cache" technology.
I will also send this thru CSS (Microsoft's secondary/"professional" support service) but that can be a multi-month engagement, just to diagnose. I'm hoping that someone is already aware of this buggy component, and how I might simply disable it.
The code to read this CSV into my Spark session is straightforward. I don't have anything fancy going on here, aside from the C# language bindings. This code on the driver would NOT have any impact the executors that are trying to load the CSV and failing. I'm certain the same bug would happen in a Python or Scala repro.
My CSV file is a bit on the hefty side (200 MiB) and I may one day use another format for this intermediate data, but I don't want Microsoft to force the issue right now, because of a buggy component that throw's NPE's.
The error happens in about 500 ms, on two executors of four cores each.
I am encountering this error about 5 or 10 percent of the time. It seems to be a fairly recent problem in the past couple months. It is just a theory, but I'm thinking the error message itself is misleading and could probably be enhanced. There may be an underlying root cause below the NPE that is being obscured because of poor structured error-handling . (For example, my stuff runs in a Synapse "Managed VNET" which has always seemed unreliable , and there may be socket exceptions that prevent files from being retrieved data consistently. The socket exceptions are the type of thing that no Microsoft programmer - of the "VegasCSVFileFormat" - would have planned for. Few of us would expect to encounter unreliable networking when connecting to resources in the same region that are only ~0ms away. But that type of problem has become a repeating theme on Synapse.)
Please let me know if anyone has experienced random NPE's on Synapse Spark, especially while using the "VegasCSVFileFormat". My goal is to figure out how to disable the Microsoft customizations in the reading of CSV files. I suspect the root cause of this problem wouldn't go away (eg. the underlying socket exception) but perhaps I will start getting better exception messages from the actual Apache libraries. I already have another ticket open with Microsoft about the unreliability of their "Managed VNET" but they haven't been motivated to troubleshoot or fix it yet. Perhaps if I can show them yet another repro then that would help.
{count} votes
-
ShaikMaheer-MSFT 38,441 Reputation points Microsoft Employee2023-11-07T06:19:05.7633333+00:00 Hi David Beavon,
Thank you for posting query in Microsoft Q&A Platform.
Could you please try below and see if that helps.Disable the intelligent cache: You can disable the intelligent cache by setting the "spark.databricks.io.cache.enabled" configuration parameter to "false". This will prevent the "VegasCSVFileFormat" component from being used and may resolve the issue.
If above not helps, then I would suggest to have support case for this issue for deeper investigation. Please let me know how it goes. Thank you.
-
David Beavon 976 Reputation points
2023-11-07T14:42:51.15+00:00 How did you learn this?
The google searches are not turning up anything regarding "VegasCSVFileFormat" and the conclusion I made was that it was a "secret sauce" which is specific to Microsoft Synapse (rather than Databricks or OSS). Can you please tell me how to determine if these configuration settings will be take effect (to change the behavior of certain spark.read formats)? I don't want to start enabling and disabling random configuration parameters without having reasonable confidence that they will be respected in the Spark platform. Life is too short for that degree of trial-and-error.
FYI, I already have a ticket open with CSS and they have been pointing me to a different configuration. You said spark.databricks.io.cache.enabled. But they said spark.synapse.vegas.useCache.
Either way this discussion seems to focus on a configuration that may or may not act as a workaround. The workaround does NOT actually solve any of the problems in the default behavior of Synapse. There are a couple underlying problems in the default behavior, from what I can see: (1) whatever the underlying bug is that causes the ADLS IO to intermittently fail, and (2) the terrible presentation of this failure to the user. What in the world is this message supposed to mean to anyone - "(java.io.IOException: NullPointerExceptionjava.lang.NullPointerException)"???
So far the CSS folks have also agreed with the strategy of trying to find a workaround too - without directly confronting the underlying problems in the default behavior. It is always a bit hard to ask CSS to engage with their PG. I have several support cases with Azure Synapse Spark, and they are VERY challenging to work on. Even my Synapse outages aren't taken seriously, and we are made to wait for weeks to receive an RCA. I think it is becoming pretty clear that Microsoft is already losing interest in this particular Spark platform - probably because their "strategic path" is to replaced this one with a SaaS Spark offering in Power BI. Of course they are many months/years away from telling the impacted customers about the change in direction. I am really starting to regret my decision to migrate to Synapse from Databricks for my Spark workloads.
Since you will probably ask, the CSS case is TrackingID #2311030040005903. ... That said, I really wish you would focus on the other support case related to the general-purpose networking failures that we encounter in the Microsoft Managed VNET. I need all the help you can give on that side of things. The teams I'm working with seem to be deliberately obstructing an investigation, and won't share much in the way of the raw logs, or their initial findings. It has been almost a month since experiencing a twenty minute outage in production.
Sign in to comment
Sign in to answer