Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,355 questions
Error generated in Synapse when I query dataverse data in Spark Notebook
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 84%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERO%3C/text%3E%3C/svg%3E)
Robert O'Neill
1
Reputation point
Hey there,
I am having an issue querying the data from the Synapse Link for Dataverse in a Spark Notebook within Synapse.
I am able to run a SQL query against the data (which appears in Synapse as a Lake Database) and it returns data. See below
However when I run a query in Spark Notebook I get the following error:
AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
Traceback (most recent call last):
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 723, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/home/trusted-service-user/cluster-env/env/lib/python3.8/site-packages/py4j/java_gateway.py", line 1304, in call
return_value = get_return_value(
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
raise converted from None
pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
See Screenshot~:
The Synapse workspace has Owner and Storage Blob Data Contributor access on the storage account.
Anyone have any ideas? I'm really stuck with this one.
Azure Data Lake Storage
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,420 questions
.NET Machine learning
.NET Machine learning
.NET: Microsoft Technologies based on the .NET software framework.Machine learning: A type of artificial intelligence focused on enabling computers to use observed data to evolve new behaviors that have not been explicitly programmed.
150 questions
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee2022-05-18T22:28:42.907+00:00 Hello @Robert O'Neill ,
Thanks for the question and using MS Q&A platform.
As we understand the ask here is how to get around the error while running the notebook scripts ., please do let us know if its not accurate.Try adding yourself to “Storage Blob Data Contributor”.. as the session run on your context you should have access to the underlaying storage..
Please do let me if you have any queries.
Thanks
Himanshu
- Please don't forget to click on
or upvote 
button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how - Want a reminder to come back and check responses? Here is how to subscribe to a notification
- If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
- Please don't forget to click on
-
Robert O'Neill 1 Reputation point
2022-05-19T08:49:25.21+00:00 Thanks for the reply Himanshu.
That's correct I am trying to query the data in Spark Notebook.
I already have Storage Blob Data Contributor access.
Sign in to comment