Azure Databricks error org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND]

Question

Azure Databricks error org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND]

Jess N 0 Microsoft Vendor

Hi All and thanks for your time:

A Databricks notebook that previously ran w/o issues (for the same data) has the following error:
org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND] Failed to find data source: sstream

Update: This seems to be a permissions issue. A user w/ elevated permissions executes the same code flawlessly.

PRADEEPCHEEKATLA-MSFT 59,141 Reputation points Microsoft Employee

2023-05-18T04:03:41.5166667+00:00

@Jess N - Thanks for the question and using MS Q&A platform.

As I see there has been update on the question: (Update: This seems to be a permissions issue. A user w/ elevated permissions executes the same code flawlessly.) is this issue resolved? OR Are you looking for any help?
Jess N 0 Reputation points Microsoft Vendor

2023-05-18T16:40:45.75+00:00

It is not fixed yet. I am on a registered device, logged in with only my work account, connected through work vpn, logged into azure, and when I login to databricks it appears similar to a personal account. Or an account with less privileges. My current privileges are the same as the datbrick's "owner" w/ the exception of the full permissions. We have the same contributor role and group role. I think the next step is to have the owner add another user, and give the same permissions, then me and the new user try running a notebook. I think that will fail and then we can get some more time w/ the owner and go through current permissions. But I am open to suggestions.

PRADEEPCHEEKATLA-MSFT 59,141 Reputation points Microsoft Employee

2023-05-18T04:03:41.5166667+00:00

@Jess N - Thanks for the question and using MS Q&A platform.

As I see there has been update on the question: (Update: This seems to be a permissions issue. A user w/ elevated permissions executes the same code flawlessly.) is this issue resolved? OR Are you looking for any help?
Jess N 0 Reputation points Microsoft Vendor

2023-05-18T16:40:45.75+00:00

It is not fixed yet. I am on a registered device, logged in with only my work account, connected through work vpn, logged into azure, and when I login to databricks it appears similar to a personal account. Or an account with less privileges. My current privileges are the same as the datbrick's "owner" w/ the exception of the full permissions. We have the same contributor role and group role. I think the next step is to have the owner add another user, and give the same permissions, then me and the new user try running a notebook. I think that will fail and then we can get some more time w/ the owner and go through current permissions. But I am open to suggestions.

Answer 1

my role allows for "Starts a managed cluster". While looking at permissions together my databricks session refreshed. I could then see my coworker's notebooks. After which I refreshed the page, opened a test notebook, started our managed cluster we use for the group, ran the notebook, and it ran and finished every cell as designed.

While we did not make any changes- maybe.... there were changes that occurred but which we were not aware of? What if our ip from vpn didn't get added to a databricks backend white list (if that exists)- I suspect I'd have the issue we started off with. That's just a hypothetical. If it happens again I will check my ip, disconnect and reconnect until i get a new one. Possibly even change work wifi/network if needed. And then update w/ results.

Regardless, everything works now. So problem solved and case closed :)

PRADEEPCHEEKATLA-MSFT 59,141 Reputation points Microsoft Employee

2023-05-19T03:26:26.85+00:00

@Jess N - Glad to know that your issue has been resolved. And thanks for sharing the solution, which might be beneficial to other community members reading this thread.

Azure Databricks error org.apache.spark.SparkClassNotFoundException: [DATA_SOURCE_NOT_FOUND]

1 answer

Activity