trying to test out Azure Databricks with $200 "free trial" -- but it seems that I cannot spin up any VMs?

Question

I have found this question asked and answered last year and the year before, but cannot tell which information might be current and correct. The only reason I created an Azure account using the "free trial" was to try out Azure Databricks. But when I tried to start doing the exercise described in
https://microsoftlearning.github.io/dp-090-databricks-ml/Instructions/Labs/01a-introduction-to-azure-databricks.html

it said that I could not spin up the cluster -- looking more carefully, it seemed like none of the VMs listed were available to my subscription.

Can you point me to an Azure Databricks tutorial that CAN be followed using the free trial / $200 in credits?

Answer 1

Hello @TeneriffeLLC ,

Thanks for the question and using MS Q&A platform.

You can use Azure Free Trial subscription to create a Single node cluster which will have one Driver node with 4 cores.

A Single Node cluster is a cluster consisting of a Spark driver and no Spark workers. Such clusters support Spark jobs and all Spark data sources, including Delta Lake. In contrast, Standard clusters require at least one Spark worker to run Spark jobs.

Single Node clusters are helpful in the following situations:

• Running single node machine learning workloads that need Spark to load and save data
• Lightweight exploratory data analysis (EDA)

Reference: Azure Databricks - Single Node clusters

To use multi-node cluster in Azure Databricks, you need to have “Pay-As-You-Go” subscription.

Why can't I use Azure Free Trial subscription?

Azure Free Trial has a limit of 4 cores, and you cannot create Azure Databricks cluster using a Free Trial Subscription because to create a spark cluster which requires more than 4 cores i.e (Minimum 4 cores for Driver type and 4 cores for Worker type).

For more details, refer to Quickstart: Run a Spark job on Azure Databricks Workspace using the Azure portal

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Thank you @PRADEEPCHEEKATLA-MSFT

The instructions I was following did say to choose the Single Node cluster, and that is what I did. Although the runtime I chose was 10.1 ML. I took this partial screenshot yesterday when I was trying to do this.

Then I got called away and had to work on other things for a while and when I came back to it today I just thought I had to "restart" my cluster, but that failed and then in looking at the Error Log I saw that the cluster had failed to start yesterday as well. This is what the error message said:

Time
2021-12-14 13:20:07 PST
Message
Cluster terminated.Reason:Cloud provider launch failure
A cloud provider error was encountered while launching worker nodes. See the Databricks guide for more information.

Azure error code: SkuNotAvailable

Azure error message: The requested size for resource

Then I tried to understand what resources / types of VMs were available to me, so I used this command line:
az vm list-skus --location westus2 --all --output table

and looked through the outputs. Sure enough the Standard_DS3_v2 VM that came up as the Node type was not available. In fact, none of the node types that were in the pull-down list for the Azure Databricks New Cluster dialog appeared to be available to me. If I waited a little bit, in fact, each one would get a little grey triangle warning sign next to it to show that it was not available to me.

It would be great if someone at MSFT would create a dummy account with a "free trial" and actually try to walk through the Databricks tutorial(s) to see if they still work. Maybe something has changed.