Use parameter for db name in Spark SQL notebook

Gerald Reif 1 Reputation point
2022-04-19T15:43:48.147+00:00

In a pipeline have an execute notebook action. In the pipeline action I hand over a base parameter of type String to the notebook. When I debug I see that a cell with the following code is injected to hand over the parameter.

-- This cell is generated from runtime parameters. Learn more: https://go.microsoft.com/fwlink/?linkid=2161015
SET db_name = 'my_db_name' ;

Later in the notebook I want to use the parameter to define the database where the action should be performed.

create table ${db_name}.mytable ...;

select * from ${db_name}.mytable;

When I run this, I get the following error message. For example the select statement:

select * from ${db_name}.mytable;

Error: 
mismatched input ''my_db_name'' expecting {
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,573 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA-MSFT 83,306 Reputation points Microsoft Employee
    2022-04-20T07:53:04.457+00:00

    Hello @Gerald Reif ,

    (UPDATE: 21/4/2022): As per the repro, when I tried the same using Apache Spark SQL, got the same error message. I'm working with the internal team to identify how to pass the base parameters in Apache Spark SQL notebooks. Meanwhile, you can use the below workaround.

    Workaround: I had tried to use parameter for db_name using Apache Pyspark and worked as excepted:

    Notebook:

    194960-image.png

    Pipeline Activity with base parameter:

    195041-image.png

    ** Pipeline worked as excepted:**

    194990-image.png

    ------------------------------------------------

    Thanks for the question and using MS Q&A platform.

    Error: When I tried to use parameter for db name as SET db_name = 'default';, with single quotes it throws the same error as shown above.

    194612-image.png

    **Success: ** When I tried to use parameter for db name as SET db_name = default; without single quotes it works as excepted.

    194603-image.png

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.