Use parameters in Lakeflow Designer

Parameters are named values defined for the visual data prep as a whole. You can reference parameters in SQL and Python operator code, in supported operator configuration fields, and in expressions that enable or disable operators. Parameters let you run the same visual data prep against different environments, for example, a test catalog during development and a production catalog in production.

Define parameters

To manage parameters, select Parameters in the left pane. Each parameter has a value that you set when you define it. When the visual data prep runs on a schedule or as a job, you can override that value. See Override parameter values in production.

Reference a parameter from an operator

How you reference a parameter depends on where you use it:

  • SQL operator: Use named parameter marker syntax, such as :environment. See Use named parameter markers.
  • Python operator: Call dbutils.widgets.get(), such as dbutils.widgets.get("environment"). See Widgets tool (dbutils.widgets).
  • Supported operator configuration fields: In the operator's configuration pane, select Parameters in the upper-right corner. Select a field and enter an expression such as :source_table. Supported fields include:
    • Source: Table, File path, and File format.
    • Output: Catalog, Schema, Table, View, Volume, File name, and Write mode.
    • Limit: Limit.
    • Combine: Set operation, Merge strategy, Union by name, and Allow missing columns.

When you open a SQL or Python operator for editing, Lakeflow Designer shows an example of how to reference the available parameters above the source editor.

Combine parameters with fixed text

In a parameter expression for a Source or Output operator, use :parameter_name to replace an entire value. To combine parameters with fixed text, enclose the text in single quotes and use ||. For example, the following Source Table expression constructs a fully qualified table name:

:catalog || '.' || :schema || '.' || :table

Enable or disable an operator with a parameter

You can use a parameter to enable or disable any configured operator. In its Parameters pane, select Disabled and enter a Boolean expression. For example, :skip_operator disables the operator when the parameter is true. A disabled operator produces no output rows.

To enable or disable an operator manually instead, see Rename, copy, delete, and disable operators.

Override parameter values in production

At runtime, a visual data prep reads its parameters the same way whether it runs interactively or as a job, so the same file works across all of your environments without changes. Supply different values for each environment as follows:

  • When you schedule a job in the UI: Override the parameter values for each schedule. For example, schedule the same visual data prep to run every day at noon with an environment parameter set to test, and again at 2 PM with environment set to production. See Schedule as a job.
  • When you deploy with a bundle: Set the parameter values through the job's parameters, and use bundle targets to supply different values per environment. Bundle development and production targets let you deploy the same job to separate environments with environment-specific settings. See Declarative Automation Bundles deployment modes and Declarative Automation Bundles configuration.

Additional resources

  • Last updated on