What is a dynamic value reference?

Article
10/04/2024

Dynamic value references describe a collection of variables available when configuring jobs and tasks. Use dynamic value references to configure conditional statements for tasks or to pass information as parameters or arguments.

Dynamic value references include information such as:

Configured values for the job, including the job name, task names, and trigger type.
Generated metadata about the job, including the job ID, the run ID, and the job run start time.
Information about how many repair attempts a job has made or retries a task has run.
The result state for a specified task.
Values configured using job parameters, task parameters, or set using task values.

Use dynamic value references

Use dynamic value references when configuring jobs or tasks. You cannot directly reference dynamic value references from assets configured using tasks like notebooks, queries, or JARs. Dynamic value references must be defined using parameters or fields that pass context into tasks.

Dynamic value references use double curly braces ({{ }}). When a job or task runs, a string literal replaces the dynamic value reference. For example, if you configure the following key-value pair as a task parameter:

{"job_run_id": "job_{{job.run_id}}"}

If your run ID is 550315892394120, the value for job_run_id evaluates to job_550315892394120.

Note

Contents of the double curly braces are not evaluated as expressions. You cannot run operations or functions in double-curly braces.

User-provided value identifiers support alphanumeric and underscore characters. Escape keys that contain special characters by surrounding the identifier with backticks (` `).

Syntax errors, including non-existent dynamic reference values and missing braces, are silently ignored and are treated as literal strings. An error message is displayed if you provide an not valid reference belonging to a known namespace, for example, {{job.notebook_url}}.

Use dynamic value references in the jobs UI

Fields that accept dynamic value references provide a shortcut to insert available dynamic value references. Click { } to see this list and insert it into the provided field.

Note

The UI does not auto-complete the keys to reference task values.

Many fields that accept dynamic value references require additional formatting to use them correctly. See Configure task parameters.

Use dynamic value references in a job JSON

Use {{ }} syntax to use dynamic values in job JSON definitions used by the Databricks CLI and REST API.

Job and task parameters have different syntax, and task parameter syntax varies by task type.

The following example shows the partial JSON syntax to configure job parameters using dynamic value references:

{
  "parameters": [
    {
      "name": "my_job_id",
      "default": "{{job.id}}"
    },
    {
      "name": "run_date",
      "default": "{{job.start_time.iso_date}}"
    }
  ]
}

The following example shows the partial JSON syntax to configure notebook task parameters using a dynamic value reference:

{
  "notebook_task": {
    "base_parameters": {
      "workspace_id": "workspace_{{workspace.id}}",
      "file_arrival_location": "{{job.trigger.file_arrival.location}}"
    }
  }
}

Review parameters for a job run

After a task completes, you can see resolved parameter values under Parameters on the run details page. See View job run details.

Supported value references

The following dynamic value references are supported:

Reference	Description
`{{job.id}}`	The unique identifier assigned to the job.
`{{job.name}}`	The name of the job at the time of the job run.
`{{job.run_id}}`	The unique identifier assigned to the job run.
`{{job.repair_count}}`	The number of repair attempts on the current job run.
`{{job.start_time.<argument>}}`	A value based on the time (in UTC timezone) that the job run started. The return value is based on the `argument` option. See Options for date and time values.
`{{job.parameters.<name>}}`	The value of the job-level parameter with the key `<name>`.
`{{job.trigger.type}}`	The trigger type of the job run. The possible values are `periodic`, `one_time`, `run_job_task`, `file_arrival`, `continuous`, and `table`.
`{{job.trigger.file_arrival.location}}`	If a file arrival trigger is configured for this job, the value of the storage location.
`{{job.trigger.time.<argument>}}`	A value based on the time (in UTC timezone) that the job run was triggered, rounded down to the closest minute for jobs with a cron schedule. The return value is based on the `argument` option. See Options for date and time values.
`{{task.name}}`	The name of the current task.
`{{task.run_id}}`	The unique identifier of the current task run.
`{{task.execution_count}}`	The number of times the current task was run (including retries and repairs).
`{{task.notebook_path}}`	The notebook path of the current notebook task.
`{{tasks.<task_name>.run_id}}`	The unique identifier assigned to the task run for `<task_name>`.
`{{tasks.<task_name>.result_state}}`	The result state of task `<task_name>`. The possible values are `success`, `failed`, `excluded`, `canceled`, `evicted`, `timedout`, `upstream_canceled`, `upstream_evicted`, and `upstream_failed`.
`{{tasks.<task_name>.error_code}}`	The error code for task `<task_name>` if an error occurred running the task. Examples of possible values are `RunExecutionError`, `ResourceNotFound`, and `UnauthorizedError`. For successful tasks, this evaluates to an empty string.
`{{tasks.<task_name>.execution_count}}`	The number of times the task `<task_name>` was run (including retries and repairs).
`{{tasks.<task_name>.notebook_path}}`	The path to the notebook for the notebook task `<task_name>`.
`{{tasks.<task_name>.values.<value_name>}}`	The task value with the key `<value_name>` that was set by task `<task_name>`.
`{{workspace.id}}`	The unique identifier assigned to the workspace.
`{{workspace.url}}`	The URL of the workspace.

You can set these references with any task. See Configure task parameters.

You can also pass parameters between tasks in a job with task values. See Use task values to pass information between tasks.

Options for date and time values

Use the following arguments to specify the return value from time-based parameter variables. All return values are based on a timestamp in the UTC timezone.

Argument	Description
`iso_weekday`	Returns a digit from 1 to 7, representing the day of the week of the timestamp.
`is_weekday`	Returns `true` if the timestamp is on a weekday.
`iso_date`	Returns the date in ISO format.
`iso_datetime`	Returns the date and time in ISO format.
`year`	Returns the year part of the timestamp.
`month`	Returns the month part of the timestamp.
`day`	Returns the day part of the timestamp.
`hour`	Returns the hour part of the timestamp.
`minute`	Returns the minute part of the timestamp.
`second`	Returns the second part of the timestamp.
`timestamp_ms`	Returns the timestamp in milliseconds.

Deprecated dynamic value references

The following dynamic value references are deprecated. The recommended replacement reference is included in the description of each variable.

Variable	Description
`{{job_id}}`	The unique identifier assigned to a job. Use `job.id` instead.
`{{run_id}}`	The unique identifier assigned to a task run. Use `task.run_id` instead.
`{{start_date}}`	The date a task run started. The format is yyyy-MM-dd in the UTC timezone. Use `job.start_time.<argument>` instead.
`{{start_time}}`	The timestamp of the run’s start of execution after the cluster is created and ready. The format is milliseconds since UNIX epoch in the UTC timezone, as returned by `System.currentTimeMillis()`. Use `job.start_time.<format>` instead.
`{{task_retry_count}}`	The number of retries that have been attempted to run a task if the first attempt fails. The value is 0 for the first attempt and increments with each retry. Use `task.execution_count` instead.
`{{parent_run_id}}`	The unique identifier assigned to the run of a job with multiple tasks. Use `job.run_id` instead.
`{{task_key}}`	The unique name assigned to a task that’s part of a job with multiple tasks. Use `task.name` instead.

Share via