Serve Feature Views using Declarative Automation Bundles

Important

This feature is in Private Preview. To try it, reach out to your Azure Databricks contact.

Declarative Automation Bundles (formerly known as Databricks Asset Bundles) enable you to programmatically validate, deploy, and run Azure Databricks resources such as Feature Views and the endpoints that serve them. See What are Declarative Automation Bundles?.

This tutorial builds a bundle that declares two streaming Feature Views, materializes them to an online store, and serves them through a FeatureSpec and a Feature Serving endpoint. You then query the endpoint, change which features it serves, and tear everything down.

The Feature Views and their materialization follow Manage Feature Views using Declarative Automation Bundles, which explains them in detail. To create a FeatureSpec and an endpoint with the Python API instead, see Serve Feature Views.

Each step shows the configuration in both YAML and Python. Pick a language and the page follows your choice.

Requirements

To complete this tutorial, you need the following:

Serving resources in a bundle

A bundle serves Feature Views through two resource types:

  • FeatureSpec: a Unity Catalog function that lists the features to serve. It holds no feature values. An endpoint that serves it reads them from the online tables that the Feature Views are materialized to. See Serve features with a FeatureSpec, and for the configuration reference, feature_spec.
  • Feature Serving endpoint: a model serving endpoint that serves a FeatureSpec instead of a model. A bundle declares it as a model_serving_endpoints resource. For the configuration reference, see model_serving_endpoint.

Serve Feature Views from a bundle

The following steps build and deploy the bundle, query the endpoint, change what it serves, and clean up.

Set up authentication

First, set up authentication between Databricks CLI on your development machine and your Azure Databricks workspace. This page assumes that you want to use OAuth user-to-machine (U2M) authentication and a corresponding Azure Databricks configuration profile named DEFAULT for authentication.

Note

U2M authentication is appropriate for trying out these steps in real time. For fully automated workflows, Databricks recommends that you use OAuth machine-to-machine (M2M) authentication instead. See the M2M authentication setup instructions in Authorize service principal access to Azure Databricks with OAuth.

  1. Use the Databricks CLI to initiate OAuth token management locally by running the following command for each target workspace.

    In the following command, replace <workspace-url> with your Azure Databricks per-workspace URL, for example https://adb-1234567890123456.7.azuredatabricks.net.

    databricks auth login --host <workspace-url>
    
  2. The Databricks CLI prompts you to save the information that you entered as an Azure Databricks configuration profile. Press Enter to accept the suggested profile name, or enter the name of a new or existing profile. Any existing profile with the same name is overwritten with the information that you entered. You can use profiles to quickly switch your authentication context across multiple workspaces.

    To get a list of any existing profiles, in a separate terminal or command prompt, use the Databricks CLI to run the command databricks auth profiles. To view a specific profile's existing settings, run the command databricks auth env --profile <profile-name>.

  3. In your web browser, complete the on-screen instructions to log in to your Azure Databricks workspace.

  4. To view a profile's current OAuth token value and the token's upcoming expiration timestamp, run one of the following commands:

    • databricks auth token --host <workspace-url>
    • databricks auth token -p <profile-name>
    • databricks auth token --host <workspace-url> -p <profile-name>

    If you have multiple profiles with the same --host value, you might need to specify the --host and -p options together to help the Databricks CLI find the correct matching OAuth token information.

Create the bundle

Create an empty directory for the bundle, with a databricks.yml file at its root and a resources directory beside it.

YAML

Each of the following steps adds one file under resources/:

databricks.yml
resources/
  streaming_features.yml
  feature_spec.yml
  serving_endpoint.yml

databricks.yml holds the bundle name, the include entry that loads every file under resources/, and the variables that the resources reference:

bundle:
  name: feature-serving-quickstart

include:
  - resources/*.yml

variables:
  catalog:
    description: The catalog that the Feature Views and the FeatureSpec are written to.
    default: main
  schema:
    description: The schema that the Feature Views and the FeatureSpec are written to.
    default: features_demo
  stream:
    description: The three-level name of a Stream registered in Unity Catalog.
    default: main.features_demo.events
  online_store:
    description: The online store that serves the features.
    default: my_online_store
  endpoint_name:
    description: The name of the Feature Serving endpoint.
    default: event-features

Python

Resources declared in Python are found only through the python mapping in databricks.yml. Without it the bundle validates and deploys successfully but contains no resources, so add it now:

bundle:
  name: feature-serving-quickstart

python:
  venv_path: .venv
  resources:
    - 'resources:load_resources'

variables:
  catalog:
    description: The catalog that the Feature Views and the FeatureSpec are written to.
    default: main
  schema:
    description: The schema that the Feature Views and the FeatureSpec are written to.
    default: features_demo
  stream:
    description: The three-level name of a Stream registered in Unity Catalog.
    default: main.features_demo.events
  online_store:
    description: The online store that serves the features.
    default: my_online_store
  endpoint_name:
    description: The name of the Feature Serving endpoint.
    default: event-features

Create the project and its virtual environment, and add the package:

uv init --bare
uv venv .venv
uv add databricks-bundles

Create resources/__init__.py with the function that databricks.yml names:

from databricks.bundles.core import Bundle, Resources


def load_resources(bundle: Bundle) -> Resources:
    resources = Resources()
    # The following steps add resources here.
    return resources

Declare the Feature Views and their materialization

Both Feature Views read the same Stream and aggregate per key over a trailing 90 minutes. events_90m counts events, and amount_sum_90m sums value.amount. One materialization set writes both to the online store in streaming mode. For what each configuration key means, see Step 4 of the Feature Views tutorial.

A streaming Feature View addresses an input column as a path into the stream record, so substitute the names your Stream's schema uses wherever the examples show value.*.

YAML

Create resources/streaming_features.yml:

resources:
  features:
    events_90m:
      full_name: ${var.catalog}.${var.schema}.events_90m
      description: Events per key over a trailing 90 minutes
      source:
        stream_source:
          full_name: ${var.stream}
      function:
        aggregation_function:
          count_function:
            input: value.amount
          time_window:
            rolling:
              window_duration: 5400s
      entities:
        - name: value.key
      timeseries_column:
        name: value.event_timestamp

    amount_sum_90m:
      full_name: ${var.catalog}.${var.schema}.amount_sum_90m
      description: Sum of amount per key over a trailing 90 minutes
      source:
        stream_source:
          full_name: ${var.stream}
      function:
        aggregation_function:
          sum:
            input: value.amount
          time_window:
            rolling:
              window_duration: 5400s
      entities:
        - name: value.key
      timeseries_column:
        name: value.event_timestamp

  materialization_sets:
    serving_online:
      materialized_features:
        - feature_name: ${resources.features.events_90m.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: serving_online
          streaming_mode:
            mode: STREAMING_MODE_TYPE_RTM
        - feature_name: ${resources.features.amount_sum_90m.full_name}
          online_store_config:
            online_store_name: ${var.online_store}
            catalog_name: ${var.catalog}
            schema_name: ${var.schema}
            table_name_prefix: serving_online
          streaming_mode:
            mode: STREAMING_MODE_TYPE_RTM

Python

Add the imports to resources/__init__.py:

from databricks.bundles.features import (
    AggregationFunction,
    CountFunction,
    DataSource,
    EntityColumn,
    Feature,
    Function,
    RollingWindow,
    StreamSource,
    SumFunction,
    TimeseriesColumn,
    TimeWindow,
)
from databricks.bundles.materialization_sets import (
    MaterializationSet,
    MaterializedFeature,
    OnlineStoreConfig,
    StreamingMode,
    StreamingModeStreamingModeType,
)

Then add this to the body of load_resources:

resources.add_feature(
    'events_90m',
    Feature(
        full_name='${var.catalog}.${var.schema}.events_90m',
        description='Events per key over a trailing 90 minutes',
        source=DataSource(stream_source=StreamSource(full_name='${var.stream}')),
        function=Function(
            aggregation_function=AggregationFunction(
                count_function=CountFunction(input='value.amount'),
                time_window=TimeWindow(rolling=RollingWindow(window_duration='5400s')),
            )
        ),
        entities=[EntityColumn(name='value.key')],
        timeseries_column=TimeseriesColumn(name='value.event_timestamp'),
    ),
)

resources.add_feature(
    'amount_sum_90m',
    Feature(
        full_name='${var.catalog}.${var.schema}.amount_sum_90m',
        description='Sum of amount per key over a trailing 90 minutes',
        source=DataSource(stream_source=StreamSource(full_name='${var.stream}')),
        function=Function(
            aggregation_function=AggregationFunction(
                sum=SumFunction(input='value.amount'),
                time_window=TimeWindow(rolling=RollingWindow(window_duration='5400s')),
            )
        ),
        entities=[EntityColumn(name='value.key')],
        timeseries_column=TimeseriesColumn(name='value.event_timestamp'),
    ),
)

resources.add_materialization_set(
    'serving_online',
    MaterializationSet(
        materialized_features=[
            MaterializedFeature(
                feature_name='${resources.features.events_90m.full_name}',
                online_store_config=OnlineStoreConfig(
                    online_store_name='${var.online_store}',
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='serving_online',
                ),
                streaming_mode=StreamingMode(
                    mode=StreamingModeStreamingModeType.STREAMING_MODE_TYPE_RTM
                ),
            ),
            MaterializedFeature(
                feature_name='${resources.features.amount_sum_90m.full_name}',
                online_store_config=OnlineStoreConfig(
                    online_store_name='${var.online_store}',
                    catalog_name='${var.catalog}',
                    schema_name='${var.schema}',
                    table_name_prefix='serving_online',
                ),
                streaming_mode=StreamingMode(
                    mode=StreamingModeStreamingModeType.STREAMING_MODE_TYPE_RTM
                ),
            ),
        ]
    ),
)

Add a FeatureSpec

The FeatureSpec lists the features the endpoint serves. Each entry in features names one Feature View with declarative_feature.

A FeatureSpec can be created only after each Feature View it names is materialized to an online store. The ${resources.features...} references in declarative_feature make the spec depend on the Feature Views events_90m and amount_sum_90m, and lifecycle.depends_on adds the dependency that no reference expresses, on the materialization set serving_online. The bundle creates the spec after all three and deletes it before them. See Rules a FeatureSpec must satisfy.

YAML

Create resources/feature_spec.yml:

resources:
  feature_specs:
    event_features:
      name: ${var.catalog}.${var.schema}.event_features
      features:
        - declarative_feature:
            feature: ${resources.features.events_90m.full_name}
        - declarative_feature:
            feature: ${resources.features.amount_sum_90m.full_name}
      lifecycle:
        depends_on:
          - resource: materialization_sets.serving_online

Python

Add the imports to resources/__init__.py:

from databricks.bundles.feature_specs import (
    DeclarativeFeature,
    DependsOn,
    FeatureSpec,
    Features,
    Lifecycle,
)

Then add this to the body of load_resources:

resources.add_feature_spec(
    'event_features',
    FeatureSpec(
        name='${var.catalog}.${var.schema}.event_features',
        features=[
            Features(
                declarative_feature=DeclarativeFeature(
                    feature='${resources.features.events_90m.full_name}'
                )
            ),
            Features(
                declarative_feature=DeclarativeFeature(
                    feature='${resources.features.amount_sum_90m.full_name}'
                )
            ),
        ],
        lifecycle=Lifecycle(depends_on=[DependsOn(resource='materialization_sets.serving_online')]),
    ),
)

Add a Feature Serving endpoint

The endpoint serves the FeatureSpec that entity_name names. Because entity_name references the spec's name, the bundle creates the endpoint after the spec, as the output in Deploy shows, and deletes it before the spec. With scale_to_zero_enabled, the endpoint scales its compute down to zero when it receives no requests.

YAML

Create resources/serving_endpoint.yml:

resources:
  model_serving_endpoints:
    event_endpoint:
      name: ${var.endpoint_name}
      config:
        served_entities:
          - entity_name: ${resources.feature_specs.event_features.name}
            workload_size: Small
            scale_to_zero_enabled: true

Python

Add the imports to resources/__init__.py:

from databricks.bundles.model_serving_endpoints import (
    EndpointCoreConfigInput,
    ModelServingEndpoint,
    ServedEntityInput,
)

Then add this to the body of load_resources:

resources.add_model_serving_endpoint(
    'event_endpoint',
    ModelServingEndpoint(
        name='${var.endpoint_name}',
        config=EndpointCoreConfigInput(
            served_entities=[
                ServedEntityInput(
                    entity_name='${resources.feature_specs.event_features.name}',
                    workload_size='Small',
                    scale_to_zero_enabled=True,
                )
            ]
        ),
    ),
)

Deploy

Check the configuration, and then review what the deployment does:

databricks bundle validate
databricks bundle plan

On a first deployment, all five resources report create:

create feature_specs.event_features
create features.amount_sum_90m
create features.events_90m
create materialization_sets.serving_online
create model_serving_endpoints.event_endpoint

Plan: 5 to add, 0 to change, 0 to delete, 0 unchanged

Deploy the bundle to your workspace:

databricks bundle deploy

The deploy creates the resources in dependency order and returns when the endpoint is ready, which takes several minutes:

Uploading bundle files to /Workspace/Users/someone@example.com/.bundle/feature-serving-quickstart/default/files...
Created features.events_90m
Created features.amount_sum_90m
Created materialization_sets.serving_online
Created feature_specs.event_features
Created model_serving_endpoints.event_endpoint
Files: 4 uploaded, 0 deleted
Resources: 5 created, 0 changed, 0 deleted, 0 unchanged

Query the endpoint

Send the keys to look up as dataframe_records. A record names each key by the last segment of the Feature View's entity path, so value.key becomes key:

databricks serving-endpoints query event-features --json '{"dataframe_records": [{"key": 1280}, {"key": 1188}, {"key": 42}, {"key": 10001}]}'

The response holds one object for each record, in the order of the request. Each object holds one value for each Feature View, keyed by the Feature View's name without its catalog and schema:

{
  "outputs": [
    {
      "events_90m": 10,
      "amount_sum_90m": 10
    },
    {
      "events_90m": 11,
      "amount_sum_90m": 11
    },
    {
      "events_90m": 9,
      "amount_sum_90m": 10
    },
    {
      "events_90m": 0,
      "amount_sum_90m": null
    }
  ]
}

A key with no events in the window returns 0 for the count and null for the sum, as key 10001 does. Every key returns these values until the materialization pipeline writes its first rows, which can be several minutes after databricks bundle deploy returns. For other ways to query an endpoint, see Query an endpoint.

Remove a Feature View from the FeatureSpec

Any change to a FeatureSpec deletes and re-creates it. A running endpoint keeps serving the features in the FeatureSpec it was built with, so a change that keeps the spec's name does not affect the endpoint. See Update dependencies of a live Feature Serving endpoint.

To see this, remove amount_sum_90m from the spec and keep its name:

YAML

   feature_specs:
     event_features:
       name: ${var.catalog}.${var.schema}.event_features
       features:
         - declarative_feature:
             feature: ${resources.features.events_90m.full_name}
-        - declarative_feature:
-            feature: ${resources.features.amount_sum_90m.full_name}
       lifecycle:

Python

         features=[
             Features(
                 declarative_feature=DeclarativeFeature(
                     feature='${resources.features.events_90m.full_name}'
                 )
             ),
-            Features(
-                declarative_feature=DeclarativeFeature(
-                    feature='${resources.features.amount_sum_90m.full_name}'
-                )
-            ),
         ],

databricks bundle plan reports a recreate of the spec only:

recreate feature_specs.event_features

Plan: 1 to add, 0 to change, 1 to delete, 4 unchanged

databricks bundle deploy asks for confirmation before it re-creates the spec:

Uploading bundle files to /Workspace/Users/someone@example.com/.bundle/feature-serving-quickstart/default/files...

This action will result in the deletion or recreation of the following feature specs.
Grants on them that are not declared in the bundle are lost:
  recreate resources.feature_specs.event_features

Would you like to proceed? [y/N]: y
Recreated feature_specs.event_features
Files: 1 uploaded, 0 deleted
Resources: 1 created, 0 changed, 1 deleted, 4 unchanged

The endpoint still returns both features:

databricks serving-endpoints query event-features --json '{"dataframe_records": [{"key": 1188}, {"key": 42}]}'
{
  "outputs": [
    {
      "events_90m": 16,
      "amount_sum_90m": 20
    },
    {
      "events_90m": 12,
      "amount_sum_90m": 11
    }
  ]
}

Rename the FeatureSpec

To change what the endpoint serves, give the FeatureSpec a new name. The endpoint's entity_name references the name, so the deploy also updates the endpoint, and the endpoint keeps answering requests with the features in the original FeatureSpec until the update completes.

YAML

   feature_specs:
     event_features:
-      name: ${var.catalog}.${var.schema}.event_features
+      name: ${var.catalog}.${var.schema}.event_features_v2

Python

     FeatureSpec(
-        name='${var.catalog}.${var.schema}.event_features',
+        name='${var.catalog}.${var.schema}.event_features_v2',

databricks bundle plan reports a recreate of the spec and an update of the endpoint:

recreate feature_specs.event_features
update model_serving_endpoints.event_endpoint

Plan: 1 to add, 1 to change, 1 to delete, 3 unchanged

databricks bundle deploy asks for confirmation, re-creates the spec, and returns when the endpoint has been updated:

Uploading bundle files to /Workspace/Users/someone@example.com/.bundle/feature-serving-quickstart/default/files...

This action will result in the deletion or recreation of the following feature specs.
Grants on them that are not declared in the bundle are lost:
  recreate resources.feature_specs.event_features

Would you like to proceed? [y/N]: y
Recreated feature_specs.event_features
Updated model_serving_endpoints.event_endpoint
Files: 1 uploaded, 0 deleted
Resources: 1 created, 1 changed, 1 deleted, 3 unchanged

The endpoint now returns only events_90m:

databricks serving-endpoints query event-features --json '{"dataframe_records": [{"key": 1188}]}'
{
  "outputs": [
    {
      "events_90m": 68
    }
  ]
}

Clean up

The materialization pipeline keeps running, and the endpoint keeps serving, until you delete them. Tear the bundle down when you are finished:

databricks bundle destroy

databricks bundle destroy lists what it deletes and asks for confirmation:

The following resources will be deleted:
  delete resources.feature_specs.event_features
  delete resources.features.amount_sum_90m
  delete resources.features.events_90m
  delete resources.materialization_sets.serving_online
  delete resources.model_serving_endpoints.event_endpoint

This action will result in the deletion of the following Feature Views.
Deleting a Feature View that a FeatureSpec serves is not blocked, and leaves the FeatureSpec stale:
  delete resources.features.amount_sum_90m
  delete resources.features.events_90m

This action will result in the deletion of the following feature specs.
Feature Serving endpoints and training code that use them can no longer load them:
  delete resources.feature_specs.event_features

All files and directories at the following location will be deleted: /Workspace/Users/someone@example.com/.bundle/feature-serving-quickstart/default

Would you like to proceed? [y/N]: y
Destroy: 5 deleted

Only the user who created a materialized feature can delete it, so run databricks bundle destroy as the same identity that deployed the bundle. See delete_materialized_feature().

Limitations

  • When a Feature View is materialized in batch mode, its online table exists only after the materialization set's first run, so an endpoint deployed in the same databricks bundle deploy fails. Run databricks bundle deploy again after the first run.
  • Deleting a Feature View that a FeatureSpec serves is not blocked, and it leaves the FeatureSpec stale.

Next steps