Feature Store cost management

This page describes how Azure Databricks Feature Store charges for compute and how to monitor and optimize those costs. Feature Store is billed at cost: you pay for the underlying serverless compute, online store, and serving infrastructure with no premium on top.

  • Feature materialization runs as serverless compute and appears in billing under the FEATURE_STORE product, billed at the same rate as serverless jobs and Lakeflow pipelines.
  • Feature serving endpoints are billed against the Model Serving SKU.
  • Online stores are billed against Lakebase compute, based on capacity unit (CU) and replica counts.

Note

Initially during Public Preview, feature materialization pipelines are not billed. After billing is enabled, they will incur ongoing serverless compute costs. Other charges, including feature serving endpoints and online stores, apply during the preview. Plan for materialization compute costs when you move workloads to production.

How Feature Store is billed

Feature materialization

Important

Feature materialization is part of Feature Views, which is in Public Preview.

When you materialize Feature Views, Azure Databricks runs a serverless pipeline to compute and write feature values to offline and online destinations. This includes batch materialization jobs, streaming pipelines, and Kafka ingestion. These pipelines run on serverless compute and are charged at cost, with a 1x multiplier and no premium over the underlying serverless platform.

In the billable usage system table, materialization usage appears with billing_origin_product set to FEATURE_STORE and a serverless compute SKU (JOBS_SERVERLESS_COMPUTE, prefixed by your workspace tier and region). The usage is is_serverless and is_photon, and usage_metadata includes data_source (DELTA_TABLE_SOURCE or KAFKA_SOURCE) and operation (FEATURE_MATERIALIZATION or KAFKA_INGESTION) so you can slice cost by batch versus streaming workloads.

To reduce compute cost, group features that share an offline destination, online destination, and trigger into a single materialize_features call so they run in one pipeline. See Materialize Feature Views. For more about how serverless compute is billed, see Run your Lakeflow Jobs with serverless compute for workflows.

Feature serving endpoints

Feature serving endpoints serve precomputed and on-demand features to real-time applications. They run on Model Serving and are billed against the Model Serving SKU (SERVERLESS_REAL_TIME_INFERENCE), the same as custom model serving endpoints. Charges scale with the compute the endpoint provisions to handle request traffic. See Feature Serving endpoints and the Model Serving pricing page.

Online stores

Online stores serve features at low latency for online inference. A Azure Databricks online store is powered by Lakebase and is billed against Lakebase compute, based on the capacity units (CUs) and read replicas it provisions. Each capacity unit allocates compute, memory, and storage to the instance, and you can add read replicas for higher availability and read throughput. See Databricks Online Feature Stores for sizing guidance, Manage computes for how capacity units map to compute, and the Lakebase pricing page.

Attribute costs with tags and serverless usage policies

Important

This feature is in Private Preview. To try it, reach out to your Azure Databricks contact.

When you create a materialization or a Stream, pass custom tags, a serverless usage policy, or both. Azure Databricks applies them to the job or Lakeflow pipeline it creates, so its spend carries your attribution in the billable usage system table. To create a policy and get its ID, see Create a serverless usage policy.

Both values are applied when the job or pipeline is created. To attribute existing compute differently, create a new materialization or Stream.

Attribute materialization compute

Pass tags, budget_policy_id, or both to materialize_features:

materialized = fe.materialize_features(
    features=features,
    offline_config=offline_config,
    trigger=trigger,
    tags={"team": "ml-platform", "project": "churn"},
    budget_policy_id="555e8888-e999-4444-a777-446655440000",
)

For the full signature, see Materialize Feature Views.

Attribute managed ingestion compute

For a Stream, set tags and budget_policy_id on the IngestionConfig you pass to create_stream:

from databricks.feature_engineering.entities import IngestionConfig, IngestionDestination

ingestion_config = IngestionConfig(
    ingestion_destination=IngestionDestination(
        delta_table_name="my_catalog.my_schema.events_ingestion"
    ),
    tags={"team": "ml-platform", "project": "churn"},
    budget_policy_id="555e8888-e999-4444-a777-446655440000",
)

For the remaining ingestion options, see Set up a Stream.

Compute the attribution applies to

The tags and the serverless usage policy apply to the compute that materializes features or ingests Stream data: the batch materialization job, the streaming Lakeflow pipeline and the job that orchestrates it, and a Stream's ingestion Lakeflow pipeline with its forward-fill and backfill jobs. When you materialize to a new online table, they also apply to the Lakeflow pipeline that keeps that table in sync.

They do not apply to the online store itself. To attribute an online store's own compute, set a serverless usage policy on the store. See Databricks Online Feature Stores.

Limitations

  • Materializing to an online table that already exists applies neither tags nor budget_policy_id, and returns no error. The Lakeflow pipeline that keeps that table in sync retains the tags it was created with. To change attribution, materialize to a new online table.
  • You can pass at most 25 tags.
  • A tag key can be up to 127 characters and a tag value up to 65,535 characters. Neither can be empty.
  • A tag key cannot begin with databricks:. That prefix is reserved, and the check ignores case.

Monitor usage and costs

You can monitor Feature Store costs using the billable usage system table, system.billing.usage. Materialization usage is identified by billing_origin_product = 'FEATURE_STORE':

SELECT
  usage_date,
  usage_metadata.data_source,
  usage_metadata.operation,
  usage_metadata.job_id,
  usage_metadata.dlt_pipeline_id,
  identity_metadata.run_as,
  sum(usage_quantity) AS dbus,
  usage_unit
FROM system.billing.usage
WHERE billing_origin_product = 'FEATURE_STORE'
GROUP BY ALL;

Use usage_metadata.data_source and operation to break costs down by batch versus streaming. The usage_metadata.job_id and usage_metadata.dlt_pipeline_id fields identify the specific materialization job or pipeline that produced each cost row, so you can group by them for per-pipeline attribution.

The custom_tags column holds the tags you passed alongside the tags carried by the serverless usage policy, so you can group or filter by custom_tags['<key>'] to attribute spend. For how policy tags reach the billing records, see Analyze serverless usage policy tags in the billing records.

Feature serving and online store costs are tracked separately:

  • Feature serving endpoints appear under the Model Serving SKU. See Monitor model serving costs.
  • Online store compute appears under the Lakebase (database) serverless SKU.

For details on the billable usage table and how to query it, see Billable usage system table reference.

Cost optimization best practices

  • Group features into shared materialization pipelines: Features that share an offline destination, online destination, and trigger can materialize together in a single pipeline, which reduces the number of pipelines you pay for.
  • Reuse online stores: You can publish multiple feature tables to a single online store. For development, testing, and training, share one online store across projects rather than creating separate stores.
  • Right-size online store capacity: Start with a small capacity unit for testing and scale up or down based on performance and cost.
  • Delete resources that are not in use: Online stores continuously incur costs. Delete online stores and materialization pipelines that are no longer needed.
  • Choose an appropriate materialization trigger: Less frequent scheduled triggers cost less than continuous or frequent re-materialization. Match the trigger to how fresh your features need to be.

Additional resources

  • Last updated on