MLflow Model Registry Webhooks on Azure Databricks

  • Article

Important

This feature is in Public Preview.

Webhooks enable you to listen for Model Registry events so your integrations can automatically trigger actions. You can use webhooks to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows. For example, you can trigger CI builds when a new model version is created or notify your team members through Slack each time a model transition to production is requested.

Webhooks are available through the Databricks REST API or the Python client databricks-registry-webhooks on PyPI.

Note

Webhooks are not available when you use Models in Unity Catalog. For an alternative, see Can I use stage transition requests or trigger webhooks on events?.

Webhook events

You can specify a webhook to trigger upon one or more of these events:

  • MODEL_VERSION_CREATED: A new model version was created for the associated model.
  • MODEL_VERSION_TRANSITIONED_STAGE: A model version’s stage was changed.
  • TRANSITION_REQUEST_CREATED: A user requested a model version’s stage be transitioned.
  • COMMENT_CREATED: A user wrote a comment on a registered model.
  • REGISTERED_MODEL_CREATED: A new registered model was created. This event type can only be specified for a registry-wide webhook, which can be created by not specifying a model name in the create request.
  • MODEL_VERSION_TAG_SET: A user set a tag on the model version.
  • MODEL_VERSION_TRANSITIONED_TO_STAGING: A model version was transitioned to staging.
  • MODEL_VERSION_TRANSITIONED_TO_PRODUCTION: A model version was transitioned to production.
  • MODEL_VERSION_TRANSITIONED_TO_ARCHIVED: A model version was archived.
  • TRANSITION_REQUEST_TO_STAGING_CREATED: A user requested a model version be transitioned to staging.
  • TRANSITION_REQUEST_TO_PRODUCTION_CREATED: A user requested a model version be transitioned to production.
  • TRANSITION_REQUEST_TO_ARCHIVED_CREATED: A user requested a model version be archived.

Types of webhooks

There are two types of webhooks based on their trigger targets:

  • Webhooks with HTTP endpoints (HTTP registry webhooks): Send triggers to an HTTP endpoint.
  • Webhooks with job triggers (job registry webhooks): Trigger a job in an Azure Databricks workspace. If IP allowlisting is enabled in the job’s workspace, you must allowlist the workspace IPs of the model registry. See IP allowlisting for job registry webhooks for more information.

There are also two types of webhooks based on their scope, with different access control requirements:

  • Model-specific webhooks: The webhook applies to a specific registered model. You must have CAN MANAGE permissions on the registered model to create, modify, delete, or test model-specific webhooks.
  • Registry-wide webhooks: The webhook is triggered by events on any registered model in the workspace, including the creation of a new registered model. To create a registry-wide webhook, omit the model_name field on creation. You must have workspace admin permissions to create, modify, delete, or test registry-wide webhooks.

Webhook payload

Each event trigger has minimal fields included in the payload for the outgoing request to the webhook endpoint.

  • Sensitive information like artifact path location is excluded. Users and principals with appropriate ACLs can use client or REST APIs to query the Model Registry for this information.
  • Payloads are not encrypted. See Security for information on how to validate that Azure Databricks is the source of the webhook.
  • The text field facilitates Slack integration. To send a Slack message, provide a Slack webhook endpoint as the webhook URL.

Job registry webhook payload

The payload for a job registry webhook depends on the type of job and is sent to the jobs/run-now endpoint in the target workspace.

Single-task jobs

Single-task jobs have one of three payloads based on the task type.

Notebook and Python wheel jobs

Notebook and Python wheel jobs have a JSON payload with a parameter dictionary that contains a field event_message.

{
  "job_id": 1234567890,
  "notebook_params": {
    "event_message": "<Webhook Payload>"
  }
}
Python, JAR, and Spark Submit jobs

Python, JAR, and Spark submit jobs have a JSON payload with a parameter list.

{
  "job_id": 1234567890,
  "python_params": ["<Webhook Payload>"]
}
All other jobs

All other types of jobs have a JSON payload with no parameters.

{
  "job_id": 1234567890
}

Multi-task jobs

Multi-task jobs have a JSON payload with all parameters populated to account for different task types.

{
  "job_id": 1234567890,
  "notebook_params": {
    "event_message": "<Webhook Payload>"
  },
  "python_named_params": {
    "event_message": "<Webhook Payload>"
  },
  "jar_params": ["<Webhook Payload>"],
  "python_params": ["<Webhook Payload>"],
  "spark_submit_params": ["<Webhook Payload>"]
}

Example payloads

event: MODEL_VERSION_TRANSITIONED_STAGE

Response

POST
/your/endpoint/for/event/model-versions/stage-transition
--data {
  "event": "MODEL_VERSION_TRANSITIONED_STAGE",
  "webhook_id": "c5596721253c4b429368cf6f4341b88a",
  "event_timestamp": 1589859029343,
  "model_name": "Airline_Delay_SparkML",
  "version": "8",
  "to_stage": "Production",
  "from_stage": "None",
  "text": "Registered model 'someModel' version 8 transitioned from None to Production."
}

event: MODEL_VERSION_TAG_SET

Response

POST
/your/endpoint/for/event/model-versions/tag-set
--data {
  "event": "MODEL_VERSION_TAG_SET",
  "webhook_id": "8d7fc634e624474f9bbfde960fdf354c",
  "event_timestamp": 1589859029343,
  "model_name": "Airline_Delay_SparkML",
  "version": "8",
  "tags": [{"key":"key1","value":"value1"},{"key":"key2","value":"value2"}],
  "text": "example@yourdomain.com set version tag(s) 'key1' => 'value1', 'key2' => 'value2' for registered model 'someModel' version 8."
}

event: COMMENT_CREATED

Response

POST
/your/endpoint/for/event/comments/create
--data {
  "event": "COMMENT_CREATED",
  "webhook_id": "8d7fc634e624474f9bbfde960fdf354c",
  "event_timestamp": 1589859029343,
  "model_name": "Airline_Delay_SparkML",
  "version": "8",
  "comment": "Raw text content of the comment",
  "text": "A user commented on registered model 'someModel' version 8."
}

Security

For security, Azure Databricks includes the X-Databricks-Signature in the header computed from the payload and the shared secret key associated with the webhook using the HMAC with SHA-256 algorithm.

In addition, you can include a standard Authorization header in the outgoing request by specifying one in the HttpUrlSpec of the webhook.

Client verification

If a shared secret is set, the payload recipient should verify the source of the HTTP request by using the shared secret to HMAC-encode the payload, and then comparing the encoded value with the X-Databricks-Signature from the header. This is particularly important if SSL certificate validation is disabled (that is, if the enable_ssl_verification field is set to false).

Note

enable_ssl_verification is true by default. For self-signed certificates, this field must be false, and the destination server must disable certificate validation.

For security purposes, Databricks recommends that you perform secret validation with the HMAC-encoded portion of the payload. If you disable host name validation, you increase the risk that a request could be maliciously routed to an unintended host.

import hmac
import hashlib
import json

secret = shared_secret.encode('utf-8')
signature_key = 'X-Databricks-Signature'

def validate_signature(request):
  if not request.headers.has_key(signature_key):
    raise Exception('No X-Signature. Webhook not be trusted.')

  x_sig = request.headers.get(signature_key)
  body = request.body.encode('utf-8')
  h = hmac.new(secret, body, hashlib.sha256)
  computed_sig = h.hexdigest()

  if not hmac.compare_digest(computed_sig, x_sig.encode()):
    raise Exception('X-Signature mismatch. Webhook not be trusted.')

Authorization header for HTTP registry webhooks

If an Authorization header is set, clients should verify the source of the HTTP request by verifying the bearer token or authorization credentials in the Authorization header.

IP allowlisting for job registry webhooks

To use a webhook that triggers job runs in a different workspace that has IP allowlisting enabled, you must allowlist the region NAT IP where the webhook is located to accept incoming requests.

If the webhook and the job are in the same workspace, you do not need to add any IPs to your allowlist.

If your job is located in an Azure multitenant region, see Azure Databricks control plane addresses. For all other regions, contact your account team to identify the IPs you need to allowlist.

Audit logging

If audit logging is enabled for your workspace, the following events are included in the audit logs:

  • Create webhook
  • Update webhook
  • List webhook
  • Delete webhook
  • Test webhook
  • Webhook trigger

Webhook trigger audit logging

For webhooks with HTTP endpoints, the HTTP request sent to the URL specified for the webhook along with the URL and enable_ssl_verification values are logged.

For webhooks with job triggers, the job_id and workspace_url values are logged.

Examples

This section includes:

HTTP registry webhook example workflow

1. Create a webhook

When an HTTPS endpoint is ready to receive the webhook event request, you can create a webhook using the webhooks Databricks REST API. For example, the webhook’s URL can point to Slack to post messages to a channel.

$ curl -X POST -H "Authorization: Bearer <access-token>" -d \
'{"model_name": "<model-name>",
  "events": ["MODEL_VERSION_CREATED"],
  "description": "Slack notifications",
  "status": "TEST_MODE",
  "http_url_spec": {
    "url": "https://hooks.slack.com/services/...",
    "secret": "anyRandomString"
    "authorization": "Bearer AbcdEfg1294"}}' https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/create

from databricks_registry_webhooks import RegistryWebhooksClient, HttpUrlSpec

http_url_spec = HttpUrlSpec(
  url="https://hooks.slack.com/services/...",
  secret="secret_string",
  authorization="Bearer AbcdEfg1294"
)
http_webhook = RegistryWebhooksClient().create_webhook(
  model_name="<model-name>",
  events=["MODEL_VERSION_CREATED"],
  http_url_spec=http_url_spec,
  description="Slack notifications",
  status="TEST_MODE"
)

Response

{"webhook": {
   "id":"1234567890",
   "creation_timestamp":1571440826026,
   "last_updated_timestamp":1582768296651,
   "status":"TEST_MODE",
   "events":["MODEL_VERSION_CREATED"],
   "http_url_spec": {
     "url": "https://hooks.slack.com/services/...",
     "enable_ssl_verification": True
}}}

You can also create an HTTP registry webhook with the Databricks Terraform provider and databricks_mlflow_webhook.

2. Test the webhook

The previous webhook was created in TEST_MODE, so a mock event can be triggered to send a request to the specified URL. However, the webhook does not trigger on a real event. The test endpoint returns the received status code and body from the specified URL.

$ curl -X POST -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/test

from databricks_registry_webhooks import RegistryWebhooksClient

http_webhook = RegistryWebhooksClient().test_webhook(
  id="1234567890"
)

Response

{
 "status":200,
 "body":"OK"
}

3. Update the webhook to active status

To enable the webhook for real events, set its status to ACTIVE through an update call, which can also be used to change any of its other properties.

$ curl -X PATCH -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890", "status": "ACTIVE"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/update

from databricks_registry_webhooks import RegistryWebhooksClient

http_webhook = RegistryWebhooksClient().update_webhook(
  id="1234567890",
  status="ACTIVE"
)

Response

{"webhook": {
   "id":"1234567890",
   "creation_timestamp":1571440826026,
   "last_updated_timestamp":1582768296651,
   "status": "ACTIVE",
   "events":["MODEL_VERSION_CREATED"],
   "http_url_spec": {
     "url": "https://hooks.slack.com/services/...",
     "enable_ssl_verification": True
}}}

4. Delete the webhook

To disable the webhook, set its status to DISABLED (using a similar update command as above), or delete it.

$ curl -X DELETE -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/delete

from databricks_registry_webhooks import RegistryWebhooksClient

http_webhook = RegistryWebhooksClient().delete_webhook(
  id="1234567890"
)

Response

{}

Job registry webhook example workflow

The workflow for managing job registry webhooks is similar to HTTP registry webhooks, with the only difference being the job_spec field that replaces the http_url_spec field.

With webhooks, you can trigger jobs in the same workspace or in a different workspace. The workspace is specified using the optional parameter workspace_url. If no workspace_url is present, the default behavior is to trigger a job in the same workspace as the webhook.

Requirements

  • An existing job.
  • A personal access token. Note that access tokens are not included in the webhook object returned by the APIs.

Note

As a security best practice, when you authenticate with automated tools, systems, scripts, and apps, Databricks recommends that you use personal access tokens belonging to service principals instead of workspace users. To create tokens for service principals, see Manage tokens for a service principal.

Create a job registry webhook

$ curl -X POST -H "Authorization: Bearer <access-token>" -d \ '{"model_name": "<model-name>",
  "events": ["TRANSITION_REQUEST_CREATED"],
  "description": "Job webhook trigger",
  "status": "TEST_MODE",
  "job_spec": {
    "job_id": "1",
    "workspace_url": "https://my-databricks-workspace.com",
    "access_token": "dapi12345..."}}'
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/create

from databricks_registry_webhooks import RegistryWebhooksClient, JobSpec

job_spec = JobSpec(
  job_id="1",
  workspace_url="https://my-databricks-workspace.com",
  access_token="dapi12345..."
)
job_webhook = RegistryWebhooksClient().create_webhook(
  model_name="<model-name>",
  events=["TRANSITION_REQUEST_CREATED"],
  job_spec=job_spec,
  description="Job webhook trigger",
  status="TEST_MODE"
)

Response

{"webhook": {
   "id":"1234567891",
   "creation_timestamp":1591440826026,
   "last_updated_timestamp":1591440826026,
   "status":"TEST_MODE",
   "events":["TRANSITION_REQUEST_CREATED"],
   "job_spec": {
     "job_id": "1",
     "workspace_url": "https://my-databricks-workspace.com"
}}}

You can also create a job registry webhook with the Databricks Terraform provider and databricks_mlflow_webhook.

List registry webhooks example

$ curl -X GET -H "Authorization: Bearer <access-token>" -d \ '{"model_name": "<model-name>"}'
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/list

from databricks_registry_webhooks import RegistryWebhooksClient

webhooks_list = RegistryWebhooksClient().list_webhooks(model_name="<model-name>")

Response

{"webhooks": [{
   "id":"1234567890",
   "creation_timestamp":1571440826026,
   "last_updated_timestamp":1582768296651,
   "status": "ACTIVE",
   "events":["MODEL_VERSION_CREATED"],
   "http_url_spec": {
     "url": "https://hooks.slack.com/services/...",
     "enable_ssl_verification": True
}},
{
   "id":"1234567891",
   "creation_timestamp":1591440826026,
   "last_updated_timestamp":1591440826026,
   "status":"TEST_MODE",
   "events":["TRANSITION_REQUEST_CREATED"],
   "job_spec": {
     "job_id": "1",
     "workspace_url": "https://my-databricks-workspace.com"
}}]}

Notebooks

MLflow Model Registry webhooks REST API example notebook

Get notebook

MLflow Model Registry webhooks Python client example notebook

Get notebook