MLflow Model Registry Webhooks on Azure Databricks
Important
This feature is in Public Preview.
Webhooks enable you to listen for Model Registry events so your integrations can automatically trigger actions. You can use webhooks to automate and integrate your machine learning pipeline with existing CI/CD tools and workflows. For example, you can trigger CI builds when a new model version is created or notify your team members through Slack each time a model transition to production is requested.
Webhooks are available through the Databricks REST API or the Python client databricks-registry-webhooks
on PyPI.
Note
Webhooks are not available when you use Models in Unity Catalog. For an alternative, see Can I use stage transition requests or trigger webhooks on events?. Sending webhooks to private endpoints (endpoints that are not accessible from the public internet) is not supported.
Webhook events
You can specify a webhook to trigger upon one or more of these events:
- MODEL_VERSION_CREATED: A new model version was created for the associated model.
- MODEL_VERSION_TRANSITIONED_STAGE: A model version’s stage was changed.
- TRANSITION_REQUEST_CREATED: A user requested a model version’s stage be transitioned.
- COMMENT_CREATED: A user wrote a comment on a registered model.
- REGISTERED_MODEL_CREATED: A new registered model was created. This event type can only be specified for a registry-wide webhook, which can be created by not specifying a model name in the create request.
- MODEL_VERSION_TAG_SET: A user set a tag on the model version.
- MODEL_VERSION_TRANSITIONED_TO_STAGING: A model version was transitioned to staging.
- MODEL_VERSION_TRANSITIONED_TO_PRODUCTION: A model version was transitioned to production.
- MODEL_VERSION_TRANSITIONED_TO_ARCHIVED: A model version was archived.
- TRANSITION_REQUEST_TO_STAGING_CREATED: A user requested a model version be transitioned to staging.
- TRANSITION_REQUEST_TO_PRODUCTION_CREATED: A user requested a model version be transitioned to production.
- TRANSITION_REQUEST_TO_ARCHIVED_CREATED: A user requested a model version be archived.
Types of webhooks
There are two types of webhooks based on their trigger targets:
- Webhooks with HTTP endpoints (HTTP registry webhooks): Send triggers to an HTTP endpoint.
- Webhooks with job triggers (job registry webhooks): Trigger a job in an Azure Databricks workspace. If IP allowlisting is enabled in the job’s workspace, you must allowlist the workspace IPs of the model registry. See IP allowlisting for job registry webhooks for more information.
There are also two types of webhooks based on their scope, with different access control requirements:
- Model-specific webhooks: The webhook applies to a specific registered model. You must have CAN MANAGE permissions on the registered model to create, modify, delete, or test model-specific webhooks.
- Registry-wide webhooks: The webhook is triggered by events on any registered model in the workspace, including the creation of a new registered model. To create a registry-wide webhook, omit the
model_name
field on creation. You must have workspace admin permissions to create, modify, delete, or test registry-wide webhooks.
Webhook payload
Each event trigger has minimal fields included in the payload for the outgoing request to the webhook endpoint.
- Sensitive information like artifact path location is excluded. Users and principals with appropriate ACLs can use client or REST APIs to query the Model Registry for this information.
- Payloads are not encrypted. See Security for information on how to validate that Azure Databricks is the source of the webhook.
- The
text
field facilitates Slack integration. To send a Slack message, provide a Slack webhook endpoint as the webhook URL.
Job registry webhook payload
The payload for a job registry webhook depends on the type of job and is sent to the jobs/run-now
endpoint in the target workspace.
Single-task jobs
Single-task jobs have one of three payloads based on the task type.
Notebook and Python wheel jobs
Notebook and Python wheel jobs have a JSON payload with a parameter dictionary that contains a field event_message
.
{
"job_id": 1234567890,
"notebook_params": {
"event_message": "<Webhook Payload>"
}
}
Python, JAR, and Spark Submit jobs
Python, JAR, and Spark submit jobs have a JSON payload with a parameter list.
{
"job_id": 1234567890,
"python_params": ["<Webhook Payload>"]
}
All other jobs
All other types of jobs have a JSON payload with no parameters.
{
"job_id": 1234567890
}
Multi-task jobs
Multi-task jobs have a JSON payload with all parameters populated to account for different task types.
{
"job_id": 1234567890,
"notebook_params": {
"event_message": "<Webhook Payload>"
},
"python_named_params": {
"event_message": "<Webhook Payload>"
},
"jar_params": ["<Webhook Payload>"],
"python_params": ["<Webhook Payload>"],
"spark_submit_params": ["<Webhook Payload>"]
}
Example payloads
event: MODEL_VERSION_TRANSITIONED_STAGE
Response
POST
/your/endpoint/for/event/model-versions/stage-transition
--data {
"event": "MODEL_VERSION_TRANSITIONED_STAGE",
"webhook_id": "c5596721253c4b429368cf6f4341b88a",
"event_timestamp": 1589859029343,
"model_name": "Airline_Delay_SparkML",
"version": "8",
"to_stage": "Production",
"from_stage": "None",
"text": "Registered model 'someModel' version 8 transitioned from None to Production."
}
event: MODEL_VERSION_TAG_SET
Response
POST
/your/endpoint/for/event/model-versions/tag-set
--data {
"event": "MODEL_VERSION_TAG_SET",
"webhook_id": "8d7fc634e624474f9bbfde960fdf354c",
"event_timestamp": 1589859029343,
"model_name": "Airline_Delay_SparkML",
"version": "8",
"tags": [{"key":"key1","value":"value1"},{"key":"key2","value":"value2"}],
"text": "example@yourdomain.com set version tag(s) 'key1' => 'value1', 'key2' => 'value2' for registered model 'someModel' version 8."
}
event: COMMENT_CREATED
Response
POST
/your/endpoint/for/event/comments/create
--data {
"event": "COMMENT_CREATED",
"webhook_id": "8d7fc634e624474f9bbfde960fdf354c",
"event_timestamp": 1589859029343,
"model_name": "Airline_Delay_SparkML",
"version": "8",
"comment": "Raw text content of the comment",
"text": "A user commented on registered model 'someModel' version 8."
}
Security
For security, Azure Databricks includes the X-Databricks-Signature in the header computed from the payload and the shared secret key associated with the webhook using the HMAC with SHA-256 algorithm.
In addition, you can include a standard Authorization header in the outgoing request by specifying one in the HttpUrlSpec
of the webhook.
Client verification
If a shared secret is set, the payload recipient should verify the source of the HTTP request by using the shared secret to HMAC-encode the payload, and then comparing the encoded value with the X-Databricks-Signature
from the header. This is particularly important if SSL certificate validation is disabled (that is, if the enable_ssl_verification
field is set to false
).
Note
enable_ssl_verification
is true
by default. For self-signed certificates, this field must be false
, and the destination server must disable certificate validation.
For security purposes, Databricks recommends that you perform secret validation with the HMAC-encoded portion of the payload. If you disable host name validation, you increase the risk that a request could be maliciously routed to an unintended host.
import hmac
import hashlib
import json
secret = shared_secret.encode('utf-8')
signature_key = 'X-Databricks-Signature'
def validate_signature(request):
if not request.headers.has_key(signature_key):
raise Exception('No X-Signature. Webhook not be trusted.')
x_sig = request.headers.get(signature_key)
body = request.body.encode('utf-8')
h = hmac.new(secret, body, hashlib.sha256)
computed_sig = h.hexdigest()
if not hmac.compare_digest(computed_sig, x_sig.encode()):
raise Exception('X-Signature mismatch. Webhook not be trusted.')
Authorization header for HTTP registry webhooks
If an Authorization header is set, clients should verify the source of the HTTP request by verifying the bearer token or authorization credentials in the Authorization header.
IP allowlisting for job registry webhooks
To use a webhook that triggers job runs in a different workspace that has IP allowlisting enabled, you must allowlist the region NAT IP where the webhook is located to accept incoming requests.
If the webhook and the job are in the same workspace, you do not need to add any IPs to your allowlist.
If your job is located in an Azure multitenant region, see Azure Databricks control plane addresses. For all other regions, contact your account team to identify the IPs you need to allowlist.
Audit logging
If audit logging is enabled for your workspace, the following events are included in the audit logs:
- Create webhook
- Update webhook
- List webhook
- Delete webhook
- Test webhook
- Webhook trigger
Webhook trigger audit logging
For webhooks with HTTP endpoints, the HTTP request sent to the URL specified for the webhook along with the URL and enable_ssl_verification
values are logged.
For webhooks with job triggers, the job_id
and workspace_url
values are logged.
Examples
This section includes:
- HTTP registry webhook workflow example.
- job registry webhook workflow example.
- list webhooks example.
- two example notebooks: one illustrating the REST API, and one illustrating the Python client.
HTTP registry webhook example workflow
1. Create a webhook
When an HTTPS endpoint is ready to receive the webhook event request, you can create a webhook using the webhooks Databricks REST API. For example, the webhook’s URL can point to Slack to post messages to a channel.
$ curl -X POST -H "Authorization: Bearer <access-token>" -d \
'{"model_name": "<model-name>",
"events": ["MODEL_VERSION_CREATED"],
"description": "Slack notifications",
"status": "TEST_MODE",
"http_url_spec": {
"url": "https://hooks.slack.com/services/...",
"secret": "anyRandomString"
"authorization": "Bearer AbcdEfg1294"}}' https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/create
from databricks_registry_webhooks import RegistryWebhooksClient, HttpUrlSpec
http_url_spec = HttpUrlSpec(
url="https://hooks.slack.com/services/...",
secret="secret_string",
authorization="Bearer AbcdEfg1294"
)
http_webhook = RegistryWebhooksClient().create_webhook(
model_name="<model-name>",
events=["MODEL_VERSION_CREATED"],
http_url_spec=http_url_spec,
description="Slack notifications",
status="TEST_MODE"
)
Response
{"webhook": {
"id":"1234567890",
"creation_timestamp":1571440826026,
"last_updated_timestamp":1582768296651,
"status":"TEST_MODE",
"events":["MODEL_VERSION_CREATED"],
"http_url_spec": {
"url": "https://hooks.slack.com/services/...",
"enable_ssl_verification": True
}}}
You can also create an HTTP registry webhook with the Databricks Terraform provider and databricks_mlflow_webhook.
2. Test the webhook
The previous webhook was created in TEST_MODE
, so a mock event can be triggered to send a request to the specified URL. However, the webhook does not trigger on a real event. The test endpoint returns the received status code and body from the specified URL.
$ curl -X POST -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/test
from databricks_registry_webhooks import RegistryWebhooksClient
http_webhook = RegistryWebhooksClient().test_webhook(
id="1234567890"
)
Response
{
"status":200,
"body":"OK"
}
3. Update the webhook to active status
To enable the webhook for real events, set its status to ACTIVE
through an update call, which can also be used to change any of its other properties.
$ curl -X PATCH -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890", "status": "ACTIVE"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/update
from databricks_registry_webhooks import RegistryWebhooksClient
http_webhook = RegistryWebhooksClient().update_webhook(
id="1234567890",
status="ACTIVE"
)
Response
{"webhook": {
"id":"1234567890",
"creation_timestamp":1571440826026,
"last_updated_timestamp":1582768296651,
"status": "ACTIVE",
"events":["MODEL_VERSION_CREATED"],
"http_url_spec": {
"url": "https://hooks.slack.com/services/...",
"enable_ssl_verification": True
}}}
4. Delete the webhook
To disable the webhook, set its status to DISABLED
(using a similar update command as above), or delete it.
$ curl -X DELETE -H "Authorization: Bearer <access-token>" -d \
'{"id": "1234567890"}' \
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/delete
from databricks_registry_webhooks import RegistryWebhooksClient
http_webhook = RegistryWebhooksClient().delete_webhook(
id="1234567890"
)
Response
{}
Job registry webhook example workflow
The workflow for managing job registry webhooks is similar to HTTP registry webhooks, with the only difference being the job_spec
field that replaces the http_url_spec
field.
With webhooks, you can trigger jobs in the same workspace or in a different workspace. The workspace is specified using the optional parameter workspace_url
. If no workspace_url
is present, the default behavior is to trigger a job in the same workspace as the webhook.
Requirements
- An existing job.
- A personal access token. Note that access tokens can only be read by the MLflow service and cannot be returned by Azure Databricks users in the Model Registry API.
Note
As a security best practice, when you authenticate with automated tools, systems, scripts, and apps, Databricks recommends that you use personal access tokens belonging to service principals instead of workspace users. To create tokens for service principals, see Manage tokens for a service principal.
Create a job registry webhook
$ curl -X POST -H "Authorization: Bearer <access-token>" -d \ '{"model_name": "<model-name>",
"events": ["TRANSITION_REQUEST_CREATED"],
"description": "Job webhook trigger",
"status": "TEST_MODE",
"job_spec": {
"job_id": "1",
"workspace_url": "https://my-databricks-workspace.com",
"access_token": "dapi12345..."}}'
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/create
from databricks_registry_webhooks import RegistryWebhooksClient, JobSpec
job_spec = JobSpec(
job_id="1",
workspace_url="https://my-databricks-workspace.com",
access_token="dapi12345..."
)
job_webhook = RegistryWebhooksClient().create_webhook(
model_name="<model-name>",
events=["TRANSITION_REQUEST_CREATED"],
job_spec=job_spec,
description="Job webhook trigger",
status="TEST_MODE"
)
Response
{"webhook": {
"id":"1234567891",
"creation_timestamp":1591440826026,
"last_updated_timestamp":1591440826026,
"status":"TEST_MODE",
"events":["TRANSITION_REQUEST_CREATED"],
"job_spec": {
"job_id": "1",
"workspace_url": "https://my-databricks-workspace.com"
}}}
You can also create a job registry webhook with the Databricks Terraform provider and databricks_mlflow_webhook.
List registry webhooks example
$ curl -X GET -H "Authorization: Bearer <access-token>" -d \ '{"model_name": "<model-name>"}'
https://<databricks-instance>/api/2.0/mlflow/registry-webhooks/list
from databricks_registry_webhooks import RegistryWebhooksClient
webhooks_list = RegistryWebhooksClient().list_webhooks(model_name="<model-name>")
Response
{"webhooks": [{
"id":"1234567890",
"creation_timestamp":1571440826026,
"last_updated_timestamp":1582768296651,
"status": "ACTIVE",
"events":["MODEL_VERSION_CREATED"],
"http_url_spec": {
"url": "https://hooks.slack.com/services/...",
"enable_ssl_verification": True
}},
{
"id":"1234567891",
"creation_timestamp":1591440826026,
"last_updated_timestamp":1591440826026,
"status":"TEST_MODE",
"events":["TRANSITION_REQUEST_CREATED"],
"job_spec": {
"job_id": "1",
"workspace_url": "https://my-databricks-workspace.com"
}}]}