Diagnostic logging in Azure Databricks

  • Article
  • 16 minutes to read

Azure Databricks provides comprehensive end-to-end diagnostic logs of activities performed by Azure Databricks users, allowing your enterprise to monitor detailed Azure Databricks usage patterns.

For a list of each of these types of events and the associated services, see Events. Some of the events are emitted in audit logs only if verbose audit logs are enabled for the workspace.

Configure verbose audit logs

In addition to the default events, you can configure a workspace to generate additional events by enabling verbose audit logs.

Additional notebook actions

Additional actions in audit log category notebook:

  • Action name runCommand, emitted after Azure Databricks runs a command in a notebook. A command is corresponds to a cell in a notebook.

    Request parameters:

    • notebookId: Notebook ID
    • executionTime: The duration of the command in seconds. This is a decimal value such as 13.789.
    • status: Status of the command. Possible values are finished (the command finished), skipped (the command was skipped), cancelled (the command was cancelled), or failed (the command failed).

  • commandId: The unique ID for this command.

  • commandText: The text of the command. For multi-line commands, lines are separated by newline characters.

Additional Databricks SQL actions

Additional actions in audit log category databrickssql:

  • Action name commandSubmit, which runs when a command is submitted to Databricks SQL.

    Request parameters:

    • commandText: User-specified SQL statement or command.
    • warehouseId: ID for the SQL warehouse.
    • commandId: ID of the command.

  • Action name commandFinish, which runs when a command completes or a command is cancelled.

    Request parameters:

    • warehouseId: ID for the SQL warehouse.
    • commandId: ID of the command.

    Check the response field for additional information related to the command result:

    • statusCode - The HTTP response code. This will be error 400 if it is a general error.

    • errorMessage - Error message.

      Note

      In some cases for certain long-running commands, the errorMessage field may not be populated on failure.

    • result: This field is empty.

Enable or disable verbose audit logs

  1. As an admin, go to the Azure Databricks admin console.
  2. Click Workspace settings.
  3. Next to Verbose Audit Logs, enable or disable the feature.

When you enable or disable verbose logging, an auditable event is emitted in the category workspace with action workspaceConfKeys. The workspaceConfKeys request parameter is enableVerboseAuditLogs. The request parameter workspaceConfValues is true (feature enabled) or false (feature disabled).

Configure diagnostic log delivery

Note

Diagnostic logs require the Premium Plan.

  1. Log in to the Azure portal as an Owner or Contributor for the Azure Databricks workspace and click your Azure Databricks Service resource.

  2. In the Monitoring section of the sidebar, click the Diagnostic settings tab.

  3. Click Turn on diagnostics.

    Azure Databricks turn on diagnostics

  4. On the Diagnostic settings page, provide the following configuration:

    Name

    Enter a name for the logs to create.

    Archive to a storage account

    To use this option, you need an existing storage account to connect to. To create a new storage account in the portal, see Create a storage account and follow the instructions to create an Azure Resource Manager, general-purpose account. Then return to this page in the portal to select your storage account. It might take a few minutes for newly created storage accounts to appear in the drop-down menu. For information about additional costs incurred by writing to a storage account, see Azure Storage pricing.

    Stream to an event hub

    To use this option, you need an existing Azure Event Hubs namespace and event hub to connect to. To create an Event Hubs namespace, see Create an Event Hubs namespace and an event hub by using the Azure portal. Then return to this page in the portal to select the Event Hubs namespace and policy name. For information about additional costs incurred by writing to an event hub, see Azure Event Hubs pricing.

    Send to Log Analytics

    To use this option, either use an existing Log Analytics workspace or create a new one by following the steps to Create a new workspace in the portal. For information about additional costs incurred by sending logs to Log Analytics, see Azure Monitor pricing.

    Azure Databricks Diagnostics settings

  5. Choose the services you want diagnostic logs for and set retention policies.

    Retention applies only to storage accounts. If you do not want to apply a retention policy and you want to retain data forever, set Retention (days) to 0.

  6. Select Save.

  7. If you receive an error that says “Failed to update diagnostics for . The subscription is not registered to use microsoft.insights,” follow the Troubleshoot Azure Diagnostics instructions to register the account and then retry this procedure.

  8. If you want to change how your diagnostic logs are saved at any point in the future, return to this page to modify the diagnostic log settings for your account.

Enable logging using PowerShell

  1. Start an Azure PowerShell session and sign in to your Azure account with the following command:

     Connect-AzAccount

    If you do not have Azure Powershell installed already, use the following commands to install Azure PowerShell and import the Azure RM module.

     Install-Module -Name Az -AllowClobber
 Import-Module AzureRM

  2. In the pop-up browser window, enter your Azure account user name and password. Azure PowerShell gets all of the subscriptions that are associated with this account, and by default, uses the first one.

    If you have more than one subscription, you might have to specify the specific subscription that was used to create your Azure Key Vault. To see the subscriptions for your account, type the following command:

    Get-AzSubscription

    To specify the subscription that’s associated with the Azure Databricks account that you’re logging, type the following command:

    Set-AzContext -SubscriptionId <subscription ID>

  3. Set your Log Analytics resource name to a variable named logAnalytics, where ResourceName is the name of the Log Analytics workspace.

    $logAnalytics = Get-AzResource -ResourceGroupName <resource group name> -ResourceName <resource name> -ResourceType "Microsoft.OperationalInsights/workspaces"

  4. Set the Azure Databricks service resource name to a variable named databricks, where ResourceName is the name of the Azure Databricks service.

    $databricks = Get-AzResource -ResourceGroupName <your resource group name> -ResourceName <your Azure Databricks service name> -ResourceType "Microsoft.Databricks/workspaces"

  5. To enable logging for Azure Databricks, use the New-AzDiagnosticSetting cmdlet with variables for the new storage account, Azure Databricks service, and the category to enable for logging. Run the following command and set the -Enabled flag to $true:

    New-AzDiagnosticSetting -ResourceId $databricks.ResourceId -WorkspaceId $logAnalytics.ResourceId -Enabled $true -name "<diagnostic setting name>" -Category <comma separated list>

Enable logging by using Azure CLI

  1. Open PowerShell.

  2. Use the following command to connect to your Azure account:

    az login

  3. Run the following diagnostic setting command:

    az monitor diagnostic-settings create --name <diagnostic name>
--resource-group <log analytics workspace resource group>
--workspace <log analytics name or object ID>
--resource <target resource object ID>
--logs '[
{
 \"category\": <category name>,
  \"enabled\": true
}
]'

REST API

Use the LogSettings API.

Request

PUT https://management.azure.com/{resourceUri}/providers/microsoft.insights/diagnosticSettings/{name}?api-version=2017-05-01-preview

Request body

{
    "properties": {
    "workspaceId": "<log analytics resourceId>",
    "logs": [
      {
        "category": "<category name>",
        "enabled": true,
        "retentionPolicy": {
          "enabled": false,
          "days": 0
        }
      }
    ]
  }
}

Diagnostic log delivery

Once logging is enabled for your account, Azure Databricks automatically starts sending diagnostic logs to your delivery location. Logs are available within 15 minutes of activation. Azure Databricks auditable events typically appear in diagnostic logs within 15 minutes in Azure Commercial regions.

Note

SSH login logs are delivered with high latency.

Diagnostic log schema

The schema of diagnostic log records is as follows:

Field Description
operationversion The schema version of the diagnostic log format.
time UTC timestamp of the action.
properties.sourceIPAddress The IP address of the source request.
properties.userAgent The browser or API client used to make the request.
properties.sessionId Session ID of the action.
identities Information about the user that makes the requests:

* email: User email address.
category The service that logged the request.
operationName The action, such as login, logout, read, or write. Formatted as Microsoft.Databricks/<category>/<actionName>, for example Microsoft.Databricks/jobs/create.
properties.requestId Unique request ID.
properties.requestParams Parameter key-value pairs used in the event.

properties.response Response to the request:

* errorMessage: The error message if there was an error.
* result: The result of the request.
* statusCode: HTTP status code that indicates whether the request succeeds or not.
properties.logId The unique identifier for the log messages.

Events

The category (the Azure Databricks service) and operationName properties identify an event in a log record. Azure Databricks provides diagnostic logs for the following categories (services):

accounts Events related to accounts, users, groups, and IP access lists.
clusters Events related to clusters.
dbfs Events related to DBFS.
deltaPipelines Events related to Delta Live Table pipelines.
featureStore Events related to the Databricks Feature Store.
genie Events related to workspace access by support personnel.
gitCredentials Events related to Git credentials for Databricks Repos. Also see repos.
globalInitScripts Events related to global init scripts.
instancePools Events related to pools.
jobs Events related to jobs.
mlflowAcledArtifact Events related to ML Flow artifacts with ACLs.
mlflowExperiment Events related to ML Flow experiments.
modelRegistry Events related to the model registry.
notebook Events related to notebooks.
repos Events related to Databricks Repos. Also see gitCredentials.
secrets Events related to secrets.
ssh Events related to SSH access, which is disabled by default.
unityCatalog Events related to Unity Catalog, which is disabled by default.
webTerminal Events related to web terminal.
workspace Events related to workspaces.

If actions take a long time, the request and response are logged separately, but the request and response pair have the same properties.requestId.

With the exception of mount-related operations, Azure Databricks diagnostic logs do not include DBFS-related operations.

Automated actions are performed by the user System-User. For example, when Azure Databricks resizes a cluster through the autoscaling feature or launches a job due to job scheduling.

The following table lists the available actions for each category. The action and category help form the operationName field for each log entry. That field is formatted as Microsoft.Databricks/<category>/<actionName>, such as Microsoft.Databricks/jobs/create.

Category Action name Request parameters
accounts IpAccessDenied path, userId
aadBrowserLogin user
aadTokenLogin user
activateUser warehouse, targetUserId, targetUserName
add warehouse, targetUserId, targetUserName
addPrincipalToGroup warehouse, targetGroupId, targetGroupName, targetUserId, targetUserName
changeDatabricksSqlAcl aclPermissionSet, resourceId, shardName, targetUserId
changeDatabricksWorkspaceAcl aclPermissionSet, resourceId, shardName, targetUserId
changeDbTokenAcl aclPermissionSet, resourceId, shardName, targetUserId
createGroup warehouse, targetGroupId, targetGroupName
createIpAccessList ipAccessListId, userId
deactivateUser warehouse, targetUserId, targetUserName
delete warehouse, targetUserId, targetUserName
deleteIpAccessList ipAccessListId, userId
disableClusterAcls warehouse
disableTableAcls warehouse
disableWorkspaceAcls warehouse
enableClusterAcls warehouse
enableTableAcls warehouse
enableWorkspaceAcls warehouse
garbageCollectDbToken tokenClientId, tokenCreationTime, tokenExpirationTime, tokenFirstAccessed, userId
generateDbToken tokenCreatedBy, tokenExpirationTime, userId
ipAccessListQuotaExceeded userId
jwtLogin user
login user
logout user
privateLinkValidationFailed userId
reachMaxQuotaDbToken userId
removeAdmin warehouse, targetUserId, targetUserName
removeGroup warehouse, targetGroupId, targetGroupName
removePrincipalFromGroup warehouse, targetGroupId, targetGroupName, targetUserId, targetUserName
revokeDbToken userId
setAdmin warehouse, targetUserId, targetUserName
tokenLogin tokenId, user
updateIpAccessList ipAccessListId, userId
updateUser warehouse, targetUserId, targetUserName
databrickssql addDashboardWidget dashboardId, widgetId
cancelQueryExecution queryExecutionId
changeWarehouseAcls aclPermissionSet, resourceId, shardName, targetUserId
changePermissions granteeAndPermission, objectId, objectType
cloneDashboard dashboardId
commandSubmit (only for verbose audit logs) [“orgId”, “sourceIpAddress”, “timestamp”, “userAgent”,”userIdentity”, “shardName” (see details)]
commandFinish (only for verbose audit logs) [“orgId”, “sourceIpAddress”, “timestamp”, “userAgent”,”userIdentity”, “shardName” (see details)]
createAlertDestination alertDestinationId, alertDestinationType
createDashboard dashboardId
createDataPreviewDashboard dashboardId
createWarehouse auto_resume, auto_stop_mins, channel, cluster_size, conf_pairs, custom_cluster_confs, enable_databricks_compute, enable_photon, enable_serverless_compute, instance_profile_arn, max_num_clusters, min_num_clusters, name, size, spot_instance_policy, tags, test_overrides
createQuery queryId
createQueryDraft queryId
createQuerySnippet querySnippetId
createRefreshSchedule alertId, dashboardId, refreshScheduleId
createSampleDashboard sampleDashboardId
createSubscription dashboardId, refreshScheduleId, subscriptionId
createVisualization queryId, visualizationId
deleteAlert alertId
deleteAlertDestination alertDestinationId
deleteDashboard dashboardId
deleteDashboardWidget widgetId
deleteWarehouse id
deleteExternalDatasource dataSourceId
deleteQuery queryId
deleteQueryDraft queryId
deleteQuerySnippet querySnippetId
deleteRefreshSchedule alertId, dashboardId, refreshScheduleId
deleteSubscription subscriptionId
deleteVisualization visualizationId
downloadQueryResult fileType, queryId, queryResultId
editWarehouse auto_stop_mins, channel, cluster_size, confs, enable_photon, enable_serverless_compute, id, instance_profile_arn, max_num_clusters, min_num_clusters, name, spot_instance_policy, tags
executeAdhocQuery dataSourceId
executeSavedQuery queryId
executeWidgetQuery widgetId
favoriteDashboard dashboardId
favoriteQuery queryId
forkQuery originalQueryId, queryId
listQueries filter_by, include_metrics, max_results, page_token
moveDashboardToTrash dashboardId
moveQueryToTrash queryId
muteAlert alertId
publishBatch statuses
publishDashboardSnapshot dashboardId, hookId, subscriptionId
restoreDashboard dashboardId
restoreQuery queryId
setWarehouseConfig data_access_config, enable_serverless_compute, instance_profile_arn, security_policy, serverless_agreement, sql_configuration_parameters, try_create_databricks_managed_starter_warehouse
snapshotDashboard dashboardId
startWarehouse id
stopWarehouse id
subscribeAlert alertId, destinationId
transferObjectOwnership newOwner, objectId, objectType
unfavoriteDashboard dashboardId
unfavoriteQuery queryId
unmuteAlert alertId
unsubscribeAlert alertId, subscriberId
updateAlert alertId, queryId
updateAlertDestination alertDestinationId
updateDashboard dashboardId
updateDashboardWidget widgetId
updateOrganizationSetting has_configured_data_access, has_explored_sql_warehouses, has_granted_permissions
updateQuery queryId
updateQueryDraft queryId
updateQuerySnippet querySnippetId
updateRefreshSchedule alertId, dashboardId, refreshScheduleId
updateVisualization visualizationId
clusters changeClusterAcl aclPermissionSet, resourceId, shardName, targetUserId
changeClusterPolicyAcl aclPermissionSet, resourceId, shardName, targetUserId
create acl_path_prefix, apply_policy_default_values, autoscale, autotermination_minutes, azure_attributes, billing_info, cluster_creator, cluster_event_notification_info, cluster_log_conf, cluster_name, cluster_source, custom_tags, data_security_mode, disk_spec, docker_image, driver_instance_pool_id, driver_instance_source, driver_node_type_id, enable_elastic_disk, enable_jobs_autostart, enable_local_disk_encryption, idempotency_token, init_scripts, instance_pool_id, instance_source, no_driver_daemon, node_type_id, num_workers, organization_id, policy_id, single_user_name, spark_conf, spark_env_vars, spark_version, ssh_public_keys, start_cluster, user_id, validate_cluster_name_uniqueness
createResult clusterId, clusterName, clusterOwnerUserId, clusterState, clusterWorkers
delete cluster_id, termination_reason
deleteResult clusterId, clusterName, clusterOwnerUserId, clusterState, clusterWorkers
edit apply_policy_default_values, autoscale, autotermination_minutes, azure_attributes, cluster_id, cluster_log_conf, cluster_name, cluster_source, custom_tags, data_security_mode, disk_spec, docker_image, driver_instance_pool_id, driver_instance_source, driver_node_type_id, enable_elastic_disk, enable_local_disk_encryption, init_scripts, instance_pool_id, instance_source, no_driver_daemon, node_type_id, num_workers, policy_id, single_user_name, spark_conf, spark_env_vars, spark_version, ssh_public_keys, validate_cluster_name_uniqueness, workload_type
permanentDelete cluster_id
resize autoscale, avoid_containers, cause, cluster_id, next_attempt_time_ms, num_workers
resizeResult clusterId, clusterName, clusterOwnerUserId, clusterState, clusterWorkers
restart cluster_id
restartResult clusterId, clusterName, clusterOwnerUserId, clusterState, clusterWorkers
start cluster_id
startResult clusterId, clusterName, clusterOwnerUserId, clusterState, clusterWorkers
dbfs addBlock data_length, handle
close handle
create bufferSize, overwrite, path
delete path, recursive
getSessionCredentials mountPoint
mkdirs path
mount mountPoint, owner
move destination_path, dst, source_path, src
put overwrite, path
unmount mountPoint
updateMount mountPoint, owner
deltaPipelines create allow_duplicate_names, clusters, configuration, continuous, development, dry_run, id, libraries, name, storage, target, channel, edition, photon, dbr_version (internal, do not use), email_notifications (internal, do not use), filters (deprecated),
delete pipeline_id
edit allow_duplicate_names, clusters, configuration, continuous, dbr_version (internal, do not use), development, email_notifications (internal, do not use), expected_last_modified, filters (deprecated), id, libraries, name, pipeline_id, storage, target, channel, edition, photon
startUpdate cause, full_refresh, job_task, pipeline_id
stop pipeline_id
featureStore addConsumer features, job_run, notebook
addDataSources feature_table, paths, tables
addProducer feature_table, job_run, notebook
changeFeatureTableAcl aclPermissionSet, resourceId, shardName, targetUserId
createFeatureTable description, name, partition_keys, primary_keys, timestamp_keys
createFeatures feature_table, features
deleteFeatureTable name
deleteTags feature_table_id, keys
getConsumers feature_table
getFeatureTable name
getFeatureTablesById ids
getFeatures feature_table, max_results
getModelServingMetadata feature_table_features
getOnlineStore cloud, feature_table, online_table, store_type
getTags feature_table_id
publishFeatureTable cloud, feature_table, host, online_table, port, read_secret_prefix, store_type, write_secret_prefix
searchFeatureTables max_results, page_token, text
setTags feature_table_id, tags
updateFeatureTable description, name
genie databricksAccess approver, authType, duration, isCriticalUser, reason, user
gitCredentials getGitCredential id
listGitCredentials none
deleteGitCredential id
updateGitCredential id, git_provider, git_username
createGitCredential git_provider, git_username]
globalInitScripts batch-reorder script_ids
create enabled, name, position, script-SHA256
delete script_id
update enabled, name, position, script-SHA256, script_id
instancePools changeInstancePoolAcl aclPermissionSet, resourceId, shardName, targetUserId
create azure_attributes, custom_tags, disk_spec, enable_elastic_disk, idle_instance_autotermination_minutes, instance_pool_name, max_capacity, min_idle_instances, node_type_id, preloaded_docker_images, preloaded_spark_versions
delete instance_pool_id
edit azure_attributes, custom_tags, disk_spec, enable_elastic_disk, idle_instance_autotermination_minutes, instance_pool_id, instance_pool_name, max_capacity, min_idle_instances, node_type_id, preloaded_spark_versions
jobs cancel run_id
cancelAllRuns job_id
changeJobAcl aclPermissionSet, resourceId, shardName, targetUserId
create access_control_list, dbt_task, email_notifications, existing_cluster_id, format, git_source, is_from_dlt, job_clusters, job_type, libraries, max_concurrent_runs, max_retries, min_retry_interval_millis, name, new_cluster, notebook_task, pipeline_task, python_wheel_task, retry_on_timeout, schedule, shell_command_task, spark_jar_task, spark_python_task, spark_submit_task, tasks, timeout_seconds
delete job_id
deleteRun run_id
repairRun latest_repair_id, rerun_tasks, run_id
reset is_from_dlt, job_id, new_settings
resetJobAcl grants, job_id
runFailed clusterId, idInJob, jobClusterType, jobId, jobTaskType, jobTerminalState, jobTriggerType, orgId, runId
runNow jar_params, job_id, notebook_params, python_params, spark_submit_params, workflow_context
runStart clusterId, idInJob, jobClusterType, jobId, jobTaskType, jobTerminalState, jobTriggerType, orgId, runId
runSucceeded clusterId, idInJob, jobClusterType, jobId, jobTaskType, jobTerminalState, jobTriggerType, orgId, runId
setTaskValue key, run_id
submitRun access_control_list, existing_cluster_id, idempotency_token, job_cluster_key, job_clusters, libraries, new_cluster, notebook_task, run_name, shell_command_task, spark_jar_task, spark_python_task, spark_submit_task, tasks, timeout_seconds, workflow_context
update fields_to_remove, job_id, new_settings
mlflowAcledArtifact readArtifact artifactLocation, experimentId, runId
writeArtifact artifactLocation, experimentId, runId
mlflowExperiment deleteMlflowExperiment experimentId, experimentName, path
moveMlflowExperiment experimentId, newPath, oldPath
renameMlflowExperimentEvent experimentId, newName, oldName, parentPath
restoreMlflowExperiment experimentId, experimentName, path
modelRegistry approveTransitionRequest archive_existing_versions, comment, name, stage, version
changeRegisteredModelAcl aclPermissionSet, resourceId, shardName, targetUserId
createComment comment, name, version
createModelVersion description, name, run_id, run_link, source, tags
createRegisteredModel description, name, tags
createRegistryWebhook description, events, http_url_spec, job_spec, model_name, status
createTransitionRequest comment, name, stage, version
deleteModelVersion name, version
deleteModelVersionTag key, name, version
deleteRegisteredModel name
deleteRegisteredModelTag key, name
deleteRegistryWebhook id
deleteTransitionRequest comment, creator, name, stage, version
finishCreateModelVersionAsync name, version
generateBatchInferenceNotebook input_data, name, output_path, stage, version
getModelVersionDownloadUri name, version
getModelVersionSignedDownloadUri name, path, version
listModelArtifacts name, path, version
listRegistryWebhooks max_results, model_name
rejectTransitionRequest comment, name, stage, version
renameRegisteredModel name, new_name
setEmailSubscriptionStatus model_name, subscription_type
setModelVersionTag key, name, value, version
setRegisteredModelTag key, name, value
setUserLevelEmailSubscriptionStatus subscription_type
testRegistryWebhook id
transitionModelVersionStage archive_existing_versions, comment, name, stage, version
updateRegistryWebhook description, events, http_url_spec, id, status
notebook attachNotebook clusterId, notebookId, path
cloneNotebook notebookId, path
createNotebook notebookId, path
deleteFolder path
deleteNotebook notebookId, notebookName, path
detachNotebook clusterId, notebookId, path
downloadLargeResults notebookFullPath, notebookId
downloadPreviewResults notebookFullPath, notebookId
importNotebook path, workspaceExportFormat
moveNotebook newPath, notebookId, oldPath
renameNotebook newName, notebookId, oldName, parentPath
restoreFolder path
restoreNotebook notebookId, notebookName, path
runCommand (only for verbose audit logs) notebookId, executionTime, status, commandId, commandText (see details)
takeNotebookSnapshot path
repos checkoutBranch branch, id
commitAndPush checkSensitiveToken, files, id, message
createRepo git_provider, git_url, path, provider, url
deleteRepo id
discard file_paths, id
getRepo id
listRepos next_page_token, path_prefix
pull id
updateRepo branch, git_provider, git_url, id, path, tag, workspace_filesystem_enabled
secrets createScope backend_azure_keyvault, initial_manage_principal, is_databricks_managed, scope, scope_backend_type
deleteAcl principal, scope
deleteScope scope
deleteSecret key, scope
getAcl principal, scope
getSecret key, scope
listAcls scope
listSecrets scope
putAcl permission, principal, scope
putSecret key, scope, string_value
sqlPermissions changeSecurableOwner principal, securable
createSecurable securable
denyPermission permission
grantPermission permission
removeAllPermissions securable
renameSecurable after, before
requestPermissions denied, permitted, requests, truncated
revokePermission permission
showPermissions principal, securable
ssh login containerId, instanceId, port, publicKey, userName
logout containerId, instanceId, userName
unityCatalog createCatalog comment, metastore_id, name, workspace_id
createDataAccessConfiguration metastore_id, name, workspace_id
createExternalLocation comment, credential_name, metastore_id, name, url, workspace_id
createMetastore metastore_id, name, storage_root, workspace_id
createMetastoreAssignment default_catalog_name, metastore_id, workspace_id
createRecipient comment, metastore_id, name, workspace_id
createSchema catalog_name, comment, metastore_id, name, workspace_id
createShare comment, metastore_id, name, workspace_id
createStagingTable catalog_name, metastore_id, name, schema_name, workspace_id
createStorageCredential azure_service_principal, comment, metastore_id, name, workspace_id
createTable catalog_name, columns, data_source_format, metastore_id, name, schema_name, storage_location, table_type, view_definition, workspace_id
deleteRecipient metastore_id, name, workspace_id
deleteSchema full_name_arg, metastore_id, workspace_id
deleteShare metastore_id, name, workspace_id
deleteTable full_name_arg, metastore_id, workspace_id
deltaSharingGetTableMetadata metastore_id, name, recipient_name, schema, share, user_agent, workspace_id
deltaSharingListAllTables metastore_id, options, recipient_name, share, user_agent, workspace_id
deltaSharingListSchemas metastore_id, options, recipient_name, share, user_agent, workspace_id
deltaSharingListShares metastore_id, options, recipient_name, user_agent, workspace_id
deltaSharingListTables metastore_id, options, recipient_name, schema, share, user_agent, workspace_id
deltaSharingQueriedTable metastore_id, recipient_name
deltaSharingQueryTable limitHint, metastore_id, name, recipient_name, schema, share, user_agent, workspace_id
generateTemporaryPathCredential credential_id, credential_type, ensure_url_exists, metastore_id, operation, url, workspace_id
generateTemporaryTableCredential credential_id, credential_type, is_permissions_enforcing_client, metastore_id, operation, table_full_name, table_id, workspace_id
getActivationUrlInfo metastore_id, recipient_name, workspace_id
getCatalog metastore_id, name_arg, workspace_id
getDataAccessConfiguration id, metastore_id, workspace_id
getExternalLocation metastore_id, name_arg, workspace_id
getMetastore id, metastore_id, workspace_id
getMetastoreSummary metastore_id, workspace_id
getPermissions metastore_id, principal, securable_full_name, securable_type, workspace_id
getRecipient metastore_id, name, workspace_id
getRecipientSharePermissions metastore_id, name, workspace_id
getSchema full_name_arg, metastore_id, workspace_id
getShare include_shared_data, metastore_id, name, workspace_id
getSharePermissions metastore_id, name, workspace_id
getStorageCredential metastore_id, name_arg, workspace_id
getTable full_name_arg, metastore_id, workspace_id
listCatalogs metastore_id, workspace_id
listDataAccessConfigurations metastore_id, workspace_id
listExternalLocations credential_name, metastore_id, url, workspace_id
listMetastores workspace_id
listRecipients metastore_id, workspace_id
listSchemas catalog_name, metastore_id, workspace_id
listShares metastore_id, workspace_id
listStorageCredentials metastore_id, workspace_id
listTableSummaries catalog_name, metastore_id, schema_name_pattern, table_name_pattern, workspace_id
listTables catalog_name, metastore_id, schema_name, workspace_id
metadataAndPermissionsSnapshot metastore_id, securables, workspace_id
metadataSnapshot metastore_id, securables, workspace_id
privilegedGetAllPermissions metastore_id, securables, workspace_id
privilegedGetTable full_name_arg, metastore_id, workspace_id
retrieveRecipientToken metastore_id, recipient_name, workspace_id
updateMetastore default_data_access_config_id, delta_sharing_enabled, delta_sharing_recipient_token_lifetime_in_seconds, id, metastore_id, name, owner, storage_root_credential_id, workspace_id
updateMetastoreAssignment default_catalog_name, metastore_id, workspace_id
updatePermissions changes, metastore_id, securable_full_name, securable_type, workspace_id
updateSchema full_name_arg, metastore_id, owner, workspace_id
updateShare metastore_id, name, updates, workspace_id
updateSharePermissions changes, metastore_id, name, workspace_id
updateTables columns, data_source_format, full_name_arg, metastore_id, storage_location, table_type, workspace_id
webTerminal startSession socketGUID, clusterId, serverPort, ProxyTargetURI
closeSession socketGUID, clusterId, serverPort, ProxyTargetURI
workspace changeWorkspaceAcl aclPermissionSet, resourceId, shardName, targetUserId
fileCreate path
fileDelete path
purgeClusterLogs logFilePath
purgeRevisionHistoryBefore property, propertyValue, treestoreId
purgeWorkspaceNodes treestoreId
workspaceConfEdit (workspace-level setting changes) workspaceConfKeys (for example, verbose audit logs uses value enableVerboseAuditLogs), workspaceConfValues (for example, for verbose audit logs this is true or false)
workspaceExport notebookFullPath, workspaceExportDirectDownload, workspaceExportFormat

Sample log output

The following JSON sample is an example of Azure Databricks log output:

{
    "TenantId": "<your-tenant-id>",
    "SourceSystem": "|Databricks|",
    "TimeGenerated": "2019-05-01T00:18:58Z",
    "ResourceId": "/SUBSCRIPTIONS/SUBSCRIPTION_ID/RESOURCEGROUPS/RESOURCE_GROUP/PROVIDERS/MICROSOFT.DATABRICKS/WORKSPACES/PAID-VNET-ADB-PORTAL",
    "OperationName": "Microsoft.Databricks/jobs/create",
    "OperationVersion": "1.0.0",
    "Category": "jobs",
    "Identity": {
        "email": "mail@contoso.com",
        "subjectName": null
    },
    "SourceIPAddress": "131.0.0.0",
    "LogId": "201b6d83-396a-4f3c-9dee-65c971ddeb2b",
    "ServiceName": "jobs",
    "UserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36",
    "SessionId": "webapp-cons-webapp-01exaj6u94682b1an89u7g166c",
    "ActionName": "create",
    "RequestId": "ServiceMain-206b2474f0620002",
    "Response": {
        "statusCode": 200,
        "result": "{\"job_id\":1}"
    },
    "RequestParams": {
        "name": "Untitled",
        "new_cluster": "{\"node_type_id\":\"Standard_DS3_v2\",\"spark_version\":\"5.2.x-scala2.11\",\"num_workers\":8,\"spark_conf\":{\"spark.databricks.delta.preview.enabled\":\"true\"},\"cluster_creator\":\"JOB_LAUNCHER\",\"spark_env_vars\":{\"PYSPARK_PYTHON\":\"/databricks/python3/bin/python3\"},\"enable_elastic_disk\":true}"
    },
    "Type": "DatabricksJobs"
}

Analyze diagnostic logs

If you selected the Send to Log Analytics option when you turned on diagnostic logging, diagnostic data from your container is typically forwarded to Azure Monitor logs within 15 minutes.

Before you view your logs, verify if your Log Analytics workspace has been upgraded to use the new Kusto query language. To check, open the Azure portal and select Log Analytics on the far left. Then select your Log Analytics workspace. If you get a message to upgrade, see Upgrade your Azure Log Analytics workspace to new log search.

To view your diagnostic data in Azure Monitor logs, open the Log Search page from the left menu or the Management area of the page. Then enter your query into the Log search box.

Azure Log Analytics

Queries

Here are some additional queries that you can enter into the Log search box. These queries are written in Kusto Query Language.

  • To query all users who have accessed the Azure Databricks workspace and their location:

    DatabricksAccounts
| where ActionName contains "login"
| extend d=parse_json(Identity)
| project UserEmail=d.email, SourceIPAddress

  • To check the Spark versions used:

    DatabricksClusters
| where ActionName == "create"
| extend d=parse_json(RequestParams)
| extend SparkVersion= d.spark_version
| summarize Count=count() by tostring(SparkVersion)