Diagnostic logging in Azure Databricks
Azure Databricks provides comprehensive end-to-end diagnostic logs of activities performed by Azure Databricks users, allowing your enterprise to monitor detailed Azure Databricks usage patterns.
For a list of each of these types of events and the associated services, see Events. Some of the events are emitted in audit logs only if verbose audit logs are enabled for the workspace.
Configure verbose audit logs
In addition to the default events, you can configure a workspace to generate additional events by enabling verbose audit logs.
Additional notebook actions
Additional actions in audit log category notebook
:
Action name
runCommand
, emitted after Azure Databricks runs a command in a notebook. A command is corresponds to a cell in a notebook.Request parameters:
notebookId
: Notebook IDexecutionTime
: The duration of the command in seconds. This is a decimal value such as13.789
.status
: Status of the command. Possible values arefinished
(the command finished),skipped
(the command was skipped),cancelled
(the command was cancelled), orfailed
(the command failed).
commandId
: The unique ID for this command.commandText
: The text of the command. For multi-line commands, lines are separated by newline characters.
Additional Databricks SQL actions
Additional actions in audit log category databrickssql
:
Action name
commandSubmit
, which runs when a command is submitted to Databricks SQL.Request parameters:
commandText
: User-specified SQL statement or command.warehouseId
: ID for the SQL warehouse.commandId
: ID of the command.
Action name
commandFinish
, which runs when a command completes or a command is cancelled.Request parameters:
warehouseId
: ID for the SQL warehouse.commandId
: ID of the command.
Check the
response
field for additional information related to the command result:statusCode
- The HTTP response code. This will be error 400 if it is a general error.errorMessage
- Error message.Note
In some cases for certain long-running commands, the
errorMessage
field may not be populated on failure.result
: This field is empty.
Enable or disable verbose audit logs
- As an admin, go to the Azure Databricks admin console.
- Click Workspace settings.
- Next to Verbose Audit Logs, enable or disable the feature.
When you enable or disable verbose logging, an auditable event is emitted in the category workspace
with action workspaceConfKeys
. The workspaceConfKeys
request parameter is enableVerboseAuditLogs
. The request parameter workspaceConfValues
is true
(feature enabled) or false
(feature disabled).
Configure diagnostic log delivery
Note
Diagnostic logs require the Premium Plan.
Log in to the Azure portal as an Owner or Contributor for the Azure Databricks workspace and click your Azure Databricks Service resource.
In the Monitoring section of the sidebar, click the Diagnostic settings tab.
Click Turn on diagnostics.
On the Diagnostic settings page, provide the following configuration:
Name
Enter a name for the logs to create.
Archive to a storage account
To use this option, you need an existing storage account to connect to. To create a new storage account in the portal, see Create a storage account and follow the instructions to create an Azure Resource Manager, general-purpose account. Then return to this page in the portal to select your storage account. It might take a few minutes for newly created storage accounts to appear in the drop-down menu. For information about additional costs incurred by writing to a storage account, see Azure Storage pricing.
Stream to an event hub
To use this option, you need an existing Azure Event Hubs namespace and event hub to connect to. To create an Event Hubs namespace, see Create an Event Hubs namespace and an event hub by using the Azure portal. Then return to this page in the portal to select the Event Hubs namespace and policy name. For information about additional costs incurred by writing to an event hub, see Azure Event Hubs pricing.
Send to Log Analytics
To use this option, either use an existing Log Analytics workspace or create a new one by following the steps to Create a new workspace in the portal. For information about additional costs incurred by sending logs to Log Analytics, see Azure Monitor pricing.
Choose the services you want diagnostic logs for and set retention policies.
Retention applies only to storage accounts. If you do not want to apply a retention policy and you want to retain data forever, set Retention (days) to 0.
Select Save.
If you receive an error that says “Failed to update diagnostics for . The subscription is not registered to use microsoft.insights,” follow the Troubleshoot Azure Diagnostics instructions to register the account and then retry this procedure.
If you want to change how your diagnostic logs are saved at any point in the future, return to this page to modify the diagnostic log settings for your account.
Enable logging using PowerShell
Start an Azure PowerShell session and sign in to your Azure account with the following command:
Connect-AzAccount
If you do not have Azure Powershell installed already, use the following commands to install Azure PowerShell and import the Azure RM module.
Install-Module -Name Az -AllowClobber Import-Module AzureRM
In the pop-up browser window, enter your Azure account user name and password. Azure PowerShell gets all of the subscriptions that are associated with this account, and by default, uses the first one.
If you have more than one subscription, you might have to specify the specific subscription that was used to create your Azure Key Vault. To see the subscriptions for your account, type the following command:
Get-AzSubscription
To specify the subscription that’s associated with the Azure Databricks account that you’re logging, type the following command:
Set-AzContext -SubscriptionId <subscription ID>
Set your Log Analytics resource name to a variable named
logAnalytics
, whereResourceName
is the name of the Log Analytics workspace.$logAnalytics = Get-AzResource -ResourceGroupName <resource group name> -ResourceName <resource name> -ResourceType "Microsoft.OperationalInsights/workspaces"
Set the Azure Databricks service resource name to a variable named
databricks
, whereResourceName
is the name of the Azure Databricks service.$databricks = Get-AzResource -ResourceGroupName <your resource group name> -ResourceName <your Azure Databricks service name> -ResourceType "Microsoft.Databricks/workspaces"
To enable logging for Azure Databricks, use the New-AzDiagnosticSetting cmdlet with variables for the new storage account, Azure Databricks service, and the category to enable for logging. Run the following command and set the
-Enabled
flag to$true
:New-AzDiagnosticSetting -ResourceId $databricks.ResourceId -WorkspaceId $logAnalytics.ResourceId -Enabled $true -name "<diagnostic setting name>" -Category <comma separated list>
Enable logging by using Azure CLI
Open PowerShell.
Use the following command to connect to your Azure account:
az login
Run the following diagnostic setting command:
az monitor diagnostic-settings create --name <diagnostic name> --resource-group <log analytics workspace resource group> --workspace <log analytics name or object ID> --resource <target resource object ID> --logs '[ { \"category\": <category name>, \"enabled\": true } ]'
REST API
Use the LogSettings API.
Request
PUT https://management.azure.com/{resourceUri}/providers/microsoft.insights/diagnosticSettings/{name}?api-version=2017-05-01-preview
Request body
{
"properties": {
"workspaceId": "<log analytics resourceId>",
"logs": [
{
"category": "<category name>",
"enabled": true,
"retentionPolicy": {
"enabled": false,
"days": 0
}
}
]
}
}
Diagnostic log delivery
Once logging is enabled for your account, Azure Databricks automatically starts sending diagnostic logs to your delivery location. Logs are available within 15 minutes of activation. Azure Databricks auditable events typically appear in diagnostic logs within 15 minutes in Azure Commercial regions.
Note
SSH login logs are delivered with high latency.
Diagnostic log schema
The schema of diagnostic log records is as follows:
Field | Description | |
---|---|---|
operationversion |
The schema version of the diagnostic log format. | |
time |
UTC timestamp of the action. | |
properties.sourceIPAddress |
The IP address of the source request. | |
properties.userAgent |
The browser or API client used to make the request. | |
properties.sessionId |
Session ID of the action. | |
identities |
Information about the user that makes the requests: * email : User email address. |
|
category |
The service that logged the request. | |
operationName |
The action, such as login, logout, read, or write. Formatted as Microsoft.Databricks/<category>/<actionName> , for example Microsoft.Databricks/jobs/create . |
|
properties.requestId |
Unique request ID. | |
properties.requestParams |
Parameter key-value pairs used in the event. |
|
properties.response |
Response to the request: * errorMessage : The error message if there was an error.* result : The result of the request.* statusCode : HTTP status code that indicates whether the request succeeds or not. |
|
properties.logId |
The unique identifier for the log messages. |
Events
The category
(the Azure Databricks service) and operationName
properties identify an event in a log record. Azure Databricks provides diagnostic logs for the following categories (services):
accounts |
Events related to accounts, users, groups, and IP access lists. |
---|---|
clusters |
Events related to clusters. |
dbfs |
Events related to DBFS. |
deltaPipelines |
Events related to Delta Live Table pipelines. |
featureStore |
Events related to the Databricks Feature Store. |
genie |
Events related to workspace access by support personnel. |
gitCredentials |
Events related to Git credentials for Databricks Repos. Also see repos . |
globalInitScripts |
Events related to global init scripts. |
instancePools |
Events related to pools. |
jobs |
Events related to jobs. |
mlflowAcledArtifact |
Events related to ML Flow artifacts with ACLs. |
mlflowExperiment |
Events related to ML Flow experiments. |
modelRegistry |
Events related to the model registry. |
notebook |
Events related to notebooks. |
repos |
Events related to Databricks Repos. Also see gitCredentials . |
secrets |
Events related to secrets. |
ssh |
Events related to SSH access, which is disabled by default. |
unityCatalog |
Events related to Unity Catalog, which is disabled by default. |
webTerminal |
Events related to web terminal. |
workspace |
Events related to workspaces. |
If actions take a long time, the request and response are logged separately, but the request and response pair have the same properties.requestId
.
With the exception of mount-related operations, Azure Databricks diagnostic logs do not include DBFS-related operations.
Automated actions are performed by the user System-User. For example, when Azure Databricks resizes a cluster through the autoscaling feature or launches a job due to job scheduling.
The following table lists the available actions for each category. The action and category help form the operationName
field for each log entry. That field is formatted as Microsoft.Databricks/<category>/<actionName>
, such as Microsoft.Databricks/jobs/create
.
Category | Action name | Request parameters |
---|---|---|
accounts |
IpAccessDenied |
path , userId |
aadBrowserLogin |
user |
|
aadTokenLogin |
user |
|
activateUser |
warehouse , targetUserId , targetUserName |
|
add |
warehouse , targetUserId , targetUserName |
|
addPrincipalToGroup |
warehouse , targetGroupId , targetGroupName , targetUserId , targetUserName |
|
changeDatabricksSqlAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
|
changeDatabricksWorkspaceAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
|
changeDbTokenAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
|
createGroup |
warehouse , targetGroupId , targetGroupName |
|
createIpAccessList |
ipAccessListId , userId |
|
deactivateUser |
warehouse , targetUserId , targetUserName |
|
delete |
warehouse , targetUserId , targetUserName |
|
deleteIpAccessList |
ipAccessListId , userId |
|
disableClusterAcls |
warehouse |
|
disableTableAcls |
warehouse |
|
disableWorkspaceAcls |
warehouse |
|
enableClusterAcls |
warehouse |
|
enableTableAcls |
warehouse |
|
enableWorkspaceAcls |
warehouse |
|
garbageCollectDbToken |
tokenClientId , tokenCreationTime , tokenExpirationTime , tokenFirstAccessed , userId |
|
generateDbToken |
tokenCreatedBy , tokenExpirationTime , userId |
|
ipAccessListQuotaExceeded |
userId |
|
jwtLogin |
user |
|
login |
user |
|
logout |
user |
|
privateLinkValidationFailed |
userId |
|
reachMaxQuotaDbToken |
userId |
|
removeAdmin |
warehouse , targetUserId , targetUserName |
|
removeGroup |
warehouse , targetGroupId , targetGroupName |
|
removePrincipalFromGroup |
warehouse , targetGroupId , targetGroupName , targetUserId , targetUserName |
|
revokeDbToken |
userId |
|
setAdmin |
warehouse , targetUserId , targetUserName |
|
tokenLogin |
tokenId , user |
|
updateIpAccessList |
ipAccessListId , userId |
|
updateUser |
warehouse , targetUserId , targetUserName |
|
databrickssql |
addDashboardWidget |
dashboardId , widgetId |
cancelQueryExecution |
queryExecutionId |
|
changeWarehouseAcls |
aclPermissionSet , resourceId , shardName , targetUserId |
|
changePermissions |
granteeAndPermission , objectId , objectType |
|
cloneDashboard |
dashboardId |
|
commandSubmit (only for verbose audit logs) |
[“orgId”, “sourceIpAddress”, “timestamp”, “userAgent”,”userIdentity”, “shardName” (see details)] | |
commandFinish (only for verbose audit logs) |
[“orgId”, “sourceIpAddress”, “timestamp”, “userAgent”,”userIdentity”, “shardName” (see details)] | |
createAlertDestination |
alertDestinationId , alertDestinationType |
|
createDashboard |
dashboardId |
|
createDataPreviewDashboard |
dashboardId |
|
createWarehouse |
auto_resume , auto_stop_mins , channel , cluster_size , conf_pairs , custom_cluster_confs , enable_databricks_compute , enable_photon , enable_serverless_compute , instance_profile_arn , max_num_clusters , min_num_clusters , name , size , spot_instance_policy , tags , test_overrides |
|
createQuery |
queryId |
|
createQueryDraft |
queryId |
|
createQuerySnippet |
querySnippetId |
|
createRefreshSchedule |
alertId , dashboardId , refreshScheduleId |
|
createSampleDashboard |
sampleDashboardId |
|
createSubscription |
dashboardId , refreshScheduleId , subscriptionId |
|
createVisualization |
queryId , visualizationId |
|
deleteAlert |
alertId |
|
deleteAlertDestination |
alertDestinationId |
|
deleteDashboard |
dashboardId |
|
deleteDashboardWidget |
widgetId |
|
deleteWarehouse |
id |
|
deleteExternalDatasource |
dataSourceId |
|
deleteQuery |
queryId |
|
deleteQueryDraft |
queryId |
|
deleteQuerySnippet |
querySnippetId |
|
deleteRefreshSchedule |
alertId , dashboardId , refreshScheduleId |
|
deleteSubscription |
subscriptionId |
|
deleteVisualization |
visualizationId |
|
downloadQueryResult |
fileType , queryId , queryResultId |
|
editWarehouse |
auto_stop_mins , channel , cluster_size , confs , enable_photon , enable_serverless_compute , id , instance_profile_arn , max_num_clusters , min_num_clusters , name , spot_instance_policy , tags |
|
executeAdhocQuery |
dataSourceId |
|
executeSavedQuery |
queryId |
|
executeWidgetQuery |
widgetId |
|
favoriteDashboard |
dashboardId |
|
favoriteQuery |
queryId |
|
forkQuery |
originalQueryId , queryId |
|
listQueries |
filter_by , include_metrics , max_results , page_token |
|
moveDashboardToTrash |
dashboardId |
|
moveQueryToTrash |
queryId |
|
muteAlert |
alertId |
|
publishBatch |
statuses |
|
publishDashboardSnapshot |
dashboardId , hookId , subscriptionId |
|
restoreDashboard |
dashboardId |
|
restoreQuery |
queryId |
|
setWarehouseConfig |
data_access_config , enable_serverless_compute , instance_profile_arn , security_policy , serverless_agreement , sql_configuration_parameters , try_create_databricks_managed_starter_warehouse |
|
snapshotDashboard |
dashboardId |
|
startWarehouse |
id |
|
stopWarehouse |
id |
|
subscribeAlert |
alertId , destinationId |
|
transferObjectOwnership |
newOwner , objectId , objectType |
|
unfavoriteDashboard |
dashboardId |
|
unfavoriteQuery |
queryId |
|
unmuteAlert |
alertId |
|
unsubscribeAlert |
alertId , subscriberId |
|
updateAlert |
alertId , queryId |
|
updateAlertDestination |
alertDestinationId |
|
updateDashboard |
dashboardId |
|
updateDashboardWidget |
widgetId |
|
updateOrganizationSetting |
has_configured_data_access , has_explored_sql_warehouses , has_granted_permissions |
|
updateQuery |
queryId |
|
updateQueryDraft |
queryId |
|
updateQuerySnippet |
querySnippetId |
|
updateRefreshSchedule |
alertId , dashboardId , refreshScheduleId |
|
updateVisualization |
visualizationId |
|
clusters |
changeClusterAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
changeClusterPolicyAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
|
create |
acl_path_prefix , apply_policy_default_values , autoscale , autotermination_minutes , azure_attributes , billing_info , cluster_creator , cluster_event_notification_info , cluster_log_conf , cluster_name , cluster_source , custom_tags , data_security_mode , disk_spec , docker_image , driver_instance_pool_id , driver_instance_source , driver_node_type_id , enable_elastic_disk , enable_jobs_autostart , enable_local_disk_encryption , idempotency_token , init_scripts , instance_pool_id , instance_source , no_driver_daemon , node_type_id , num_workers , organization_id , policy_id , single_user_name , spark_conf , spark_env_vars , spark_version , ssh_public_keys , start_cluster , user_id , validate_cluster_name_uniqueness |
|
createResult |
clusterId , clusterName , clusterOwnerUserId , clusterState , clusterWorkers |
|
delete |
cluster_id , termination_reason |
|
deleteResult |
clusterId , clusterName , clusterOwnerUserId , clusterState , clusterWorkers |
|
edit |
apply_policy_default_values , autoscale , autotermination_minutes , azure_attributes , cluster_id , cluster_log_conf , cluster_name , cluster_source , custom_tags , data_security_mode , disk_spec , docker_image , driver_instance_pool_id , driver_instance_source , driver_node_type_id , enable_elastic_disk , enable_local_disk_encryption , init_scripts , instance_pool_id , instance_source , no_driver_daemon , node_type_id , num_workers , policy_id , single_user_name , spark_conf , spark_env_vars , spark_version , ssh_public_keys , validate_cluster_name_uniqueness , workload_type |
|
permanentDelete |
cluster_id |
|
resize |
autoscale , avoid_containers , cause , cluster_id , next_attempt_time_ms , num_workers |
|
resizeResult |
clusterId , clusterName , clusterOwnerUserId , clusterState , clusterWorkers |
|
restart |
cluster_id |
|
restartResult |
clusterId , clusterName , clusterOwnerUserId , clusterState , clusterWorkers |
|
start |
cluster_id |
|
startResult |
clusterId , clusterName , clusterOwnerUserId , clusterState , clusterWorkers |
|
dbfs |
addBlock |
data_length , handle |
close |
handle |
|
create |
bufferSize , overwrite , path |
|
delete |
path , recursive |
|
getSessionCredentials |
mountPoint |
|
mkdirs |
path |
|
mount |
mountPoint , owner |
|
move |
destination_path , dst , source_path , src |
|
put |
overwrite , path |
|
unmount |
mountPoint |
|
updateMount |
mountPoint , owner |
|
deltaPipelines |
create |
allow_duplicate_names , clusters , configuration , continuous , development , dry_run , id , libraries , name , storage , target , channel , edition , photon , dbr_version (internal, do not use), email_notifications (internal, do not use), filters (deprecated), |
delete |
pipeline_id |
|
edit |
allow_duplicate_names , clusters , configuration , continuous , dbr_version (internal, do not use), development , email_notifications (internal, do not use), expected_last_modified , filters (deprecated), id , libraries , name , pipeline_id , storage , target , channel , edition , photon |
|
startUpdate |
cause , full_refresh , job_task , pipeline_id |
|
stop |
pipeline_id |
|
featureStore |
addConsumer |
features , job_run , notebook |
addDataSources |
feature_table , paths , tables |
|
addProducer |
feature_table , job_run , notebook |
|
changeFeatureTableAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
|
createFeatureTable |
description , name , partition_keys , primary_keys , timestamp_keys |
|
createFeatures |
feature_table , features |
|
deleteFeatureTable |
name |
|
deleteTags |
feature_table_id , keys |
|
getConsumers |
feature_table |
|
getFeatureTable |
name |
|
getFeatureTablesById |
ids |
|
getFeatures |
feature_table , max_results |
|
getModelServingMetadata |
feature_table_features |
|
getOnlineStore |
cloud , feature_table , online_table , store_type |
|
getTags |
feature_table_id |
|
publishFeatureTable |
cloud , feature_table , host , online_table , port , read_secret_prefix , store_type , write_secret_prefix |
|
searchFeatureTables |
max_results , page_token , text |
|
setTags |
feature_table_id , tags |
|
updateFeatureTable |
description , name |
|
genie |
databricksAccess |
approver , authType , duration , isCriticalUser , reason , user |
gitCredentials |
getGitCredential |
id |
listGitCredentials |
none | |
deleteGitCredential |
id |
|
updateGitCredential |
id , git_provider , git_username |
|
createGitCredential |
git_provider , git_username ] |
|
globalInitScripts |
batch-reorder |
script_ids |
create |
enabled , name , position , script-SHA256 |
|
delete |
script_id |
|
update |
enabled , name , position , script-SHA256 , script_id |
|
instancePools |
changeInstancePoolAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
create |
azure_attributes , custom_tags , disk_spec , enable_elastic_disk , idle_instance_autotermination_minutes , instance_pool_name , max_capacity , min_idle_instances , node_type_id , preloaded_docker_images , preloaded_spark_versions |
|
delete |
instance_pool_id |
|
edit |
azure_attributes , custom_tags , disk_spec , enable_elastic_disk , idle_instance_autotermination_minutes , instance_pool_id , instance_pool_name , max_capacity , min_idle_instances , node_type_id , preloaded_spark_versions |
|
jobs |
cancel |
run_id |
cancelAllRuns |
job_id |
|
changeJobAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
|
create |
access_control_list , dbt_task , email_notifications , existing_cluster_id , format , git_source , is_from_dlt , job_clusters , job_type , libraries , max_concurrent_runs , max_retries , min_retry_interval_millis , name , new_cluster , notebook_task , pipeline_task , python_wheel_task , retry_on_timeout , schedule , shell_command_task , spark_jar_task , spark_python_task , spark_submit_task , tasks , timeout_seconds |
|
delete |
job_id |
|
deleteRun |
run_id |
|
repairRun |
latest_repair_id , rerun_tasks , run_id |
|
reset |
is_from_dlt , job_id , new_settings |
|
resetJobAcl |
grants , job_id |
|
runFailed |
clusterId , idInJob , jobClusterType , jobId , jobTaskType , jobTerminalState , jobTriggerType , orgId , runId |
|
runNow |
jar_params , job_id , notebook_params , python_params , spark_submit_params , workflow_context |
|
runStart |
clusterId , idInJob , jobClusterType , jobId , jobTaskType , jobTerminalState , jobTriggerType , orgId , runId |
|
runSucceeded |
clusterId , idInJob , jobClusterType , jobId , jobTaskType , jobTerminalState , jobTriggerType , orgId , runId |
|
setTaskValue |
key , run_id |
|
submitRun |
access_control_list , existing_cluster_id , idempotency_token , job_cluster_key , job_clusters , libraries , new_cluster , notebook_task , run_name , shell_command_task , spark_jar_task , spark_python_task , spark_submit_task , tasks , timeout_seconds , workflow_context |
|
update |
fields_to_remove , job_id , new_settings |
|
mlflowAcledArtifact |
readArtifact |
artifactLocation , experimentId , runId |
writeArtifact |
artifactLocation , experimentId , runId |
|
mlflowExperiment |
deleteMlflowExperiment |
experimentId , experimentName , path |
moveMlflowExperiment |
experimentId , newPath , oldPath |
|
renameMlflowExperimentEvent |
experimentId , newName , oldName , parentPath |
|
restoreMlflowExperiment |
experimentId , experimentName , path |
|
modelRegistry |
approveTransitionRequest |
archive_existing_versions , comment , name , stage , version |
changeRegisteredModelAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
|
createComment |
comment , name , version |
|
createModelVersion |
description , name , run_id , run_link , source , tags |
|
createRegisteredModel |
description , name , tags |
|
createRegistryWebhook |
description , events , http_url_spec , job_spec , model_name , status |
|
createTransitionRequest |
comment , name , stage , version |
|
deleteModelVersion |
name , version |
|
deleteModelVersionTag |
key , name , version |
|
deleteRegisteredModel |
name |
|
deleteRegisteredModelTag |
key , name |
|
deleteRegistryWebhook |
id |
|
deleteTransitionRequest |
comment , creator , name , stage , version |
|
finishCreateModelVersionAsync |
name , version |
|
generateBatchInferenceNotebook |
input_data , name , output_path , stage , version |
|
getModelVersionDownloadUri |
name , version |
|
getModelVersionSignedDownloadUri |
name , path , version |
|
listModelArtifacts |
name , path , version |
|
listRegistryWebhooks |
max_results , model_name |
|
rejectTransitionRequest |
comment , name , stage , version |
|
renameRegisteredModel |
name , new_name |
|
setEmailSubscriptionStatus |
model_name , subscription_type |
|
setModelVersionTag |
key , name , value , version |
|
setRegisteredModelTag |
key , name , value |
|
setUserLevelEmailSubscriptionStatus |
subscription_type |
|
testRegistryWebhook |
id |
|
transitionModelVersionStage |
archive_existing_versions , comment , name , stage , version |
|
updateRegistryWebhook |
description , events , http_url_spec , id , status |
|
notebook |
attachNotebook |
clusterId , notebookId , path |
cloneNotebook |
notebookId , path |
|
createNotebook |
notebookId , path |
|
deleteFolder |
path |
|
deleteNotebook |
notebookId , notebookName , path |
|
detachNotebook |
clusterId , notebookId , path |
|
downloadLargeResults |
notebookFullPath , notebookId |
|
downloadPreviewResults |
notebookFullPath , notebookId |
|
importNotebook |
path , workspaceExportFormat |
|
moveNotebook |
newPath , notebookId , oldPath |
|
renameNotebook |
newName , notebookId , oldName , parentPath |
|
restoreFolder |
path |
|
restoreNotebook |
notebookId , notebookName , path |
|
runCommand (only for verbose audit logs) |
notebookId , executionTime , status , commandId , commandText (see details) |
|
takeNotebookSnapshot |
path |
|
repos |
checkoutBranch |
branch , id |
commitAndPush |
checkSensitiveToken , files , id , message |
|
createRepo |
git_provider , git_url , path , provider , url |
|
deleteRepo |
id |
|
discard |
file_paths , id |
|
getRepo |
id |
|
listRepos |
next_page_token , path_prefix |
|
pull |
id |
|
updateRepo |
branch , git_provider , git_url , id , path , tag , workspace_filesystem_enabled |
|
secrets |
createScope |
backend_azure_keyvault , initial_manage_principal , is_databricks_managed , scope , scope_backend_type |
deleteAcl |
principal , scope |
|
deleteScope |
scope |
|
deleteSecret |
key , scope |
|
getAcl |
principal , scope |
|
getSecret |
key , scope |
|
listAcls |
scope |
|
listSecrets |
scope |
|
putAcl |
permission , principal , scope |
|
putSecret |
key , scope , string_value |
|
sqlPermissions |
changeSecurableOwner |
principal , securable |
createSecurable |
securable |
|
denyPermission |
permission |
|
grantPermission |
permission |
|
removeAllPermissions |
securable |
|
renameSecurable |
after , before |
|
requestPermissions |
denied , permitted , requests , truncated |
|
revokePermission |
permission |
|
showPermissions |
principal , securable |
|
ssh |
login |
containerId , instanceId , port , publicKey , userName |
logout |
containerId , instanceId , userName |
|
unityCatalog |
createCatalog |
comment , metastore_id , name , workspace_id |
createDataAccessConfiguration |
metastore_id , name , workspace_id |
|
createExternalLocation |
comment , credential_name , metastore_id , name , url , workspace_id |
|
createMetastore |
metastore_id , name , storage_root , workspace_id |
|
createMetastoreAssignment |
default_catalog_name , metastore_id , workspace_id |
|
createRecipient |
comment , metastore_id , name , workspace_id |
|
createSchema |
catalog_name , comment , metastore_id , name , workspace_id |
|
createShare |
comment , metastore_id , name , workspace_id |
|
createStagingTable |
catalog_name , metastore_id , name , schema_name , workspace_id |
|
createStorageCredential |
azure_service_principal , comment , metastore_id , name , workspace_id |
|
createTable |
catalog_name , columns , data_source_format , metastore_id , name , schema_name , storage_location , table_type , view_definition , workspace_id |
|
deleteRecipient |
metastore_id , name , workspace_id |
|
deleteSchema |
full_name_arg , metastore_id , workspace_id |
|
deleteShare |
metastore_id , name , workspace_id |
|
deleteTable |
full_name_arg , metastore_id , workspace_id |
|
deltaSharingGetTableMetadata |
metastore_id , name , recipient_name , schema , share , user_agent , workspace_id |
|
deltaSharingListAllTables |
metastore_id , options , recipient_name , share , user_agent , workspace_id |
|
deltaSharingListSchemas |
metastore_id , options , recipient_name , share , user_agent , workspace_id |
|
deltaSharingListShares |
metastore_id , options , recipient_name , user_agent , workspace_id |
|
deltaSharingListTables |
metastore_id , options , recipient_name , schema , share , user_agent , workspace_id |
|
deltaSharingQueriedTable |
metastore_id , recipient_name |
|
deltaSharingQueryTable |
limitHint , metastore_id , name , recipient_name , schema , share , user_agent , workspace_id |
|
generateTemporaryPathCredential |
credential_id , credential_type , ensure_url_exists , metastore_id , operation , url , workspace_id |
|
generateTemporaryTableCredential |
credential_id , credential_type , is_permissions_enforcing_client , metastore_id , operation , table_full_name , table_id , workspace_id |
|
getActivationUrlInfo |
metastore_id , recipient_name , workspace_id |
|
getCatalog |
metastore_id , name_arg , workspace_id |
|
getDataAccessConfiguration |
id , metastore_id , workspace_id |
|
getExternalLocation |
metastore_id , name_arg , workspace_id |
|
getMetastore |
id , metastore_id , workspace_id |
|
getMetastoreSummary |
metastore_id , workspace_id |
|
getPermissions |
metastore_id , principal , securable_full_name , securable_type , workspace_id |
|
getRecipient |
metastore_id , name , workspace_id |
|
getRecipientSharePermissions |
metastore_id , name , workspace_id |
|
getSchema |
full_name_arg , metastore_id , workspace_id |
|
getShare |
include_shared_data , metastore_id , name , workspace_id |
|
getSharePermissions |
metastore_id , name , workspace_id |
|
getStorageCredential |
metastore_id , name_arg , workspace_id |
|
getTable |
full_name_arg , metastore_id , workspace_id |
|
listCatalogs |
metastore_id , workspace_id |
|
listDataAccessConfigurations |
metastore_id , workspace_id |
|
listExternalLocations |
credential_name , metastore_id , url , workspace_id |
|
listMetastores |
workspace_id |
|
listRecipients |
metastore_id , workspace_id |
|
listSchemas |
catalog_name , metastore_id , workspace_id |
|
listShares |
metastore_id , workspace_id |
|
listStorageCredentials |
metastore_id , workspace_id |
|
listTableSummaries |
catalog_name , metastore_id , schema_name_pattern , table_name_pattern , workspace_id |
|
listTables |
catalog_name , metastore_id , schema_name , workspace_id |
|
metadataAndPermissionsSnapshot |
metastore_id , securables , workspace_id |
|
metadataSnapshot |
metastore_id , securables , workspace_id |
|
privilegedGetAllPermissions |
metastore_id , securables , workspace_id |
|
privilegedGetTable |
full_name_arg , metastore_id , workspace_id |
|
retrieveRecipientToken |
metastore_id , recipient_name , workspace_id |
|
updateMetastore |
default_data_access_config_id , delta_sharing_enabled , delta_sharing_recipient_token_lifetime_in_seconds , id , metastore_id , name , owner , storage_root_credential_id , workspace_id |
|
updateMetastoreAssignment |
default_catalog_name , metastore_id , workspace_id |
|
updatePermissions |
changes , metastore_id , securable_full_name , securable_type , workspace_id |
|
updateSchema |
full_name_arg , metastore_id , owner , workspace_id |
|
updateShare |
metastore_id , name , updates , workspace_id |
|
updateSharePermissions |
changes , metastore_id , name , workspace_id |
|
updateTables |
columns , data_source_format , full_name_arg , metastore_id , storage_location , table_type , workspace_id |
|
webTerminal |
startSession |
socketGUID , clusterId , serverPort , ProxyTargetURI |
closeSession |
socketGUID , clusterId , serverPort , ProxyTargetURI |
|
workspace |
changeWorkspaceAcl |
aclPermissionSet , resourceId , shardName , targetUserId |
fileCreate |
path |
|
fileDelete |
path |
|
purgeClusterLogs |
logFilePath |
|
purgeRevisionHistoryBefore |
property , propertyValue , treestoreId |
|
purgeWorkspaceNodes |
treestoreId |
|
workspaceConfEdit (workspace-level setting changes) |
workspaceConfKeys (for example, verbose audit logs uses value enableVerboseAuditLogs ), workspaceConfValues (for example, for verbose audit logs this is true or false ) |
|
workspaceExport |
notebookFullPath , workspaceExportDirectDownload , workspaceExportFormat |
Sample log output
The following JSON sample is an example of Azure Databricks log output:
{
"TenantId": "<your-tenant-id>",
"SourceSystem": "|Databricks|",
"TimeGenerated": "2019-05-01T00:18:58Z",
"ResourceId": "/SUBSCRIPTIONS/SUBSCRIPTION_ID/RESOURCEGROUPS/RESOURCE_GROUP/PROVIDERS/MICROSOFT.DATABRICKS/WORKSPACES/PAID-VNET-ADB-PORTAL",
"OperationName": "Microsoft.Databricks/jobs/create",
"OperationVersion": "1.0.0",
"Category": "jobs",
"Identity": {
"email": "mail@contoso.com",
"subjectName": null
},
"SourceIPAddress": "131.0.0.0",
"LogId": "201b6d83-396a-4f3c-9dee-65c971ddeb2b",
"ServiceName": "jobs",
"UserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36",
"SessionId": "webapp-cons-webapp-01exaj6u94682b1an89u7g166c",
"ActionName": "create",
"RequestId": "ServiceMain-206b2474f0620002",
"Response": {
"statusCode": 200,
"result": "{\"job_id\":1}"
},
"RequestParams": {
"name": "Untitled",
"new_cluster": "{\"node_type_id\":\"Standard_DS3_v2\",\"spark_version\":\"5.2.x-scala2.11\",\"num_workers\":8,\"spark_conf\":{\"spark.databricks.delta.preview.enabled\":\"true\"},\"cluster_creator\":\"JOB_LAUNCHER\",\"spark_env_vars\":{\"PYSPARK_PYTHON\":\"/databricks/python3/bin/python3\"},\"enable_elastic_disk\":true}"
},
"Type": "DatabricksJobs"
}
Analyze diagnostic logs
If you selected the Send to Log Analytics option when you turned on diagnostic logging, diagnostic data from your container is typically forwarded to Azure Monitor logs within 15 minutes.
Before you view your logs, verify if your Log Analytics workspace has been upgraded to use the new Kusto query language. To check, open the Azure portal and select Log Analytics on the far left. Then select your Log Analytics workspace. If you get a message to upgrade, see Upgrade your Azure Log Analytics workspace to new log search.
To view your diagnostic data in Azure Monitor logs, open the Log Search page from the left menu or the Management area of the page. Then enter your query into the Log search box.
Queries
Here are some additional queries that you can enter into the Log search box. These queries are written in Kusto Query Language.
To query all users who have accessed the Azure Databricks workspace and their location:
DatabricksAccounts | where ActionName contains "login" | extend d=parse_json(Identity) | project UserEmail=d.email, SourceIPAddress
To check the Spark versions used:
DatabricksClusters | where ActionName == "create" | extend d=parse_json(RequestParams) | extend SparkVersion= d.spark_version | summarize Count=count() by tostring(SparkVersion)
Feedback
Submit and view feedback for