Init script logging

Init script start and finish events are captured in cluster event logs. Details are captured in cluster logs. Global init script create, edit, and delete events are also captured in account-level diagnostic logs.

Init script events

Cluster event logs capture two init script events: INIT_SCRIPTS_STARTED and INIT_SCRIPTS_FINISHED, indicating which scripts are scheduled for execution and which have completed successfully. INIT_SCRIPTS_FINISHED also captures execution duration.

Global init scripts are indicated in the log event details by the key "global" and cluster-scoped init scripts are indicated by the key "cluster".

Note

Cluster event logs do not log init script events for each cluster node; only one node is selected to represent them all.

Where are init script logs written?

If cluster log delivery is configured for a cluster, the init script logs are written to /<cluster-log-path>/<cluster-id>/init_scripts.

Logs for each container in the cluster are written to a subdirectory called init_scripts/<cluster-id>_<container-ip>.

For example, if cluster-log-path is set to cluster-logs, the path to the logs for a specific container would be: dbfs:/cluster-logs/<cluster-id>/init_scripts/<cluster-id>_<container-ip>.

If the cluster is configured to write logs to DBFS, you can view the logs using the File system utility (dbutils.fs) or the DBFS CLI (legacy). For example, if the cluster ID is 1001-234039-abcde739:

dbfs ls dbfs:/cluster-logs/1001-234039-abcde739/init_scripts

1001-234039-abcde739_10_97_225_166
1001-234039-abcde739_10_97_231_88
1001-234039-abcde739_10_97_244_199

dbfs ls dbfs:/cluster-logs/1001-234039-abcde739/init_scripts/1001-234039-abcde739_10_97_225_166

<timestamp>_<log-id>_<init-script-name>.sh.stderr.log
<timestamp>_<log-id>_<init-script-name>.sh.stdout.log

When cluster log delivery is not configured, logs are written to /databricks/init_scripts. You can use standard shell commands in a notebook to list and view the logs:

%sh
ls /databricks/init_scripts/
cat /databricks/init_scripts/<timestamp>_<log-id>_<init-script-name>.sh.stdout.log

Every time a cluster launches, it writes a log to the init script log folder.

Important

Any user who creates a cluster and enables cluster log delivery can view the stderr and stdout output from global init scripts. You should ensure that your global init scripts do not output any sensitive information.

Init script events in diagnostic logs

Azure Databricks diagnostic logs capture global init script create, edit, and delete events under the event type globalInitScripts. See Global init scripts events.