Issue: When databrick notebooks are triggered by the ADF pipeline get an error "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached"
"The spark driver has stopped unexpectedly" is due to insufficient driver memory.
We observed when the error occurs from the ADF pipelines, the notebooks are running concurrently.
The error was resolved after changing the Notebooks run to sequential via the ADF pipeline.