How do you add Airflow requirements to a Workflow Orchestration Manager instance without it crashing?

Question

The Workflow Orchestration Manager has an option to add pypi dependencies. However, when I add for example "apache-airflow-providers-databricks", the instance does not start anymore (it tries to start for ~1 hour and then stops with an unspecified error).

Answer 1

Try to verify if the version of Apache Airflow you are using is compatible with the version of the provider you want to install (apache-airflow-providers-databricks) because sometimes, certain providers are only compatible with specific versions of Airflow.

To follow the best practises, you can use a requirements.txt file to list all dependencies and locate it in the root directory of your project.

Answer 2

Each Airflow Version is only tested with a single set of compatible dependencies, including their providers. WOM does not install the correct version by default. The easiest way to ensure, that the right versions are installed is to add to add a constraint file to your dependencies, see https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#reproducible-airflow-installation.

This constrains all other requirements you install to the versions that are guaranteed to work by the Airflow team. Make sure to use the right constraint file, at the time of writing it should be <github-url>/constraints-2.6.3/constraints-3.8.txt