How to install dependencies to build wheel for a python package using pyproject.toml?

Question

I've a conda.yaml file with which I've created a custom environment. Now to run my repo for training, I've to 'pip install -e .' because I've a pyproject.toml file contains all the necessary dependencies to run this repo. But when I want to run a training everytime, I've to run pip install -e ., this is not ideal. Is there any way to cache this as part of the Azure ML environment so that I don’t have to reinstall it every time I start a new run or submit a job?

Answer 1

Hi Derick John

We can extract the dependencies from pyproject.toml and put it in a requirements.txt and create a custom docker image based out of Azure default images to install the requirement.txt

FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04
# Set working directory
WORKDIR /app
# Copy requirements
COPY requirements.txt .
# Install dependencies
RUN pip install --upgrade pip && pip install -r requirements.txt

You can also do the same probably with pyprojec.toml file

FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04
# Set up conda environment
COPY conda.yaml .
RUN conda env create -f conda.yaml
# Activate environment and install your package
COPY . /workspace
WORKDIR /workspace
RUN /bin/bash -c "source activate  && pip install -e ."

You have to provide .yaml file, pyprojec.toml/requirement.txt along the custom docker image location while creating your custom environment.

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-environments-v2?view=azureml-api-2&tabs=python#create-a-custom-environment

You can follow this guide to build a custom docker image and push it to container registry.

https://learn.microsoft.com/en-us/azure/container-registry/container-registry-quickstart-task-cli

But please make sure to provide admin access and Azure AML workspace as contributor to your container registry

If it helps, please let me know.

Thank you.

Answer 2

Hello Derick,
You can avoid re-running pip install -e . on every job by baking your package into the ML environment itself.

For example, add this to your conda.yaml:

dependencies:
  - python=3.8
  - pip
  - pip:
    - -e .
    - <any-other-pip-deps>

Then create and register an Azure ML Environment from that conda.yaml. When you reuse that named environment, Azure ML will cache the built wheel (including all pyproject.toml-specified dependencies) so you don’t need to reinstall on each run.

Or you may build a custom Docker image that already has your package installed and point your runs to use that.

Hope it helps, Cheers