how to create diff environments for different python packages in production

Question

We have couple of libraries which are created to be used by user while working on their azure ML workspace.

Now, both of these libraries have diff packages which could be different then what a user is using. Eg : User might be using an environment which is using numpy x.1 but package A might be using x.2 and package B needs x.3. This is a possibility since all these packages are developed by different teams.

Now, what could be the best way to handle this problem in real world. So far, I am able to come up with below approaches :

1. Install these files in different docker container where the needed packages are installed. And get the desired output done in separate environments.
2. Use Custom Environment options provided by Azure itself. And run the incompatibles ones in different environment.

So, I wanted to know if there is any right way of doing this in real world. I see that we should create a different environment for each project but what about the case when we have different packages which needs different versions of common dependencies. How to handle such case?

Answer 1

@ja Thanks for the details.In your specific case, you can consider using virtual environments and package managers to manage dependencies for each library. You can create separate virtual environments for each library, with its own set of dependencies and package versions. You can also use package managers like pip and conda to install and manage packages in each virtual environment. This can help you avoid conflicts between different packages and ensure that each library has the required dependencies.

Yes, Users can either select from an existing environment or create a new one. If they choose to create a new one, they can follow the environment creation wizard to create an environment based on Python virtual environment, conda yaml, docker image or Dockerfile. 

AzureML offers compute instance for single-node interactive experience based on conda environments. Data scientists can quickly check out a VM like their dev box with pre-configured and up-to-date ML packages, deep learning frameworks, GPU drivers.