Export custom model/code

Alberto Vaccariello 21

If I use transformer/estimator from the sklearn package it's sufficient to export a model.pkl artifact then simply use it following the various tutorials on microsoft site.
However, when I use scikit-learn's FunctionTransformer or TransformerMixin class to incorporate custom transformations how can I export the custom code to azure machine learning together with the model? (model.pkl do not contains the custom code)
What is the best practice?

Thanks.

Ramr-msft 17,731 Reputation points

2021-05-13T15:03:20.54+00:00

@Alberto Vaccariello Thanks for the question. Can you please share the custom code that you are trying.

Alberto Vaccariello 21

I have a complete package/library project with a pyproject.toml file to manage dependencies with poetry.
Just as a quick example, I have a file
model.py
:

class SqrtTransformer(TransformerMixin, BaseEstimator):
    def __init__(self, param: str='dummy'):
        self.param = param

    def fit(self, X: np.ndarray, y: Any=None) -> TransformerMixin:
        ...
        return self

    def transform(self, X: np.ndarray) -> np.ndarray:
        ...
        return np.sqrt(X)

It's basically the template in the sklearn github repo (I have removed comments for clarity).
Then in another file I do something like:

    from mypackage.model import SqrtTransformer
    ...
    preprocessing = SqrtTransformer()
    preprocessing.fit(X)
    Y = preprocessing.transform(X)

    joblib.dump(preprocessing, MODEL_FILE)

Alberto Vaccariello 21 Reputation points

2021-05-13T16:32:24.447+00:00

NOTE: I know that this simple transformation I can do it in other simpler way, I just want to test how to use a custom sklearn model with the Azure Machine Learning service.
I have a pretty standard
score.py
file, but when I deploy to a container I get errors about not existing mypackage.model module. It's ok because the .pkl file do not contains the code and what I need a simple way to deploy it as well.
So far the solution I have found is to generate the project wheel, publish it to the DevOps Artifact and include the package in the pip requirements, but I was hoping for a better and simpler solution.
I have also tried to upload the wheel using
Environment.add_private_pip_wheel
but it do not replace the file if already exists (raise an exception or just ignore it if
exist_ok=True
is passed) and I need to replace the file everytime. So I need to check and delete the file on the blob storage before calling that function.
Is there a better/simpler way?
Ramr-msft 17,731 Reputation points

2021-06-09T06:58:57.017+00:00

@Alberto Vaccariello Thanks for the update. We would recommend to raise a Azure support desk ticket from Help+Support blade from Azure portal for your resource if you have a support plan for your subscription. This will help you to share the details securely and work with an engineer who can provide more insights about the issue that if it can be replicated.
Karthick Perumal 1 Reputation point

2022-07-19T08:30:08.827+00:00

I have the same issue with using custom Transformers/estimators. Is the question already answered and if yes, can you please provide an update. Thanks

Share via

Export custom model/code