How to pass multiple models for Batch Deployment?

Furqan Hashim 20 Reputation points
2023-03-14T08:32:01.18+00:00

I have an Azure cluster instance with 100 nodes and I want to perform batch inference on that cluster.

I have more than 1 model that will be used to perform inferencing. These models will be accessed in scoring file. I am passing the list of models to Batch Deployment Class using code attached. I get an error as below

Error creating models asset: must be type Optional[Union[str, Asset]]

modelsList = ['model1', 'model2', 'model3']

deployment = BatchDeployment(
    name="ml-inference",
    description="Batch deployment for scoring",
    endpoint_name=endpoint_name,
    model=modelsList,
    code_path=".",
    scoring_script="batch_inference.py",
    environment="ml-batch-env:1",
    compute=compute_name,
    instance_count=2,
    max_concurrency_per_instance=2,
    mini_batch_size=10,
    output_action=BatchDeploymentOutputAction.APPEND_ROW,
    output_file_name="predictions.csv",
    retry_settings=BatchRetrySettings(max_retries=3, timeout=30),
    logging_level="info",
)
Not Monitored
Not Monitored
Tag not monitored by Microsoft.
35,827 questions
{count} votes