Azure ML Pipeline pyarrow dependency for installing transformers

Poojitha G 1 Reputation point
2022-02-28T15:35:23.127+00:00

I was trying to import transformers in AzureML designer pipeline, it says for importing transformers and datasets the version of pyarrow needs to >=3.0.0, but then after upgrading pyarrow's version to 3.0.0 and importing transformers pyarrow version is reset to original version of 0.16.0. attaching few error samples. please have a look.

Got exception when invoking script: 'RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):To use datasets, the module pyarrow>=3.0.0 is required, and the current version of pyarrow doesn't match this condition.If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of pyarrow.' azureml-designer-core 0.0.68 requires pyarrow==0.16.0, but you'll have pyarrow 3.0.0 which is incompatible.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,611 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. YutongTie-MSFT 46,996 Reputation points
    2022-03-01T01:14:11.423+00:00

    Hello @Poojitha G

    Thanks for reaching out to us for this issue, I will forward this issue to product team for investigation. If you could provide related guidance to me so that we can address this quicker it would be nice.

    I see your issue related to this thread in GitHub as well: https://github.com/Azure/MachineLearningNotebooks/issues/1698

    I will let you know if any update.

    Regards,
    Yutong