Azure ML Pipeline pyarrow dependency for installing transformers

Question

I was trying to import transformers in AzureML designer pipeline, it says for importing transformers and datasets the version of pyarrow needs to >=3.0.0, but then after upgrading pyarrow's version to 3.0.0 and importing transformers pyarrow version is reset to original version of 0.16.0. attaching few error samples. please have a look.

Got exception when invoking script: 'RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):To use datasets, the module pyarrow>=3.0.0 is required, and the current version of pyarrow doesn't match this condition.If you are running this in a Google Colab, you should probably just restart the runtime to use the right version of pyarrow.' azureml-designer-core 0.0.68 requires pyarrow==0.16.0, but you'll have pyarrow 3.0.0 which is incompatible.

Answer 1

Hello @Poojitha G

Thanks for reaching out to us for this issue, I will forward this issue to product team for investigation. If you could provide related guidance to me so that we can address this quicker it would be nice.

I see your issue related to this thread in GitHub as well: https://github.com/Azure/MachineLearningNotebooks/issues/1698

I will let you know if any update.

Regards,
Yutong