Has Anyone Been Able to Successfully Parallelize Their Code on a VM?

Question

I wrote a script to train a large number of time series models on my Azure VM and it uses the multiprocessing.Pool package to parallelize the code to reduce the training time period. The code runs perfectly on my local machine, but does not parallelize across the available cores when I run the script on my Azure VM. If anyone has run into this problem with the Pool package or has alternate suggestions as to how to parallelize training on a VM, I'd appreciate your insight. Thanks!

Answer 1

@Gregory Jacobs Thanks for the question. For parallel/distributed training - if deep learning, frameworks like PyTorch and Tensorflow can typically be distributed directly or use something like Horovod to easily scale out on Azure ML Clusters - see an example here.

Another I've been watching is HyperGBM/Hypernets: DataCanvasIO/Hypernets: A General Automated Machine Learning framework to simplify the development of End-to-end AutoML toolkits in specific domains. (github.com), DataCanvasIO/HyperGBM: A full pipeline AutoML tool for tabular data (github.com).