How to utilize all CPU on azure vm by python multiprocessing?

Question

Hi,

Lately, I've got my python script running on azure VM and hope that by leveraging more CPU can help my program that runs faster.

My python script includes logic for multiprocessing and it works 100% when I run it on-premises PC.

However, when I run it on azure VM, it looks as if the program is running using 1 CPU only (avg CPU capacity less than 5%).

Here is the CPU validation result in python:

import multiprocessing

multiprocessing.cpu_count()

16

import psutil

psutil.cpu_count()

16

psutil.cpu_count(logical=False)

8

psutil.cpu_count(logical=True)

16

len(psutil.Process().cpu_affinity())

16

=================================

My azure VM spec as as follows:

OS: Ubuntu Server 18.04

Size: Standard D16s v3 (16 vcpus, 64 GiB memory)

Python editor: Spyder

=================================

======================================
20200427

I've tested it on another azure VM with spec below but the same result came up

OS: data science virtual machine (windows) family: GPU size: Standard NV8as_v4 (8 vcpus, 28 GiB memory)

By monitoring resource monitor, the system does create multi workers (number of cores - 1) that I need, but each worker does not fully utilize all CPU capacity to parallelly run the job.

Thank you for your answer in advance.

Stay healthy!!!

Bests,
Kelvin

Answer 1

@kelvin33882020 - I am running into the same issue when trying to use the multiprocessing package on a VM. Were you able to resolve this issue? If so, I'd appreciate hearing how you were able to get it running properly.