az upgrade fails from 2.33 to 2.44 fails with jmespath dependency error

Question

I've read in answers to other questions that updates to the Azure CLI version would help get extensions (like az ml) working in the way the documentation states they do.

On az upgrade I see an installation error (Standard Azure ML notebook VM, Ubuntu 18.04) looks ultimately due to the jmespath dependency unavailable.

/opt/az/lib/python3.6/ doesn't exist, but /opt/az/lib/python3.10 does.

What do you advise?

az upgrade  
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus  
Your current Azure CLI version is 2.33.0. Latest version available is 2.40.0.  
Please check the release notes first: https://learn.microsoft.com/cli/azure/release-notes-azure-cli  
Do you want to continue? (Y/n): y  
  
... a lot of apt update calls ...  
  
Get:1 https://packages.microsoft.com/repos/azure-cli bionic/main amd64 azure-cli all 2.40.0-1~bionic [77.4 MB]  
Fetched 77.4 MB in 1s (64.0 MB/s)      
(Reading database ... 231878 files and directories currently installed.)  
Preparing to unpack .../azure-cli_2.40.0-1~bionic_all.deb ...  
Unpacking azure-cli (2.40.0-1~bionic) over (2.33.0-1~bionic) ...  
Setting up azure-cli (2.40.0-1~bionic) ...  
Auto upgrade failed. name 'exit_code' is not defined  
Traceback (most recent call last):  
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/util/custom.py", line 153, in upgrade_version  
  File "/opt/az/lib/python3.6/importlib/__init__.py", line 166, in reload  
  File "<frozen importlib._bootstrap>", line 600, in _exec  
AttributeError: 'NoneType' object has no attribute 'name'  
  
During handling of the above exception, another exception occurred:  
  
Traceback (most recent call last):  
  File "/opt/az/lib/python3.6/runpy.py", line 193, in _run_module_as_main  
  File "/opt/az/lib/python3.6/runpy.py", line 85, in _run_code  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/__main__.py", line 48, in <module>  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/__main__.py", line 35, in cli_main  
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 243, in invoke  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 135, in exception_handler  
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/util.py", line 57, in handle_exception  
ModuleNotFoundError: No module named 'jmespath'