HPC 2019 Linux worker node relocation error

Question

HPC 2019 Linux worker node relocation error

ZoliSz 1

Hi all,

We have an HPC 2019 deployment on-prem.
When trying to add an ubuntu 20.04 worker to the cluster hpcagent service starts , but the node does not join the cluster as nodemanager keeps crashing.

hpclinuxagent.log:
ERROR:HPC node manager process crashes: 127
Restart HPC node manager process after 60 seconds

When starting the node manager manually I can see the following error:

relocation error: /lib/x86_64-linux-gnu/libnss_files.so.2: symbol __libc_readline_unlocked version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Versions:
HPC: HPC 2019 6.1.7531.0
OS: Ubuntu 20.04.4 LTS
libc: libc6/focal,now 2.31-0ubuntu9.9

Before this we used the same node in HPC 2016 without issues.
Also tried installing a completely fresh Ubuntu and got the same error.

Any ideas on what is causing this and how to fix ?

Thanks,
Zoltan

vipullag-MSFT 26,537 Reputation points Moderator

2022-07-01T04:04:06.113+00:00

@ZoliSz

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Do you mean a Ubuntu 20.04 compute node worked with HPC Pack 2016 but can't do in HPC Pack 2019?
ZoliSz 1 Reputation point

2022-07-01T10:12:06.117+00:00

Hi Vipullag,

Thanks for your response and followup question.

Yes, that is correct.

To test this I again joined this node to our old HPC 2016 cluster and there it works fine. The same node, same OS could join to the HPC2016 cluster without issues.
When trying this node with HPC2019, I get this "relocation error" mentioned above.

Looking at the linux setup files for HPC there are not many differences between versions.
However there is a difference in the nodemanager binary ( /opt/hpcnodemanager/nodemanager ). I think the issue is in this HPC2019 version of nodemanager binary.

Regards,
Zoltan

1 answer

Your answer

vipullag-MSFT 26,537 Reputation points Moderator

2022-07-01T04:04:06.113+00:00

@ZoliSz

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Do you mean a Ubuntu 20.04 compute node worked with HPC Pack 2016 but can't do in HPC Pack 2019?
ZoliSz 1 Reputation point

2022-07-01T10:12:06.117+00:00

Hi Vipullag,

Thanks for your response and followup question.

Yes, that is correct.

To test this I again joined this node to our old HPC 2016 cluster and there it works fine. The same node, same OS could join to the HPC2016 cluster without issues.
When trying this node with HPC2019, I get this "relocation error" mentioned above.

Looking at the linux setup files for HPC there are not many differences between versions.
However there is a difference in the nodemanager binary ( /opt/hpcnodemanager/nodemanager ). I think the issue is in this HPC2019 version of nodemanager binary.

Regards,
Zoltan

Answer 1

vipullag-MSFT 26,537 Moderator

@ZoliSz

Apologies in delayed response on this.

I checked with internal team on this and got confirmation that current Linux node manager in HPC Pack 2019 can not run on Ubuntu20.04.

Yes, you were correct that this is working on HPC 2016.

Hope that helps.
If the suggested response helped you resolve your issue, please 'Accept as answer', so that it can help others in the community looking for help on similar topics.

Richard (Jun) Li 0 Reputation points

2023-12-15T20:46:59.8066667+00:00

Get the exactly same issue with HPC Pack 2019 version 6.0.7214.0 on Ubuntu 20.04 LTS.
Just want to point out that the Microsoft website of HPC 2019 requirement says Ubuntu 20.04 is supported OS.
https://learn.microsoft.com/en-us/powershell/high-performance-computing/system-requirements-for-hpc-pack?view=hpc19-ps

Could someone follow up this contradiction?