FIPS deployment fails with "ERROR Daemon Daemon Failed to decrypt /var/lib/waagent/Certificates.p7m"

Question

Hi ,

We are deploying our custom image (built on photon) on Azure. This image has FIPS enabled. We use Walinux agent to read customData to configure the appliance during boot time.

We are deploying this way in Azure since many years.

Few days back our deployment started failing with below logs in Azure console logs

2024-07-01T06:48:12.794028Z WARNING Daemon Fetching the goal state failed: [Errno 2] No such file or directory: '/var/lib/waagent/Certificates.pem'

2024-07-01T06:48:12.808102Z INFO Daemon Fetch goal state completed

2024-07-01T06:48:12.814769Z INFO Daemon Daemon WireServer is not responding. Reset dhcp endpoint

2024-07-01T06:48:12.821493Z INFO Daemon Daemon Protocol endpoint not found: [ProtocolError] Error fetching goal state

Inner error: [Errno 2] No such file or directory: '/var/lib/waagent/Certificates.pem'

2024-07-01T06:48:12.834931Z INFO Daemon Daemon Retry detect protocol: retry=231

2024/07/01 06:48:13.797142839 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:14.808611728 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:15.819193558 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:16.829892671 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:17.840904163 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:18.856328184 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:19.866942338 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:20.878608919 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:21.890392684 sysconfig: INFO Looking for Azure's CustomData

2024-07-01T06:48:22.850735Z INFO Daemon Daemon WireServer endpoint is not found. Rerun dhcp handler

2024-07-01T06:48:22.854490Z INFO Daemon Daemon Test for route to 168.63.129.16

2024-07-01T06:48:22.864387Z INFO Daemon Daemon Route to 168.63.129.16 exists

2024-07-01T06:48:22.870227Z INFO Daemon Daemon Wire server endpoint:168.63.129.16

2024-07-01T06:48:22.881445Z INFO Daemon Daemon Fabric preferred wire protocol version:2015-04-05

2024-07-01T06:48:22.888234Z INFO Daemon Daemon Wire protocol version:2012-11-30

2024-07-01T06:48:22.894364Z INFO Daemon Daemon Server preferred version:2015-04-05

2024/07/01 06:48:22.902659169 sysconfig: INFO Looking for Azure's CustomData

2024/07/01 06:48:23.916844784 sysconfig: INFO Looking for Azure's CustomData

2024-07-01T06:48:24.757092Z INFO Daemon Daemon Initializing goal state during protocol detection

2024-07-01T06:48:24.757603Z INFO Daemon Daemon Forcing an update of the goal state.

2024-07-01T06:48:24.772942Z INFO Daemon Fetched a new incarnation for the WireServer goal state [incarnation 1]

2024-07-01T06:48:24.774433Z WARNING Daemon Daemon [PERIODIC] Too many files under: /var/lib/waagent/events, current count: 1000, removing oldest event files

2024-07-01T06:48:24.803260Z INFO Daemon Daemon HostGAPlugin version: 1.0.8.151

2024-07-01T06:48:24.805015Z INFO Daemon

2024-07-01T06:48:24.811328Z INFO Daemon Fetched new vmSettings [HostGAPlugin correlation ID: 06bb1c98-7440-4f5f-8acc-4b78b2ec247a eTag: 6511292689474461495 source: Fabric]

2024-07-01T06:48:24.813124Z INFO Daemon The vmSettings originated via Fabric; will ignore them.

2024-07-01T06:48:24.821338Z INFO Daemon

2024-07-01T06:48:24.821529Z INFO Daemon Fetching full goal state from the WireServer [incarnation 1]

2024-07-01T06:48:24.829603Z INFO Daemon Daemon Downloading artifacts profile blob

2024/07/01 06:48:24.926464586 sysconfig: INFO Looking for Azure's CustomData

2024-07-01T06:48:24.944523Z ERROR Daemon Daemon Failed to decrypt /var/lib/waagent/Certificates.p7m (return code: 1)

[stdout]

[stderr] Error verifying PKCS12 MAC; no PKCS12KDF support. Use -nomacver if MAC verification is not required.

Comparing the delta between previous working and not working deployment , we found out that openssl version was updated.

openssl version in working case was 3.0.13-4.ph4 and in not working case it is 3.0.14-1.ph4.

We verified by creating a new image with downgraded version of openssl i.e 3.0.13-4.ph4, and it works just fine.

This issue is not seen with image where FIPS is disabled i.e it works just fine on the latest version 3.0.14-1.ph4.

Can you please let us know what is causing this issue ?

Thanks,

Sharath

Answer 1

@sharath babu It looks like when the openssl upgrade some files changed and were possible deleted. From the error it shows Inner error: [Errno 2] No such file or directory: '/var/lib/waagent/Certificates.pem'

What version of waagent is installed? After upgrading openssl are you able to update waagent?