Why does hardware clock (PHC0) not return UTC

Question

Why does hardware clock (PHC0) not return UTC

Steven Sommars 0

Running a Linux (Ubuntu 24.02.2 LTS) VM on an Azure hypervisor. The default time synchronization configuration includes chrony 4.5 with one source, a PHC reference clock. This would be great, except that the host clock is in error by 10-30 msec. Dropping PHC and using the four Google anycast NTP servers works much better.

What did I do wrong?

2 answers

Your answer

Answer 1

Steven Sommars 0

User's image

This graph shows the long-term difference between the PHC(/dev/ptp0) and NTP-disciplined clocks. There are strong short term variations as well.

If it matters: cat /sys/devices/system/clocksource/clocksource0/current_clocksource
hyperv_clocksource_tsc_page

Either I'm doing something wrong, or PHC is broken.

Anonymous

2025-03-04T17:26:31.39+00:00
Hi Steven Sommars,
Thank you for reaching out to us on the Microsoft Q&A forum.

It seems you're facing issues with the Precision Hardware Clock (PHC) not returning UTC on your Ubuntu 24.02.2 LTS VM running on an Azure hypervisor. Here are some insights and possible solutions:

From Your Organization:

COSMIC Linux Troubleshooting: There may be challenges with Linux debuggability tools and their integration with Azure Watson and Azure Profiler for automatic crash reports and profiler traces. These limitations could be complicating the diagnosis of the PHC issue.

Azure Profiler Overview: The Azure Profiler is a useful tool to identify performance bottlenecks, such as high idle time or hotspots in code. It might help you identify any underlying performance issues that are affecting the PHC's accuracy. From the Web:

Ensure UTC Configuration: To ensure that your hardware clock is set to UTC, check and adjust your /etc/default/rcS file. Set UTC=yes to ensure the hardware clock follows UTC. Alternatively, you can use the hwclock tool to adjust the hardware clock to UTC if necessary.

PHC and NTP Synchronization: The discrepancy between the PHC and NTP-disciplined clocks could be due to the host clock's inherent error. In this case, using multiple NTP sources—such as Google's anycast NTP servers—can provide more reliable time synchronization.

Chrony Configuration: It may also be worth reviewing your Chrony configuration to ensure that it is properly referencing the PHC as a source. If the issue persists, consider reaching out to Azure support or your internal team for further assistance. Would you like more specific guidance on any of these steps, or do you need help troubleshooting the configuration further?

Please refer blow Documentation for your refence. https://learn.microsoft.com/en-us/azure/virtual-machines/linux/time-sync
https://learn.microsoft.com/en-us/azure/virtual-machines/windows/time-sync If the information is helpful, please consider by clicking the "Upvote" on the post. If you have any further queries, please let us know in the comment. Thank you.
Steven Sommars 0 Reputation points

2025-03-05T00:48:06.8233333+00:00
Thanks for the suggestions.

This is not a performance, debugging, or crash issue. The concern is PHC (PTP hardware clock) which, by design, should return time with an offset of at most a couple of microseconds compared to UTC.

"The discrepancy between the PHC and NTP-disciplined clocks could be due to the host clock's inherent error."

Are you saying that Microsoft Azure PHC doesn't work? If so, Microsoft documentation should state this, plus the default Linux images should be changed to not use PHC.

I have tried several Chrony configurations:

Synchronized to PHC only. Result: System time is in error when comparing to stratum 1 clocks.

Synchronized to NTP stratum 1 only. (Google stratum 1 servers) Result: The above graph shows the excessive error I observe

Synchronized to NTP stratum 1 and PHC. Result: Chrony synchronizes to NTP stratum 1. PHC is marked as a falseticker.

I suspect that PHC is not working. Can Microsoft confirm or refute this? The offset reached 100 msec today!
Anonymous

2025-03-05T06:07:59.2766667+00:00

Hi Steven Sommars,

i am looking into the issue i will get back to you soon.

Thank you
Anonymous

2025-03-05T16:57:49.93+00:00
Hi Steven Sommars,

Microsoft provides PTP (Precision Time Protocol) support in Windows Server 2019 and Windows 10 (version 1809 and later), mainly through the Windows Time Service (W32Time). This service synchronizes the system clock with an external time source, such as a PTP grandmaster clock.

Windows Time Service helps synchronize clocks on devices in a domain, ensuring consistency in time across systems. With PTP support, Windows can synchronize time with high precision, typically within microseconds, especially when using specialized hardware like PTP-capable network interfaces.

To configure PTP on Windows, the W32Time service can be adjusted to use PTP or NTP servers. You can set up a Windows system as a PTP client (slave), allowing it to sync with an external PTP server. Please go through this document for your reference https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-service-tech-ref https://learn.microsoft.com/en-us/windows-server/networking/windows-time-service/windows-time-service-tools-and-settings?tabs=config If the comment was helpful, please don't forget to upvote. Thank you.
Steven Sommars 0 Reputation points

2025-03-05T17:13:42.16+00:00

I'm not running Windows, so this comment does not apply
Anonymous

2025-03-07T13:25:26.5066667+00:00
Hi Steven Sommars,

Edit the Chrony Configuration:

sudo nano /etc/chrony/chrony.conf

Comment Out the PHC Line: Add '# 'before the PHC reference clock line.

Add Google NTP Servers: Add the following lines:
server time.google.com iburst server time1.google.com iburst server time2.google.com iburst server time3.google.com iburst

Save and Exit (in nano, press CTRL + X, then Y, and Enter).

Restart Chrony:

sudo systemctl restart chronyd

Verify Configuration:

chronyc sources

If the information is helpful, please consider by clicking the "Upvote".

If you have any further queries, please let us know we are glad to help you.

Thank you.
Steven Sommars 0 Reputation points

2025-03-07T14:15:04.3166667+00:00

Already did this.

Please tell development that PHC is broken. Looks like a system bug.
Sudheer Reddy 2,055 Reputation points Microsoft External Staff Moderator

2025-03-11T06:47:40.6066667+00:00

Hi Steven Sommars,

Thank you for your patience.

Based on the above discussion, the issue is with Azure's PHC implementation affecting time synchronization accuracy.

PHC (Precision Time Protocol Hardware Clock) and NTP (Network Time Protocol) both can be used for time synchronization, but currently, no official documentation explicitly states that Azure's PHC implementation for precise synchronization tasks errors or drift. While Azure's PHC implementation is designed to provide accurate time synchronization, but its effectiveness will sometimes be compromised by host clock errors as the PHC relies entirely on the hypervisor's host clock.

Currently, for workloads which requires high precision and to improve efficiency for better alignment with UTC, external NTP servers will be the preferred choice.

https://azure.microsoft.com/en-us/blog/time-synchronization-for-financial-services-in-azure/

If the information is helpful, please click on "Upvote"

If you have any queries, please do let us know, we will help you.
Steven Sommars 0 Reputation points

2025-03-19T01:51:08.6133333+00:00

There is a bug somewhere in the Azure infrastructure. I saw it on my VMs in West US2 and US East. Should I report it somewhere other than this Q&A forum?

For comparison I ran the same tests on an AWS EC2 VM & they work as expected.
kobulloc-MSFT 26,811 Reputation points Microsoft Employee Moderator

2025-03-27T22:36:24.5133333+00:00
Hello, @Steven Sommars ! I had a conversation with VM time SMEs and will need to review again what you've done so far to determine where we are running into issues. Of note:

PTP is the most accurate, with NTP only being recommended if there is a specific environment requirement

Image source plays a role, as images may not be properly configured for PTP out of the box

To help us better troubleshoot this, what configuration steps have you taken? Can you provide the kernel version?
Steven Sommars 0 Reputation points

2025-03-28T15:04:10.11+00:00

6.8.0-1021-azure
Ubuntu 24.04.2 LTS

Problem has been seen in all five regions I've tested.

There is little configuration. Either /etc/chrony/chrony.conf has the line:
refclock PHC /dev/ptp_hyperv poll 3 dpoll -2 offset 0
or it does not. NTP servers can also be included.

PTP should be more accurate than NTP. However the Azure PTP implementation as experienced on the Azure servers is much worse. Please ask the time SMEs to read the paper I wrote (see associated private discussion).
kobulloc-MSFT 26,811 Reputation points Microsoft Employee Moderator

2025-03-28T19:05:33.8266667+00:00

Thank you, @Steven Sommars . There's not much configuration, but confirming the following is very helpful:

refclock PHC /dev/ptp_hyperv poll 3 dpoll -2 offset 0 stratum 2

makestep 1.0 -1

The makestep 1.0 -1 command addresses time drift (although a full 1 second in the example above) so I want to make sure that we've called that out:

By default, chronyd accelerates or slows the system clock to fix any time drift. If the drift becomes too large, chrony fails to fix the drift. To overcome this, the makestep parameter in /etc/chrony.conf can be changed to force a time sync if the drift exceeds the threshold specified.

From the conversation I had with our SME, I also want to note that Azure Hosts are considered Stratum 2 since they sync to Stratum 1 devices while VM’s are Stratum 3 since they sync with the Azure Hosts.

Still, the values you are seeing are different than those we are seeing and I'm going to get someone to take a closer look at this to determine why you are seeing the results you are getting.
Steven Sommars 0 Reputation points

2025-03-28T21:24:15.99+00:00

The identical refclock line is present in all VMs.
Each VM has a makestep line"
makestep 1 3
This is slightly different than what you list. However the "1" requires a full 1 second offset before time is stepped. All of the errors I've seen to date are 0.11 seconds or less.
This difference is inconsequential.

Stratum 1 versus stratum 2 makes little difference. The majority of PTP/PHC servers interpret the PTP device as a local (stratum 0) clock; thus the server provides stratum 1 service. The root delay and root dispersion may require adjustment to adequately represent the clock offset and error.

The error I see varies with time and region:
East US
Max observed error 10000 microseconds
West US 2
Max observed error 110000 microseconds
North Central US (Chicago)
Max observed error 7000 microseconds
Central US (Des Moines)
Max observed error 18000 microseconds
Germany West Central:
Max observed error 600 microseconds

It is likely that the error varies within a given region.

The problem lies in the PTP subsystem, not in chrony.

Answer 2

Steven Sommars 0

It seems that my observations are well-founded. The Azure PTP doesn't really use IEEE 1588 PTP. Instead the Azure uses a PTP-like API to transfer time from the Hyper-V hypervisor. On the six VMs I tried that hypervisor was very poorly synchronized.

The Azure PTP document should be updated to reflect reality. This "PTP" is typically not better than NTP. It may be adequate for some users.

Share via

Why does hardware clock (PHC0) not return UTC

2 answers

Your answer