RTC gets set from UTC to local time every 11 minutes
Environment
- Red Hat Enterprise Linux (RHEL) 7, 8
- ntpd
- chronyd
Issue
- RTC gets set from UTC to local time every 11 minutes even if we manually set it to UTC on the command line.
Resolution
-
There were This content is not included.RHEL7 RFE / This content is not included.RHEL8 RFE filed to address this issue, but were closed as a known bug which won't be fixed soon.
-
Available known workaround is to reboot your system and run the following commands before 11 minutes mode gets activated again:
# hwclock --utc --systohc
# reboot
If it gets completed in the 11 minutes interval after the first reboot, RTC will stay set to UTC after the second reboot.
Root Cause
If RTC is set to local time after reboot (by default it is set to UTC after clean installation) and at the same time ntpd/chronyd is running, kernel sets internal variable persistent_clock_is_local and keeps updating the RTC with local time every 11 minutes:
Once persistent_clock_is_local is set 1, it cannot be set to 0 (there is just no code in the kernel to do this),
that's why every 11 mins the local time offset will be applied, and even if RTC was set to UTC manually,
it will be updated to local time back.
Diagnostic Steps
Stap script to monitor RTC changes:
#!/usr/bin/env stap
probe kernel.function("__set_rtc_time") {
printf ("%s %s(%d) called __set_rtc_time(%s) time=%s\n", ctime(), execname(), pid(), $$parms, $time$$)
print_backtrace()
}
We inspected the call trace caught by the stap script:
# stap -v monitor-rtc.stp
...
Fri Mar 16 16:01:32 2018 kworker/1:2(18990) called __set_rtc_time(time=0xffff88019d2a3d6c) time={hour=12, min=1, sec=32}
0xffffffff810340f5 : mach_set_rtc_mmss+0x45/0x200 [kernel]
0xffffffff8103448f : update_persistent_clock+0x1f/0x30 [kernel]
0xffffffff810eda04 : sync_cmos_clock+0x94/0x120 [kernel]
0xffffffff810aa3ba : process_one_work+0x17a/0x440 [kernel]
0xffffffff810ab086 : worker_thread+0x126/0x3c0 [kernel]
0xffffffff810b252f : kthread+0xcf/0xe0 [kernel]
0xffffffff816b8798 : ret_from_fork+0x58/0x90 [kernel]
The point of interest here is sync_cmos_clock():
=== kernel/time/ntp.c
static void sync_cmos_clock(struct work_struct *work)
{
…
if (persistent_clock_is_local)
adjust.tv_sec -= (sys_tz.tz_minuteswest * 60);
…
fail = update_persistent_clock(adjust);
…
}
The key thing here is how clock adjustment is handled. The "persistent_clock_is_local" is checked, and if it is set, the clock value is updated to reflect the local time offset.
Next, we are checking how this variable is being set:
[1641c4cc6eaa][~/work/src/linux]$ grep -Rn persistent_clock_is_local
include/linux/timekeeping.h:208:extern int persistent_clock_is_local;
kernel/time/ntp.c:522: if (persistent_clock_is_local)
kernel/time.c:122:int persistent_clock_is_local;
kernel/time.c:145: persistent_clock_is_local = 1;
This variable is set here:
140 static inline void warp_clock(void)
141 {
142 if (sys_tz.tz_minuteswest != 0) {
…
145 persistent_clock_is_local = 1;
…
149 }
150 }
And this function is called only from one place:
163 int do_sys_settimeofday(const struct timespec *tv, const struct timezone *tz)
164 {
165 static int firsttime = 1;
…
175 if (tz) {
…
178 if (firsttime) {
179 firsttime = 0;
180 if (!tv)
181 warp_clock();
182 }
183 }
…
187 }
So, setting persistent_clock_is_local to 1 is performed only once and then is checked on every sync_cmos_clock() invocation every 11 mins (if NTP daemon is running).
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.