fsck leaves system unusable after changing the time backwards and rebooting

Environment

• Red Hat Enterprise Linux 4
• Red Hat Enterprise Linux 5
• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 7
• Ext3 or Ext4 file system

Issue

• The date/time on the system were changed backwards by a user to a point before the last mount time of filesystem. After changing the date/time the system is then rebooted, due to power outage or user command etc. On reboot, fsck sees that the last mount time was in the future and blocks until the user enters a root password and runs fsck manually on the console.

• We need some way to prevent this scenario from leaving the system unusable. Our current workaround is to edit /etc/fstab file to disable fsck checking for a single reboot after the user sets the date/time, but this means we have no fsck checking at all for that reboot, and we’re still vulnerable to hitting the above scenario due to an unplanned reboot between setting the time and configuring fstab.

Resolution

• The package e2fsprogs version 1.41.10 shipped in RHEL 6 release has new option to /etc/e2fsck.conf file called broken_system_clock, this option was not present in e2fsprogs package prior to 1.41.10

• The e2fsck program has some heuristics that assume that the system clock is correct. When the option broken_system_clock is set to 1, e2fsck will attempt to use heuristics to determine if the time can not be trusted; and to skip time-based checks if this is true. If this boolean is set to true, then e2fsck will always assume that the system clock can not be trusted, which would help to resolve above issue.

• The option broken_system_clock could be set in /etc/e2fsck.conf file as shown below:

[options]
broken_system_clock = 1

Note
/etc/e2fsck.conf file is not present in default configuration, it needs to create it if the default behavior of e2fsck has to be changed.

Root Cause

• fsck uses the last filesystem mount time as a consistency check, and if the last mount time is ahead the current system time it will be considered a filesystem inconsistency if broken_system_clock option is not enabled.

Diagnostic Steps

This issue could be reproduced easily by setting the system time back far enough from the last mount time of the filesystem.
(Note that it only happens if the time is changed *backwards* - setting it into the future gives no problems).
The error message we see is as follows:

Checking filesystems  
IMAGE2: Superblock last mount time (Thu Mar 10 10:14:23 2011,  
Now = Tue Feb 1 14:20:34 2011) is in the future.

IMAGE2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.  
(i.e., without -a or -p options)

*** An error occurred during the file system check.  
*** Dropping you to a shell; the system will reboot  
*** when you leave the shell.  
Give root password for maintenance  
(or type Control-D to continue):

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.