Clarifying XFS behaviour for dates before 1901 and after 2038

Clarifying XFS behaviour for dates before 1901 and after 2038

[Jonathan Woithe]

Hi all

Today I was running some file timestamping tests to get a feel for the
bigtime XFS option and to confirm that it was doing what I expected. 
Everything made sense until a certain point.

There are two systems:

 * PC-1: Slackware64 15.0, xfsprogs 5.13.0, 5.15.27 kernel

 * PC-2: Slackware64 14.2, xfsprogs 4.3.0, 4.4.19 kernel

On PC-1 with an xfs created many years ago, xfs_info reports bigtime=0 (as
expected).  Two tests were run:

 * > touch -d '1800/01/01 02:23:45.67' foobar
   > ls --full-time foobar
   -rw-r--r-- 1 jwoithe users 0 1901-12-14 06:15:52.000000000 +0930 foobar

 * > touch -d '2100/01/01 02:23:45.67' foobar
   > ls --full-time foobar
   -rw-r--r-- 1 jwoithe users 0 2038-01-19 13:44:07.000000000 +1030 foobar

Both results are entirely as expected: the times are clamped to the minimum
and maximum values.  The +0930 timezone in the 1901 date is due to there
being no daylight saving in operation in 1901.

A newly created xfs is also on PC-1 where bigtime was requested during
mkfs.xfs.  Bigtime is confirmed set according to xfs_info.  Three tests were
run:

 * > touch -d '1800/01/01 02:23:45.67' foobar
   > ls --full-time foobar
   -rw-r--r-- 1 radar users 0 1901-12-14 06:15:52.000000000 +0930 foobar

 * > touch -d '2100/01/01 02:23:45.67' foobar
   > ls --full-time foobar
   -rw-r--r-- 1 radar users 0 2100-01-01 02:23:45.670000000 +1030 foobar

 * > touch -d '2800/01/01 02:23:45.67' foobar
   > ls --full-time foobar
   -rw-r--r-- 1 radar users 0 2486-07-03 05:50:24.000000000 +0930 foobar

Again, everything is as expected.  Bigtime expands the maximum time out to
2486 as advertised.  The +0930 timezone in the last result is due to there
being no daylight saving in July.

Turning to PC-2, things became confusing.  This older enviroment also has an
xfs created many years ago.  Two tests were run:

 * > touch -d '1800/01/01 02:23:45.67' foobar
   > ls --full-time foobar
   -rw-r--r-- 1 jwoithe users 0 1800-01-01 02:23:45.670000000 +0914 foobar

 * > touch -d '2100/01/01 02:23:45.67' foobar
   > ls --full-time foobar
   -rw-r--r-- 1 jwoithe users 0 2100-01-01 02:23:45.670000000 +1030 foobar

Since the kernel on PC-2 is way earlier than 5.10 and its xfs filesystems
predate bigtime, I would have expected the times to be clamped between
1901 and 2038.  However, it seems that the system somehow manages to store
the out-of-bound years.  Doing so has an interesting effect on the timezone
offset for the pre-1901 years, but for years beyond 2038 there is no
directly observable problem.  Incidently, running

  stat foobar

happily reports the extended date in this case too:

  Access: 2100-01-01 02:23:45.670000000 +1030
  Modify: 2100-01-01 02:23:45.670000000 +1030

For a giggle I tried

  > touch -d '21000/01/01 02:23:45.67' foobar

and the system still managed to store the 5-digit year:

  -rw-r--r-- 1 jwoithe users 0 21000-01-01 02:23:45.670000000 +1030 foobar

This isn't what I expected.  Given an old userspace with an old kernel and
old xfs filesystem, dates outside the 1901-2038 range should not be
possible.  Given the apparent corruption of the timezone field when a year
before 1901 is set, one naive thought is that the apparent success of these
extended years on PC-2 (the old system) is due to a lack of bounds checking
on the time value and (presumedly) some overflow within on-disc structures
as a result.  This would have been noticed way before now though.

I am therefore curious about the reason for the above behaviour.  What
subtlety am I missing?

While this may be a well known quirk, it is rather difficult to search for
online.  I've tried a few things but they haven't turned up any relevant
matches.  Apologies if this is an FAQ that I can't locate.

Regards
  jonathan

Re: Clarifying XFS behaviour for dates before 1901 and after 2038

[Darrick J. Wong]

On Wed, Mar 09, 2022 at 05:53:03PM +1030, Jonathan Woithe wrote:
> Hi all
> 
> Today I was running some file timestamping tests to get a feel for the
> bigtime XFS option and to confirm that it was doing what I expected. 
> Everything made sense until a certain point.
> 
> There are two systems:
> 
>  * PC-1: Slackware64 15.0, xfsprogs 5.13.0, 5.15.27 kernel
> 
>  * PC-2: Slackware64 14.2, xfsprogs 4.3.0, 4.4.19 kernel
> 
> On PC-1 with an xfs created many years ago, xfs_info reports bigtime=0 (as
> expected).  Two tests were run:
> 
>  * > touch -d '1800/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 jwoithe users 0 1901-12-14 06:15:52.000000000 +0930 foobar
> 
>  * > touch -d '2100/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 jwoithe users 0 2038-01-19 13:44:07.000000000 +1030 foobar
> 
> Both results are entirely as expected: the times are clamped to the minimum
> and maximum values.  The +0930 timezone in the 1901 date is due to there
> being no daylight saving in operation in 1901.
> 
> A newly created xfs is also on PC-1 where bigtime was requested during
> mkfs.xfs.  Bigtime is confirmed set according to xfs_info.  Three tests were
> run:
> 
>  * > touch -d '1800/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 radar users 0 1901-12-14 06:15:52.000000000 +0930 foobar
> 
>  * > touch -d '2100/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 radar users 0 2100-01-01 02:23:45.670000000 +1030 foobar
> 
>  * > touch -d '2800/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 radar users 0 2486-07-03 05:50:24.000000000 +0930 foobar
> 
> Again, everything is as expected.  Bigtime expands the maximum time out to
> 2486 as advertised.  The +0930 timezone in the last result is due to there
> being no daylight saving in July.
> 
> Turning to PC-2, things became confusing.  This older enviroment also has an
> xfs created many years ago.  Two tests were run:
> 
>  * > touch -d '1800/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 jwoithe users 0 1800-01-01 02:23:45.670000000 +0914 foobar
> 
>  * > touch -d '2100/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 jwoithe users 0 2100-01-01 02:23:45.670000000 +1030 foobar
> 
> Since the kernel on PC-2 is way earlier than 5.10 and its xfs filesystems
> predate bigtime, I would have expected the times to be clamped between
> 1901 and 2038.  However, it seems that the system somehow manages to store
> the out-of-bound years.  Doing so has an interesting effect on the timezone
> offset for the pre-1901 years, but for years beyond 2038 there is no
> directly observable problem.  Incidently, running
> 
>   stat foobar
> 
> happily reports the extended date in this case too:
> 
>   Access: 2100-01-01 02:23:45.670000000 +1030
>   Modify: 2100-01-01 02:23:45.670000000 +1030
> 
> For a giggle I tried
> 
>   > touch -d '21000/01/01 02:23:45.67' foobar
> 
> and the system still managed to store the 5-digit year:
> 
>   -rw-r--r-- 1 jwoithe users 0 21000-01-01 02:23:45.670000000 +1030 foobar
> 
> This isn't what I expected.  Given an old userspace with an old kernel and
> old xfs filesystem, dates outside the 1901-2038 range should not be
> possible.  Given the apparent corruption of the timezone field when a year
> before 1901 is set, one naive thought is that the apparent success of these
> extended years on PC-2 (the old system) is due to a lack of bounds checking
> on the time value and (presumedly) some overflow within on-disc structures
> as a result.  This would have been noticed way before now though.
> 
> I am therefore curious about the reason for the above behaviour.  What
> subtlety am I missing?

Older kernels (pre-5.4 I think?) permitted userspace to store arbitrary
64-bit timestamps in the in-memory inode.  The fs would truncate (== rip
off the upper bits) them when writing them to disk, and then you'd get
the shattered remnants the next time the inode got reloaded from disk.

Nowadays, filesystems advertise the timestamp range they support, and
the VFS clamps the in-memory timestamp to that range.

--D

> While this may be a well known quirk, it is rather difficult to search for
> online.  I've tried a few things but they haven't turned up any relevant
> matches.  Apologies if this is an FAQ that I can't locate.
> 
> Regards
>   jonathan

Re: Clarifying XFS behaviour for dates before 1901 and after 2038

[Jonathan Woithe]

Hi Darrick

On Wed, Mar 09, 2022 at 08:00:19AM -0800, Darrick J. Wong wrote:
> On Wed, Mar 09, 2022 at 05:53:03PM +1030, Jonathan Woithe wrote:
> > Since the kernel on PC-2 is way earlier than 5.10 and its xfs filesystems
> > predate bigtime, I would have expected the times to be clamped between
> > 1901 and 2038.  However, it seems that the system somehow manages to store
> > the out-of-bound years.  Doing so has an interesting effect on the timezone
> > offset for the pre-1901 years, but for years beyond 2038 there is no
> > directly observable problem. ...
> > :
> > This isn't what I expected.  Given an old userspace with an old kernel and
> > old xfs filesystem, dates outside the 1901-2038 range should not be
> > possible.  Given the apparent corruption of the timezone field when a year
> > before 1901 is set, one naive thought is that the apparent success of these
> > extended years on PC-2 (the old system) is due to a lack of bounds checking
> > on the time value and (presumedly) some overflow within on-disc structures
> > as a result.  This would have been noticed way before now though.
> > 
> > I am therefore curious about the reason for the above behaviour.  What
> > subtlety am I missing?
> 
> Older kernels (pre-5.4 I think?) permitted userspace to store arbitrary
> 64-bit timestamps in the in-memory inode.  The fs would truncate (== rip
> off the upper bits) them when writing them to disk, and then you'd get
> the shattered remnants the next time the inode got reloaded from disk.
> 
> Nowadays, filesystems advertise the timestamp range they support, and
> the VFS clamps the in-memory timestamp to that range.

Thanks so much for taking the time to respond.  That all makes sense and is
useful information to keep in mind.

Regards
  jonathan

Re: Clarifying XFS behaviour for dates before 1901 and after 2038

[Jonathan Woithe]

Hi all

On Wed, Mar 09, 2022 at 05:53:03PM +1030, Jonathan Woithe wrote:
> Turning to PC-2, things became confusing.  This older environment also has an
> xfs created many years ago.  Two tests were run:
> 
>  * > touch -d '1800/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 jwoithe users 0 1800-01-01 02:23:45.670000000 +0914 foobar
> 
>  * > touch -d '2100/01/01 02:23:45.67' foobar
>    > ls --full-time foobar
>    -rw-r--r-- 1 jwoithe users 0 2100-01-01 02:23:45.670000000 +1030 foobar
> 
> ... However, it seems that the system somehow manages to store the
> out-of-bound years.  Doing so has an interesting effect on the timezone
> offset for the pre-1901 years, ...

Following the information provided by Darrick I still wanted to understand
the "+0914" timezone offset.  I've done some research and it turns out that
prior to 1 February 1895 the time zone in this part of the world really was
UT+0914 (or, to be entirely accurate, UT+0914.5) [1].  In those days noon
was defined to be the time that sun was directly overhead, hense the
odd-ball timezone.  The "+0914" timezone reported in the year-1800 case is
therefore correct.

Regards
  jonathan

[1] https://sahistoryhub.history.sa.gov.au/subjects/time