archive mirror
 help / color / mirror / Atom feed
From: Josh Triplett <>
To: "Darrick J. Wong" <>
Cc: "Theodore Y. Ts'o" <>,
	Linus Torvalds <>,
	Andreas Dilger <>,
	Jan Kara <>,
	Linux Kernel Mailing List <>,
Subject: Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps
Date: Thu, 8 Oct 2020 15:38:58 -0700	[thread overview]
Message-ID: <20201008223858.GC45658@localhost> (raw)
In-Reply-To: <20201008175448.GA6532@magnolia>

On Thu, Oct 08, 2020 at 10:54:48AM -0700, Darrick J. Wong wrote:
> IMO, the prominent free software filesystem projects offer (at least)
> four layers of social structures to keep everyone on the same page,
> including people writing competing implementations.

(I certainly hope that this isn't a *competing* implementation. It's
more that it serves a different purpose than the existing tools.)

> The first is "Does
> everything we write still work with the kernel", which I guess you
> clearly did since you're complaining about this change in 5.9-rc2.


> The second layer is "Does it pass fsck?" which, given that you're asking
> for changes to e2fsck elsewhere in this thread and I couldn't figure out
> how to generate a shared-bitmaps ext4 fs that didn't also cause e2fsck
> to complain about the overlap doesn't make me confident that you went
> beyond the first step before shipping something.

I did ensure that, other than the one top-level complaint that the
bitmaps overlapped, I got no complaints from e2fsck. I also confirmed
that with a patch to that one item, e2fsck passed with no issues.

> The third layer is consulting the ondisk format documentation to see if
> it points out anything that the kernel or fsck didn't notice.  That
> one's kind of on Ted for not updating Documentation/ to spell out what
> SHARED_BLOCKS actually means, but that just leads me to the fourth thing.

I've been making *extensive* use of Documentation/filesystems/ext4 by
the way, and I greatly appreciate it. I know you've done a ton of work
in that area.

> The fourth layer is emailing the list to ask "Hey, I was thinking of
> ___, does anyone see a problem with my interpretation?".  That clearly
> hasn't been done for shared bitmaps until now, which is all the more
> strange since you /did/ ask linux-ext4 about the inline data topic
> months ago.

That one was on me, you're right. Because Ted is the maintainer of
e2fsprogs in Debian, and conversations about ext4 often happen in the
Debian BTS, reporting a bug on e2fsprogs there felt like starting an
upstream conversation. That was my mistake; in the future, I'll make
sure that things make it to linux-ext4. I already did so for , and I
*should* have gone back and done so for the shared bitmap blocks issue.

> ext* and XFS have been around for 25 years.  The software validation of
> both is imperfect and incomplete because the complexity of the ondisk
> formats is vast and we haven't caught up with the technical debt that
> resulted from not writing rigorous checks that would have been very
> difficult with mid-90s hardware.  I know, because I've been writing
> online checking for XFS and have seen the wide the gap between what the
> existing software looks for and what the ondisk format documentation
> allows.
> The only chance that we as a community have at managing the complexity
> of a filesystem is to use those four tools I outlined above to try to
> keep the mental models of everyone who participates in close enough
> alignment.  That's how we usually avoid discussions that end in rancor
> and confusion.
> That was a very long way of reiterating "If you're going to interpret
> aspects of the software, please come talk to us before you start writing
> code".  That is key to ext4's long history of compatibility, because a
> project cannot maintain continuity when actors develop conflicting code
> in the shadows.  Look at all the mutations FAT and UFS that have
> appeared over the years-- the codebases became a mess as a result.

Understood, agreed, and acknowledged. I'll make sure that any future
potentially "adventurous" filesystem experiments get discussed on
linux-ext4 first, not just in a distro bugtracker with a limited

> > > the head", and continued with the notion that anything other than
> > > e2fsprogs making something to be mounted by mount(2) and handled by
> > > fs/ext4 is being "inflicted", and if the goal didn't still seem to be
> > > "how do we make it go away so that only e2fsprogs and the kernel ever
> > > touch ext4". I started this thread because I'd written some userspace
> > > code, a new version of the kernel made that userspace code stop working,
> > > so I wanted to report that the moment I'd discovered that, along with a
> > > potential way to address it with as little disrupton to ext4 as
> > > possible.
> Ted: I don't think it's a good idea to make SHARED_BLOCKS disable block
> validity checking by default.  You could someday enable users to write
> to block-sharing files by teaching ext4 how to COW, at which point you'd
> need correct bitmaps and block validity to prevent accidental overwrite
> of critical metadata.  At that point you'd either be stuck with the
> precedent that SHARED_BLOCKS implies noblock_validity, or you'd end up
> breaking Josh's images again.

That's a fair point (though I think a writable COW version of
SHARED_BLOCKS would need a different flag). It'd make more sense to key
this logic off of RO_COMPAT_READONLY or similar. But even better:

> "noblock_validity" in the superblock mount options field of the images
> you create.

Yeah, I can do that. That's a much better solution, thank you. It would
have been problematic to have to change the userspace that mounts the
filesystem to pass new mount options ("noblock_validity") for the new
kernel. But if I can embed it in the filesystem, that'll work.

I'll do that, and please feel free to drop the original proposed patch
as it's no longer needed. Thanks, Darrick.

- Josh Triplett

  reply	other threads:[~2020-10-08 22:39 UTC|newest]

Thread overview: 35+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-04 23:17 Linux 5.9-rc8 Linus Torvalds
2020-10-05  8:14 ` ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps Josh Triplett
2020-10-05  9:46   ` Jan Kara
2020-10-05 10:16     ` Josh Triplett
2020-10-05 16:19       ` Jan Kara
2020-10-05 16:20   ` Jan Kara
2020-10-05 17:36   ` Darrick J. Wong
2020-10-06  0:04     ` Theodore Y. Ts'o
2020-10-06  0:32     ` Josh Triplett
2020-10-06  2:51       ` Darrick J. Wong
2020-10-06  3:18         ` Theodore Y. Ts'o
2020-10-06  5:03           ` Josh Triplett
2020-10-06  6:03             ` Josh Triplett
2020-10-06 13:35             ` Theodore Y. Ts'o
2020-10-07  8:03               ` Josh Triplett
2020-10-07 14:32                 ` Theodore Y. Ts'o
2020-10-07 20:14                   ` Josh Triplett
2020-10-08  2:10                     ` Theodore Y. Ts'o
2020-10-08 17:54                       ` Darrick J. Wong
2020-10-08 22:38                         ` Josh Triplett [this message]
2020-10-09  2:54                           ` Darrick J. Wong
2020-10-09 19:08                             ` Josh Triplett
2020-10-08 22:22                       ` Josh Triplett
2020-10-09 14:37                         ` Theodore Y. Ts'o
2020-10-09 20:30                           ` Josh Triplett
2021-01-10 18:41                           ` Malicious fs images was " Pavel Machek
2021-01-11 18:51                             ` Darrick J. Wong
2021-01-11 19:39                               ` Eric Biggers
2021-01-12 21:43                             ` Theodore Ts'o
2021-01-12 22:28                               ` Pavel Machek
2021-01-13  5:09                                 ` Theodore Ts'o
2020-10-08  2:57                     ` Andreas Dilger
2020-10-08 19:12                       ` Josh Triplett
2020-10-08 19:25                         ` Andreas Dilger
2020-10-08 22:28                           ` Josh Triplett

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201008223858.GC45658@localhost \ \ \ \ \ \ \ \ \
    --subject='Re: ext4 regression in v5.9-rc2 from e7bfb5c9bb3d on ro fs with overlapped bitmaps' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox