[LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[Josef Bacik]

Hello,

There have been a few common themes that have come up over the years that I feel
like we should address.

1. Adding new file systems.  This is a long and painful process, and I think it
   should be more painful in order to encourage more FUSE use cases.  However, a
   lot of the reason it's painful is because it's a "send to Linus and see what
   happens" sort of process.  Obviously a lot of us get a say, but there's a
   sort of arbitrary point where it becomes "well send a PR to Linus and see
   what he thinks."  This is annoying for people who review stuff who may have
   legitimate concerns, and it's annoying for new file systems who aren't sure
   what feedback they're supposed to take seriously and what feedback is safe to
   ignore.

2. Removing file systems.  We've gotten some good guidance from Greg and others
   on this, but this still becomes a thing where nobody feels particularly
   empowered to send the first patch of actually removing a file system.  In
   super obvious cases it's easier, but there's a lot of non-obvious cases where
   we kind of sit here and talk about it without doing anything.

3. API changes.  Sometimes we make API changes in the core code and then
   provided helpers for the other file systems to use until they're converted,
   and that long tail goes on forever.  We generally avoid doing work that
   touches all the file systems because we have to coordinate with at least 4
   major trees. I'm particularly guilty of this one, I didn't even notice when
   the new mount API went in, and then I wasn't sufficiently motivated to work
   on it until it intersected with some other work I was doing.  I was easily
   halfway through the work when I found out that Christian had done all of the
   work for us previously, which brings me to #4.

4. We all have our own ways of doing things, but we're all really similar at the
   same time.  In btrfs land we prefer small, bitesize patches.  This makes it
   easier for review, easier for bisecting, etc.  This exists because we take in
   3x the number of changes as any other file system, we have been bitten
   several times by some 6'4" jackass with a swearing problem with a 6000 line
   patch with an unhelpful changelog.  I've had developers get frustrated with
   our way of running our tree because it's setup differently than others. At
   the end of the day however a lot of our policies exist to make it as easy as
   possible for everybody involved to understand what is going on, and to make
   sure we don't repeat previous mistakes.  At the same time we all do a lot of
   the same things, emphasize patch review and testing.

There are other related problems, but these are the big ones as I see them.

I would like to propose we organize ourselves more akin to the other large
subsystems.  We are one of the few where everybody sends their own PR to Linus,
so oftentimes the first time we're testing eachothers code is when we all rebase
our respective trees onto -rc1.  I think we could benefit from getting more
organized amongst ourselves, having a single tree we all flow into, and then
have that tree flow into Linus.

I'm also not a fan of single maintainers in general, much less for this large of
an undertaking.  I would also propose that we have a maintainership group where
we rotate the responsibilities of the mechanics of running a tree like this.
I'm nothing if not unreliable so I wouldn't be part of this group per se, but I
don't think we should just make Christian do it.  This would be a big job, and
it would need to be shared.

I would also propose that along with this single tree and group maintainership
we organize some guidelines about the above problems and all collectively agree
on how we're going to address them.  Having clear guidelines for adding new file
systems, clear guidelines for removing them.  Giving developers the ability to
make big API changes outside of the individual file systems trees to make it
easier to get things merged instead of having to base against 4 or 5 different
trees.  Develop some guidelines about how we want patches to look, how we want
testing to be done, etc. so people can move through our different communities
and not have drastically different experiences.

This is a massive proposal, and not one that we're going to be able to nail down
and implement quickly or easily.  But I think in the long term it'll make
working in our community simpler, more predictable, and less frustrating for
everybody.  Thanks,

Josef

Re: [LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[Matthew Wilcox]

On Tue, Apr 16, 2024 at 02:04:14PM -0400, Josef Bacik wrote:
> I would like to propose we organize ourselves more akin to the other
> large subsystems.  We are one of the few where everybody sends their
> own PR to Linus, so oftentimes the first time we're testing eachothers
> code is when we all rebase our respective trees onto -rc1.  I think
> we could benefit from getting more organized amongst ourselves, having
> a single tree we all flow into, and then have that tree flow into Linus.

This sounds like a great idea to me.  As someone who does a lot of
changes that touch a lot of filesystems, I'd benefit from this model.
It's very frustrating to be told "Oh, submit patches against tree X
which isn't included in linux-next".

A potential downside is that it increases the risk of an ntfs3 style
disaster where the code is essentially dumped on all other fs maintainers.
But I like the idea of a maintainer group which allows people to slide
in and out of the "patch pumpkin" role.  Particularly if it lets more
junior developers take a turn at wrangling patches.

Re: [Lsf-pc] [LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[Jan Kara]

Hello!

On Tue 16-04-24 14:04:14, Josef Bacik wrote:
> I would like to propose we organize ourselves more akin to the other large
> subsystems.  We are one of the few where everybody sends their own PR to Linus,
> so oftentimes the first time we're testing eachothers code is when we all rebase
> our respective trees onto -rc1.  I think we could benefit from getting more
> organized amongst ourselves, having a single tree we all flow into, and then
> have that tree flow into Linus.
> 
> I'm also not a fan of single maintainers in general, much less for this large of
> an undertaking.  I would also propose that we have a maintainership group where
> we rotate the responsibilities of the mechanics of running a tree like this.
> I'm nothing if not unreliable so I wouldn't be part of this group per se, but I
> don't think we should just make Christian do it.  This would be a big job, and
> it would need to be shared.
> 
> I would also propose that along with this single tree and group maintainership
> we organize some guidelines about the above problems and all collectively agree
> on how we're going to address them.  Having clear guidelines for adding new file
> systems, clear guidelines for removing them.  Giving developers the ability to
> make big API changes outside of the individual file systems trees to make it
> easier to get things merged instead of having to base against 4 or 5 different
> trees.  Develop some guidelines about how we want patches to look, how we want
> testing to be done, etc. so people can move through our different communities
> and not have drastically different experiences.
> 
> This is a massive proposal, and not one that we're going to be able to nail down
> and implement quickly or easily.  But I think in the long term it'll make
> working in our community simpler, more predictable, and less frustrating for
> everybody.  Thanks,

I think this is good discussion for the FS track. I'd be certainly
interested in this.

									Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[David Howells]

Matthew Wilcox <willy@infradead.org> wrote:

> On Tue, Apr 16, 2024 at 02:04:14PM -0400, Josef Bacik wrote:
> > I would like to propose we organize ourselves more akin to the other
> > large subsystems.  We are one of the few where everybody sends their
> > own PR to Linus, so oftentimes the first time we're testing eachothers
> > code is when we all rebase our respective trees onto -rc1.  I think
> > we could benefit from getting more organized amongst ourselves, having
> > a single tree we all flow into, and then have that tree flow into Linus.
> 
> This sounds like a great idea to me.  As someone who does a lot of
> changes that touch a lot of filesystems, I'd benefit from this model.
> It's very frustrating to be told "Oh, submit patches against tree X
> which isn't included in linux-next".
> 
> A potential downside is that it increases the risk of an ntfs3 style
> disaster where the code is essentially dumped on all other fs maintainers.
> But I like the idea of a maintainer group which allows people to slide
> in and out of the "patch pumpkin" role.  Particularly if it lets more
> junior developers take a turn at wrangling patches.

Would it make sense to have an MM + FS tree, given that a lot of MM changes
affect filesystems too?  Or, at least, a common branch between the MM and FS
trees for things that affect both?

David

Re: [LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[Christian Brauner]

On Tue, Apr 16, 2024 at 02:04:14PM -0400, Josef Bacik wrote:
> Hello,
> 
> There have been a few common themes that have come up over the years that I feel
> like we should address.
> 
> 1. Adding new file systems.  This is a long and painful process, and I think it
>    should be more painful in order to encourage more FUSE use cases.  However, a
>    lot of the reason it's painful is because it's a "send to Linus and see what
>    happens" sort of process.  Obviously a lot of us get a say, but there's a
>    sort of arbitrary point where it becomes "well send a PR to Linus and see
>    what he thinks."  This is annoying for people who review stuff who may have
>    legitimate concerns, and it's annoying for new file systems who aren't sure
>    what feedback they're supposed to take seriously and what feedback is safe to
>    ignore.
> 
> 2. Removing file systems.  We've gotten some good guidance from Greg and others
>    on this, but this still becomes a thing where nobody feels particularly
>    empowered to send the first patch of actually removing a file system.  In
>    super obvious cases it's easier, but there's a lot of non-obvious cases where
>    we kind of sit here and talk about it without doing anything.
> 
> 3. API changes.  Sometimes we make API changes in the core code and then
>    provided helpers for the other file systems to use until they're converted,
>    and that long tail goes on forever.  We generally avoid doing work that
>    touches all the file systems because we have to coordinate with at least 4
>    major trees. I'm particularly guilty of this one, I didn't even notice when
>    the new mount API went in, and then I wasn't sufficiently motivated to work
>    on it until it intersected with some other work I was doing.  I was easily
>    halfway through the work when I found out that Christian had done all of the
>    work for us previously, which brings me to #4.
> 
> 4. We all have our own ways of doing things, but we're all really similar at the
>    same time.  In btrfs land we prefer small, bitesize patches.  This makes it
>    easier for review, easier for bisecting, etc.  This exists because we take in
>    3x the number of changes as any other file system, we have been bitten
>    several times by some 6'4" jackass with a swearing problem with a 6000 line
>    patch with an unhelpful changelog.  I've had developers get frustrated with
>    our way of running our tree because it's setup differently than others. At
>    the end of the day however a lot of our policies exist to make it as easy as
>    possible for everybody involved to understand what is going on, and to make
>    sure we don't repeat previous mistakes.  At the same time we all do a lot of
>    the same things, emphasize patch review and testing.
> 
> There are other related problems, but these are the big ones as I see them.
> 
> I would like to propose we organize ourselves more akin to the other large
> subsystems.  We are one of the few where everybody sends their own PR to Linus,
> so oftentimes the first time we're testing eachothers code is when we all rebase
> our respective trees onto -rc1.  I think we could benefit from getting more
> organized amongst ourselves, having a single tree we all flow into, and then
> have that tree flow into Linus.
> 
> I'm also not a fan of single maintainers in general, much less for this large of
> an undertaking.  I would also propose that we have a maintainership group where
> we rotate the responsibilities of the mechanics of running a tree like this.
> I'm nothing if not unreliable so I wouldn't be part of this group per se, but I
> don't think we should just make Christian do it.  This would be a big job, and
> it would need to be shared.

Context here is that some of us discussed this idea off-list a while
ago. Various aspects of this sounded appealing to me.

One of my crucial points is that the current fs/ maintainers are
responsible and will decide on the addition of any new maintainers. As
of v6.8 Jan Kara has been added as co-maintainer of fs/.

The number of trees merged into vfs/vfs.git has slowly increased.
There's already a few people that send pulls and there'll be another
tree addition during this cycle.

New subtrees are also now formalized in MAINTAINERS similar to other
hierarchical trees. New trees that are added and that get sent to fs/
are marked with FILESYSTEMS [subsystem] and we've gotten a few of those
last year actually.

I'm generally supportive of moving into this direction. But I have no
interest in this being decided over the heads of the current fs/
maintainers (or individual filesystem maintainers for that matter).

This should be voluntary. Individual fses or subsystems are welcome to
send pulls. If there's bigger fses that want to start doing this then a
conversations on expectations from both sides needs to happen. If we do
this then we start with one or two non-trivial fses and then go from
there. And it also needs to work for Linus obviously.

Rotating tree management for a cycle seems fine to me once we've reached
a point where it's really more than can be handled. Might actually mean
one can get some work on something bigger done. Writing pidfs on the
side while maintaining was certainly fun.

> I would also propose that along with this single tree and group maintainership
> we organize some guidelines about the above problems and all collectively agree
> on how we're going to address them.  Having clear guidelines for adding new file
> systems, clear guidelines for removing them.  Giving developers the ability to
> make big API changes outside of the individual file systems trees to make it
> easier to get things merged instead of having to base against 4 or 5 different
> trees.  Develop some guidelines about how we want patches to look, how we want
> testing to be done, etc. so people can move through our different communities
> and not have drastically different experiences.
> 
> This is a massive proposal, and not one that we're going to be able to nail down
> and implement quickly or easily.  But I think in the long term it'll make
> working in our community simpler, more predictable, and less frustrating for
> everybody.  Thanks,
> 
> Josef

Re: [LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[Kent Overstreet]

On Tue, Apr 16, 2024 at 09:34:47PM +0100, Matthew Wilcox wrote:
> On Tue, Apr 16, 2024 at 02:04:14PM -0400, Josef Bacik wrote:
> > I would like to propose we organize ourselves more akin to the other
> > large subsystems.  We are one of the few where everybody sends their
> > own PR to Linus, so oftentimes the first time we're testing eachothers
> > code is when we all rebase our respective trees onto -rc1.  I think
> > we could benefit from getting more organized amongst ourselves, having
> > a single tree we all flow into, and then have that tree flow into Linus.
> 
> This sounds like a great idea to me.  As someone who does a lot of
> changes that touch a lot of filesystems, I'd benefit from this model.
> It's very frustrating to be told "Oh, submit patches against tree X
> which isn't included in linux-next".

I think an even better starting point would just be (more) common test
infrastructure. We've already got fstests, what we need is a shared
cluster (two racks of machines?) that is dedicated to automated testing
on _any_ kernel filesystem.

I've got the code for this all ready to go, as soon as someone is
willing to pony up on hardware.

That would mean people like Willy who are doing cross filesystem testing
would have a _lot_ less manual work to do, and having a cluster that
watches our git branches and kicks off tests when someone pushes (i.e.
what I already have, just on a bigger scale) would mean that we'd be
full test suite results back in 5-10 minutes after writing the code and
pushing. That sort of thing is amazing for productivity... no more
sitting around twiddling thumbs waiting for the evening test run...

Re: [LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[Darrick J. Wong]

On Tue, Apr 16, 2024 at 02:04:14PM -0400, Josef Bacik wrote:
> Hello,
> 
> There have been a few common themes that have come up over the years that I feel
> like we should address.
> 
> 1. Adding new file systems.  This is a long and painful process, and I think it
>    should be more painful in order to encourage more FUSE use cases.  However, a
>    lot of the reason it's painful is because it's a "send to Linus and see what
>    happens" sort of process.  Obviously a lot of us get a say, but there's a
>    sort of arbitrary point where it becomes "well send a PR to Linus and see
>    what he thinks."  This is annoying for people who review stuff who may have
>    legitimate concerns, and it's annoying for new file systems who aren't sure
>    what feedback they're supposed to take seriously and what feedback is safe to
>    ignore.

What if we used the time at LSF to hammer out some guideline docs that
define one or two tiers of support for filesystems and what qualities
that filesystem and its project must have to attain that tier.

Tier 1 would be (for example) ext/xfs/btrfs/bcachefs, and anything
intended to be used as daily driver filesystem on Linux.  Regular
testing via fstests and contributions to for-next are required.  There
ought to be some defined roles (e.g. release manager, QA leads, various
levels of developers, documenters, LTS sustaining, bug triager, etc.).
The roles can be assigned to a group of people who each can work on the
code at least 50% of the time.

Tier 2 would be the smaller fs projects (jfs, etc).  Periodic testing is
still required, but there wouldn't need to be as much structure as tier
1.  Anyone building a hardened linux distribution would likely not
enable the tier 2 filesystems.  Perhaps the more general distributions
could devise a way to ask the device's user if they really want to mount
a tier 2 fs on its storage.

Tier 3 is bitrotting in the kernel and will get kicked out if it gets
in the way of a treewide cleanup or suffers major security problems.

(This isn't a fleshed out proposal, merely my reflections on the way
things are more or less right now.)

Then we say that a filesystem must be a good fit with the goals and
requirements of tiers 1 or 2 to get added, and will be deprecated and
removed if they fall out of tier 2 for more than a year.

(Oh hey, you advocate for something like this below!  Cool!)

> 2. Removing file systems.  We've gotten some good guidance from Greg and others
>    on this, but this still becomes a thing where nobody feels particularly
>    empowered to send the first patch of actually removing a file system.  In
>    super obvious cases it's easier, but there's a lot of non-obvious cases where
>    we kind of sit here and talk about it without doing anything.
> 
> 3. API changes.  Sometimes we make API changes in the core code and then
>    provided helpers for the other file systems to use until they're converted,
>    and that long tail goes on forever.

You mean like buffer heads? ;)

>                                         We generally avoid doing work that
>    touches all the file systems because we have to coordinate with at least 4
>    major trees.

I don't like adding user interfaces -- we spend a /lot/ of time arguing
about minor points of new ioctls and syscalls, not really enough time
documenting or writing tests for them, and then after v20 gets posted it
turns into "send a PR to Linus and see if he complains".

From my observation, people (myself included) act like they're afraid
that an interface is baked in stone forever and therefore everything has
to be perfect.  Nobody seems to think it practicable to have a kernel
call nursery to try out new functionality that can be withdrawn or
changed ... but if it's critical to get it right on the first try, why
isn't there a bunch of guidelines and a working group to support this?

>                 I'm particularly guilty of this one, I didn't even notice when
>    the new mount API went in, and then I wasn't sufficiently motivated to work
>    on it until it intersected with some other work I was doing.  I was easily
>    halfway through the work when I found out that Christian had done all of the
>    work for us previously, which brings me to #4.

<rant>
The transition to the new mount API has not been managed.  The author
wrote the initial patches, converted a few filesystems, and then moved
on to other things without communicating much of a plan to get the rest
of the filesystems converted.  Now his manager(!) seems to be doing the
rest of the work himself.  There still aren't even manual pages for the
new syscalls, and it's been five years since the code got merged.
</rant>

> 4. We all have our own ways of doing things, but we're all really similar at the
>    same time.  In btrfs land we prefer small, bitesize patches.  This makes it
>    easier for review, easier for bisecting, etc.  This exists because we take in
>    3x the number of changes as any other file system, we have been bitten
>    several times by some 6'4" jackass with a swearing problem with a 6000 line
>    patch with an unhelpful changelog.  I've had developers get frustrated with

At the opposite extreme, we have a 6'0" jackass with a swearing problem
who sends 600 patches in a week with changelogs that are bitrotting
because he can't keep track of all that, and review is painfully slow. :P

>    our way of running our tree because it's setup differently than others. At
>    the end of the day however a lot of our policies exist to make it as easy as
>    possible for everybody involved to understand what is going on, and to make
>    sure we don't repeat previous mistakes.  At the same time we all do a lot of
>    the same things, emphasize patch review and testing.
> 
> There are other related problems, but these are the big ones as I see them.
> 
> I would like to propose we organize ourselves more akin to the other large

IOWs, this is "5. Maintaining file systems."? :)

> subsystems.  We are one of the few where everybody sends their own PR to Linus,
> so oftentimes the first time we're testing eachothers code is when we all rebase
> our respective trees onto -rc1.  I think we could benefit from getting more
> organized amongst ourselves, having a single tree we all flow into, and then
> have that tree flow into Linus.

I think it takes too long to get feedback on patches.  There's too much
knowledge that is either locked up in peoples' brains or available only
after deep studying of a lot of kernel code, which means that junior
engineers tell me they don't see any point in reviewing patches because
someone senior will see the flaws in a given patch that they don't.  I
think that reflects a need for better mentorship so that those junior
people can get the experience they need to become senior.

(At the same time, I'm a senior person, and I'm so burnt out that I
would not and do not _make_ a good mentor.)

> I'm also not a fan of single maintainers in general, much less for this large of
> an undertaking.  I would also propose that we have a maintainership group where
> we rotate the responsibilities of the mechanics of running a tree like this.

+1 to this.  Single maintainership is a *very bad idea* for complex
codebases like filesystems.  xfs have burned through/out two maintainers
in 10 years; I'm surprised that btrfs and ext still have theirs.

Back when I was maintainer of fs/iomap/, I was careful to push all the
changes for the next cycle into for-next during the week after -rc4 so
that all five filesystems that use it would have a month or so where the
-next testing would (hopefully) shake out the problems.  That did not
make me popular.

> I'm nothing if not unreliable so I wouldn't be part of this group per se, but I
> don't think we should just make Christian do it.  This would be a big job, and
> it would need to be shared.

Heh.

> I would also propose that along with this single tree and group maintainership
> we organize some guidelines about the above problems and all collectively agree
> on how we're going to address them.  Having clear guidelines for adding new file
> systems, clear guidelines for removing them.  Giving developers the ability to
> make big API changes outside of the individual file systems trees to make it
> easier to get things merged instead of having to base against 4 or 5 different
> trees.  Develop some guidelines about how we want patches to look, how we want
> testing to be done, etc. so people can move through our different communities
> and not have drastically different experiences.

Can we also find a way to attract some people who specialize in fs
tooling so that the rest of us can retire our homebrew gluepots of bash
scripts (speaking only for myself, obviously).

> This is a massive proposal, and not one that we're going to be able to nail down
> and implement quickly or easily.  But I think in the long term it'll make
> working in our community simpler, more predictable, and less frustrating for
> everybody.  Thanks,

Frankly I would be totally happy if we had a driver to cache file
mappings provided through a fuse-like interface and feed them to iomap
as needed.  Then we could eject most filesystms from the kernel and
never have to deal with it ever again.  That would probably suck hard
for most of the COW filesystems though...

--D

> 
> Josef
> 

Re: [LSF/MM/BPF TOPIC] Changing how we do file system maintenance

[Steve French]

On Wed, Apr 17, 2024 at 4:12 PM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Tue, Apr 16, 2024 at 09:34:47PM +0100, Matthew Wilcox wrote:
> > On Tue, Apr 16, 2024 at 02:04:14PM -0400, Josef Bacik wrote:
> > > I would like to propose we organize ourselves more akin to the other
> > > large subsystems.  We are one of the few where everybody sends their
> > > own PR to Linus, so oftentimes the first time we're testing eachothers
> > > code is when we all rebase our respective trees onto -rc1.  I think
> > > we could benefit from getting more organized amongst ourselves, having
> > > a single tree we all flow into, and then have that tree flow into Linus.
> >
> > This sounds like a great idea to me.  As someone who does a lot of
> > changes that touch a lot of filesystems, I'd benefit from this model.
> > It's very frustrating to be told "Oh, submit patches against tree X
> > which isn't included in linux-next".
>
> I think an even better starting point would just be (more) common test
> infrastructure. We've already got fstests, what we need is a shared
> cluster (two racks of machines?) that is dedicated to automated testing
> on _any_ kernel filesystem.

Yes - this can be very helpful.  Although we have to deal with many types of
non-Linux servers with cifs.ko (MacOS, Visuality, Windows, Azure, NetApp etc.)
at least we have two Linux servers that could be used as test targets
easily (e.g. with localhost mounts).   A lot of what I struggle with though is
how to sanely automate testing of pull requests to branches (as large
projects like Samba) and the automated backports of fixes, and
handle good automated collection of traces/logs/dmesg when tests fail.

> I've got the code for this all ready to go, as soon as someone is
> willing to pony up on hardware.
>
> That would mean people like Willy who are doing cross filesystem testing
> would have a _lot_ less manual work to do, and having a cluster that
> watches our git branches and kicks off tests when someone pushes (i.e.
> what I already have, just on a bigger scale) would mean that we'd be
> full test suite results back in 5-10 minutes after writing the code and
> pushing. That sort of thing is amazing for productivity... no more
> sitting around twiddling thumbs waiting for the evening test run...

I agree ...


-- 
Thanks,

Steve