Getting a new fs in the kernel

Getting a new fs in the kernel

[Amy Parker]

Kernel development newcomer here. I've begun creating a concept for a
new filesystem, and ideally once it's completed, rich, and stable I'd
try to get it into the kernel.

What would be the process for this? I'd assume a patch sequence, but
they'd all be dependent on each other, and sending in tons of
dependent patches doesn't sound like a great idea. I've seen requests
for pulls, but since I'm new here I don't really know what to do.

Thank you for guidance!

Best regards,
Amy Parker
she/her/hers

Re: Getting a new fs in the kernel

[Andreas Dilger]

On Jan 26, 2021, at 09:25, Amy Parker <enbyamy@gmail.com> wrote:
> 
> Kernel development newcomer here. I've begun creating a concept for a
> new filesystem, and ideally once it's completed, rich, and stable I'd
> try to get it into the kernel.

Hello Amy, and welcome. 

I guess the first question to ask is what would be unique and valuable
about the new filesystem compared to existing filesystems in the kernel?

Given that maintaining a new filesystem adds ongoing maintenance
efforts, there has to be some added value before it would be accepted.
Also, given that filesystems are storing critical data for users, and
problems in the code can lead to data loss that can't be fixed by a reboot,
like many other software bugs, it takes a very long time for filesystems
to become stable enough for general use (the general rule of thumb is 10
years before a new filesystem is stable enough for general use). 

Often, if you have ideas for new functionality, it makes more sense to
add this into the framework of an existing filesystem (e.g. data verity,
data encryption, metadata checksum, DAX, etc. were all added to ext4).

Not only does this simplify efforts in terms of ongoing maintenance, but
it also means many more users will benefit from your development effort
in a much shorter timeframe. Otherwise, users would have to stop
using their existing filesystem before
they started using yours, and that is
a very slow process, because your filesystem would have to be much
better at *something* before they would make that switch. 

> What would be the process for this? I'd assume a patch sequence, but
> they'd all be dependent on each other, and sending in tons of
> dependent patches doesn't sound like a great idea. I've seen requests
> for pulls, but since I'm new here I don't really know what to do.

Probably the first step, before submitting any patches, would be to
provide a description of your work, and how it improves on the current
filesystems in the kernel. It may be that there are existing projects that
duplicate this effort, and combining resources will result in a 100% done
filesystem rather than two 80% done
projects...

Note that I don't want to discourage you from participating in the Linux
filesystem development community, but there are definitely considerations
going both ways wrt. accepting a new filesystem into the kernel. It may be
that your ideas, time, and efforts are better spent in contributing to an
exiting project.  It may also be that you have something groundbreaking
work, and I look forward to reading about what that is. 

Cheers, Andreas

Re: Getting a new fs in the kernel

[Amy Parker]

On Tue, Jan 26, 2021 at 9:15 AM Andreas Dilger <adilger@dilger.ca> wrote:
>
> On Jan 26, 2021, at 09:25, Amy Parker <enbyamy@gmail.com> wrote:
> >
> > Kernel development newcomer here. I've begun creating a concept for a
> > new filesystem, and ideally once it's completed, rich, and stable I'd
> > try to get it into the kernel.
>
> Hello Amy, and welcome.
>
> I guess the first question to ask is what would be unique and valuable
> about the new filesystem compared to existing filesystems in the kernel?

Currently planning those bits out.

>
> Given that maintaining a new filesystem adds ongoing maintenance
> efforts, there has to be some added value before it would be accepted.
> Also, given that filesystems are storing critical data for users, and
> problems in the code can lead to data loss that can't be fixed by a reboot,
> like many other software bugs, it takes a very long time for filesystems
> to become stable enough for general use (the general rule of thumb is 10
> years before a new filesystem is stable enough for general use).

Yeah, I understood that. Wasn't expecting this to go quickly or
anything, just wanted to know for the future.

>
> Often, if you have ideas for new functionality, it makes more sense to
> add this into the framework of an existing filesystem (e.g. data verity,
> data encryption, metadata checksum, DAX, etc. were all added to ext4).

Been considering doing this too.

>
> Not only does this simplify efforts in terms of ongoing maintenance, but
> it also means many more users will benefit from your development effort
> in a much shorter timeframe. Otherwise, users would have to stop
> using their existing filesystem before
> they started using yours, and that is
> a very slow process, because your filesystem would have to be much
> better at *something* before they would make that switch.

True, alright.

>
> > What would be the process for this? I'd assume a patch sequence, but
> > they'd all be dependent on each other, and sending in tons of
> > dependent patches doesn't sound like a great idea. I've seen requests
> > for pulls, but since I'm new here I don't really know what to do.
>
> Probably the first step, before submitting any patches, would be to
> provide a description of your work, and how it improves on the current
> filesystems in the kernel. It may be that there are existing projects that
> duplicate this effort, and combining resources will result in a 100% done
> filesystem rather than two 80% done
> projects...

Alright, if I continue on with this I'll do that.

>
> Note that I don't want to discourage you from participating in the Linux
> filesystem development community, but there are definitely considerations
> going both ways wrt. accepting a new filesystem into the kernel. It may be
> that your ideas, time, and efforts are better spent in contributing to an
> exiting project.  It may also be that you have something groundbreaking
> work, and I look forward to reading about what that is.

Alright, thank you so much!

Best regards,
Amy Parker
(she/her/hers)

Re: Getting a new fs in the kernel

[Chaitanya Kulkarni]

Amy,

On 1/26/21 8:26 AM, Amy Parker wrote:
> Kernel development newcomer here. I've begun creating a concept for a
> new filesystem, and ideally once it's completed, rich, and stable I'd
> try to get it into the kernel.
>
> What would be the process for this? I'd assume a patch sequence, but
> they'd all be dependent on each other, and sending in tons of
> dependent patches doesn't sound like a great idea. I've seen requests
> for pulls, but since I'm new here I don't really know what to do.
>
> Thank you for guidance!
>
> Best regards,
> Amy Parker
> she/her/hers
>
From what I've seen you can post the long patch-series as an RFC and get the

discussion started.

The priority should be ease of review and not the total patch-count.

Re: Getting a new fs in the kernel

[Amy Parker]

On Tue, Jan 26, 2021 at 11:06 AM Chaitanya Kulkarni
<Chaitanya.Kulkarni@wdc.com> wrote:
>
> Amy,
>
> On 1/26/21 8:26 AM, Amy Parker wrote:
> > Kernel development newcomer here. I've begun creating a concept for a
> > new filesystem, and ideally once it's completed, rich, and stable I'd
> > try to get it into the kernel.
> >
> > What would be the process for this? I'd assume a patch sequence, but
> > they'd all be dependent on each other, and sending in tons of
> > dependent patches doesn't sound like a great idea. I've seen requests
> > for pulls, but since I'm new here I don't really know what to do.
> >
> > Thank you for guidance!
> >
> > Best regards,
> > Amy Parker
> > she/her/hers
> >
> From what I've seen you can post the long patch-series as an RFC and get the
>
> discussion started.
>
> The priority should be ease of review and not the total patch-count.

Alright, thank you for the advice, Chaitanya!

Best regards,
Amy Parker
(she/her/hers)

Re: Getting a new fs in the kernel

[Theodore Ts'o]

On Tue, Jan 26, 2021 at 07:06:55PM +0000, Chaitanya Kulkarni wrote:
> From what I've seen you can post the long patch-series as an RFC and get the
> 
> discussion started.
> 
> The priority should be ease of review and not the total patch-count.

File systems are also complicated enough that it's useful to make the
patches available via a git repo, and it's highly recommended that you
are rebasing it against the latest kernel on a regular basis.

I also strongly recommend that once you get something that mostly
works, that you start doing regression testing of the file system.
Most of the major file systems in Linux use xfstests for their
testing.  One of the things that I've done is to package up xfstests
as a test appliance, suitable for running under KVM or using Google
Compute Engine, as a VM, to make it super easy for people to run
regression tests.  (One of my original goals for packaging it up was
to make it easy for graduate students who were creating research file
systems to try running regression tests so they could find potential
problems --- and understand how hard it is to make a robust,
production-ready file system, by giving them a realtively well
documented, turn-key system for running file system regression tests.)

For more information, see:

    https://thunk.org/gce-xfstests
    https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md
    https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-xfstests.md
    https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md

The final thing I'll point out is that file system development is a
team sport.  Industry estimates are that it takes between 50 and 200
person-years to create a production-ready, general purpose enterprise
file system.  For example, ZFS took seven years to develop, starting
with a core team of 4, and growing to over 14 developers by the time
it was announced.  And that didn't include all of the QA, release
engineering, testers, performance engineers, to get it integrated into
the Solaris product.  Even after it was announced, it was a good four
years before customers trusted it for production workloads.

If you look at the major file systems in Linux: ext4, xfs, btrfs,
f2fs, etc., you'll find that none of them are solo endeavors, and all
of them have multiple companies who are employing the developers who
work on them.  Figuring out how to convince companies that there are
good business reasons for them to support the developers of your file
system is important, since in order to keep things going for the long
haul, it really needs to be more than a single person's hobby.

Good luck!

					- Ted

Re: Getting a new fs in the kernel

[Amy Parker]

On Tue, Jan 26, 2021 at 9:36 PM Theodore Ts'o <tytso@mit.edu> wrote:
>
> On Tue, Jan 26, 2021 at 07:06:55PM +0000, Chaitanya Kulkarni wrote:
> > From what I've seen you can post the long patch-series as an RFC and get the
> >
> > discussion started.
> >
> > The priority should be ease of review and not the total patch-count.
>
> File systems are also complicated enough that it's useful to make the
> patches available via a git repo, and it's highly recommended that you
> are rebasing it against the latest kernel on a regular basis.

Was already setting up some local git infrastructure for this.

>
> I also strongly recommend that once you get something that mostly
> works, that you start doing regression testing of the file system.

"'Regression testing? What's that? If it compiles, it is good; if it
boots up, it is perfect."

In all seriousness, though, yeah, already been planning for stuff like that.

> Most of the major file systems in Linux use xfstests for their
> testing.

Decently familiar with xfstests, used it for some previous change
testing I had to do.

> One of the things that I've done is to package up xfstests
> as a test appliance, suitable for running under KVM or using Google
> Compute Engine, as a VM, to make it super easy for people to run
> regression tests.  (One of my original goals for packaging it up was
> to make it easy for graduate students who were creating research file
> systems to try running regression tests so they could find potential
> problems --- and understand how hard it is to make a robust,
> production-ready file system, by giving them a realtively well
> documented, turn-key system for running file system regression tests.)
>
> For more information, see:
>
>     https://thunk.org/gce-xfstests
>     https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md
>     https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-xfstests.md
>     https://github.com/tytso/xfstests-bld/blob/master/Documentation/gce-xfstests.md

Thank you so much for that!

>
> The final thing I'll point out is that file system development is a
> team sport.  Industry estimates are that it takes between 50 and 200
> person-years to create a production-ready, general purpose enterprise
> file system.  For example, ZFS took seven years to develop, starting
> with a core team of 4, and growing to over 14 developers by the time
> it was announced.  And that didn't include all of the QA, release
> engineering, testers, performance engineers, to get it integrated into
> the Solaris product.  Even after it was announced, it was a good four
> years before customers trusted it for production workloads.

Wasn't expecting to do that completely solo, I get that it takes a
significant amount of people time to build something as important as a
production filesystem. Once I get some basic stuff lined out for it,
if I decide to continue, already working on getting people to help
assist with its development.

>
> If you look at the major file systems in Linux: ext4, xfs, btrfs,
> f2fs, etc., you'll find that none of them are solo endeavors, and all
> of them have multiple companies who are employing the developers who
> work on them.  Figuring out how to convince companies that there are
> good business reasons for them to support the developers of your file
> system is important, since in order to keep things going for the long
> haul, it really needs to be more than a single person's hobby.

Yeah, got that.

>
> Good luck!
>
>                                         - Ted

Thank you!

Best regards,
Amy Parker
(she/her/hers)

Re: Getting a new fs in the kernel

[Randy Dunlap]

On 1/26/21 8:23 AM, Amy Parker wrote:
> Kernel development newcomer here. I've begun creating a concept for a
> new filesystem, and ideally once it's completed, rich, and stable I'd
> try to get it into the kernel.

Ideally you would not wait until it's complete, rich, and stable.
RERO: release early, release often

> What would be the process for this? I'd assume a patch sequence, but
> they'd all be dependent on each other, and sending in tons of
> dependent patches doesn't sound like a great idea. I've seen requests
> for pulls, but since I'm new here I don't really know what to do.
> 
> Thank you for guidance!
> 
> Best regards,
> Amy Parker
> she/her/hers

-- 
~Randy
netiquette: https://people.kernel.org/tglx/notes-about-netiquette

Re: Getting a new fs in the kernel

[Matthew Wilcox]

On Tue, Jan 26, 2021 at 08:23:03AM -0800, Amy Parker wrote:
> Kernel development newcomer here. I've begun creating a concept for a
> new filesystem, and ideally once it's completed, rich, and stable I'd
> try to get it into the kernel.
> 
> What would be the process for this? I'd assume a patch sequence, but
> they'd all be dependent on each other, and sending in tons of
> dependent patches doesn't sound like a great idea. I've seen requests
> for pulls, but since I'm new here I don't really know what to do.

Hi Amy,

Writing a new filesystem is fun!  Everyone should do it.

Releasing a filesystem is gut-churning.  You're committing to a filesystem
format that has to be supported for ~ever.

Supporting a new filesystem is a weighty responsibility.  People are
depending on you to store their data reliably.  And they demand boring
and annoying features like xattrs, acls, support for time after 2038.

We have quite a lot of actively developed filesystems for users to choose
from already -- ext4, btrfs, xfs are the main three.  So you're going
to face a challenge persuading people to switch.

Finally, each filesystem represents a (small) maintainance burden to
people who need to make changes that cross all filesystems.  So it'd
be nice to have a good justification for why we should include that
cost.

Depending exactly what your concept is, it might make more sense to
make it part of an existing filesystem.  Or develop it separately
and have an existing filesystem integrate it.

Anyway, I've been at this for twenty years, so maybe I'm just grouchy
about new filesystems.  By all means work on it and see if it makes
sense, but there's a fairly low probability that it gets merged.

Re: Getting a new fs in the kernel

[Amy Parker]

On Tue, Jan 26, 2021 at 11:18 AM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Tue, Jan 26, 2021 at 08:23:03AM -0800, Amy Parker wrote:
> > Kernel development newcomer here. I've begun creating a concept for a
> > new filesystem, and ideally once it's completed, rich, and stable I'd
> > try to get it into the kernel.
> >
> > What would be the process for this? I'd assume a patch sequence, but
> > they'd all be dependent on each other, and sending in tons of
> > dependent patches doesn't sound like a great idea. I've seen requests
> > for pulls, but since I'm new here I don't really know what to do.
>
> Hi Amy,
>
> Writing a new filesystem is fun!  Everyone should do it.
>
> Releasing a filesystem is gut-churning.  You're committing to a filesystem
> format that has to be supported for ~ever.

I'm bored and need something to dedicate myself to as a long-term commitment.

>
> Supporting a new filesystem is a weighty responsibility.  People are
> depending on you to store their data reliably.  And they demand boring
> and annoying features like xattrs, acls, support for time after 2038.



>
> We have quite a lot of actively developed filesystems for users to choose
> from already -- ext4, btrfs, xfs are the main three.  So you're going
> to face a challenge persuading people to switch.
>

Yeah, understandable.

> Finally, each filesystem represents a (small) maintainance burden to
> people who need to make changes that cross all filesystems.  So it'd
> be nice to have a good justification for why we should include that
> cost.

Alright, I'll keep that in mind.

>
> Depending exactly what your concept is, it might make more sense to
> make it part of an existing filesystem.  Or develop it separately
> and have an existing filesystem integrate it.

That's what other people have suggested as well, so I'll start
considering trying to add any features I come up with into other
filesystems as well.

>
> Anyway, I've been at this for twenty years, so maybe I'm just grouchy
> about new filesystems.  By all means work on it and see if it makes
> sense, but there's a fairly low probability that it gets merged.

Alright. Thanks for the advice!

Best regards,
Amy Parker
(she/her/hers)

Re: Getting a new fs in the kernel

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

Hi!

> > Writing a new filesystem is fun!  Everyone should do it.
> >
> > Releasing a filesystem is gut-churning.  You're committing to a filesystem
> > format that has to be supported for ~ever.
> 
> I'm bored and need something to dedicate myself to as a long-term commitment.

You may want to look at "nvfs: a filesystem for persistent memory"...

Best regards,
									Pavel
									
-- 
http://www.livejournal.com/~pavelmachek

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]