fs/bcachefs/

fs/bcachefs/

[Kent Overstreet]

Hi stable team - please don't take patches for fs/bcachefs/ except from
myself; I'll be doing backports and sending pull requests after stuff
has been tested by my CI.

Thanks, and let me know if there's any other workflow things I should
know about
-Kent

Re: fs/bcachefs/

[Sasha Levin]

On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
>Hi stable team - please don't take patches for fs/bcachefs/ except from
>myself; I'll be doing backports and sending pull requests after stuff
>has been tested by my CI.
>
>Thanks, and let me know if there's any other workflow things I should
>know about

Sure, we can ignore fs/bcachefs/ patches.

Note that the proposed workflow would only work through patches coming
through your tree, but patches in other subsystems that could affect
bcachefs might break it in stable trees without being caught by your CI.

I'd recommend integrating your pre-release tests with something like
kernelci, which would let us catch bcachefs-affecting issues coming from
other sources.

What more, if we do the above, we could in the future avoid special
casing bcachefs :)

-- 
Thanks,
Sasha

Re: fs/bcachefs/

[Greg KH]

On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> Hi stable team - please don't take patches for fs/bcachefs/ except from
> myself; I'll be doing backports and sending pull requests after stuff
> has been tested by my CI.

Now done:
	https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?id=9bf1f8f1ca9ae53cf3bc8781e4efdb6ebaee70db

We will ignore it for any "Fixes:" tags, or AUTOSEL, but if you
explicitly add a "cc: stable@" in the signed-off-by area, we will pick
that up.

> Thanks, and let me know if there's any other workflow things I should
> know about

This is going to cause you more work, but less for us, so thanks!

greg k-h

Re: fs/bcachefs/

[Kent Overstreet]

On Tue, Jan 16, 2024 at 03:13:08PM +0100, Greg KH wrote:
> On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > myself; I'll be doing backports and sending pull requests after stuff
> > has been tested by my CI.
> 
> Now done:
> 	https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?id=9bf1f8f1ca9ae53cf3bc8781e4efdb6ebaee70db
> 
> We will ignore it for any "Fixes:" tags, or AUTOSEL, but if you
> explicitly add a "cc: stable@" in the signed-off-by area, we will pick
> that up.

Would it work for your process to ignore cc: stable@ as well?

I want a tag that I can have tooling grep for later that means "this
patch should be backported later, after seeing sufficient testing", and
cc: stable@ has that meaning in current usage.

Or if you'd like that to be reserved for yourself we could think of a
new one.

> 
> > Thanks, and let me know if there's any other workflow things I should
> > know about
> 
> This is going to cause you more work, but less for us, so thanks!

per our conversation on IRC I'm going to write some simple tooling for
grepping through the git log for patches that may want to be backported
and haven't yet made it to a stable tree.

Also, Sasha, is there any reason your autosel tool couldn't be made
available for others to use that way?

Yes, it's more work for myself, but I have strong opinions that I as the
person who knows the code ought to be picking the patches to backport,
and doing the backports and resolving merge conflicts when necessary;
but collaborating on tooling would be _fantastic_.

Re: fs/bcachefs/

[Greg KH]

On Tue, Jan 16, 2024 at 12:26:38PM -0500, Kent Overstreet wrote:
> On Tue, Jan 16, 2024 at 03:13:08PM +0100, Greg KH wrote:
> > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > myself; I'll be doing backports and sending pull requests after stuff
> > > has been tested by my CI.
> > 
> > Now done:
> > 	https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?id=9bf1f8f1ca9ae53cf3bc8781e4efdb6ebaee70db
> > 
> > We will ignore it for any "Fixes:" tags, or AUTOSEL, but if you
> > explicitly add a "cc: stable@" in the signed-off-by area, we will pick
> > that up.
> 
> Would it work for your process to ignore cc: stable@ as well?

Nope!  That's explicit, you add it for when you want us to pick up a
change when it hits Linus's tree.  If you don't want that to happen,
then don't add it.

> I want a tag that I can have tooling grep for later that means "this
> patch should be backported later, after seeing sufficient testing", and
> cc: stable@ has that meaning in current usage.

Just use a "Fixes:" tag then, that's what networking did for years
before they gave up and now use cc: stable.

thanks,

greg k-h

Re: fs/bcachefs/

[Kent Overstreet]

On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > myself; I'll be doing backports and sending pull requests after stuff
> > has been tested by my CI.
> > 
> > Thanks, and let me know if there's any other workflow things I should
> > know about
> 
> Sure, we can ignore fs/bcachefs/ patches.

I see that you even acked this.

What the fuck?

> 
> Note that the proposed workflow would only work through patches coming
> through your tree, but patches in other subsystems that could affect
> bcachefs might break it in stable trees without being caught by your CI.
> 
> I'd recommend integrating your pre-release tests with something like
> kernelci, which would let us catch bcachefs-affecting issues coming from
> other sources.
> 
> What more, if we do the above, we could in the future avoid special
> casing bcachefs :)
> 
> -- 
> Thanks,
> Sasha

Re: fs/bcachefs/

[Greg KH]

On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > myself; I'll be doing backports and sending pull requests after stuff
> > > has been tested by my CI.
> > > 
> > > Thanks, and let me know if there's any other workflow things I should
> > > know about
> > 
> > Sure, we can ignore fs/bcachefs/ patches.
> 
> I see that you even acked this.
> 
> What the fuck?

Accidents happen, you were copied on those patches.  I'll go drop them
now, not a big deal.

thanks,

greg k-h

Re: fs/bcachefs/

[Greg KH]

On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > has been tested by my CI.
> > > > 
> > > > Thanks, and let me know if there's any other workflow things I should
> > > > know about
> > > 
> > > Sure, we can ignore fs/bcachefs/ patches.
> > 
> > I see that you even acked this.
> > 
> > What the fuck?
> 
> Accidents happen, you were copied on those patches.  I'll go drop them
> now, not a big deal.

Wait, why are you doing "Fixes:" with an empty tag in your commits like
1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?

That's messing with scripts and doesn't make much sense.  Please put a
real git id in there as the documentation suggests to.

thanks,

greg k-h

Re: fs/bcachefs/

[Sasha Levin]

On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
>On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
>> On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
>> > Hi stable team - please don't take patches for fs/bcachefs/ except from
>> > myself; I'll be doing backports and sending pull requests after stuff
>> > has been tested by my CI.
>> >
>> > Thanks, and let me know if there's any other workflow things I should
>> > know about
>>
>> Sure, we can ignore fs/bcachefs/ patches.
>
>I see that you even acked this.
>
>What the fuck?

Is this really how you communicate with other humans?


So what happened here is that my script apparently crashes on empty
"Fixes:" annotations (it can deal with invalid commits, various
combinations of sha1 and description, but apparently not with just
nothing at all).

And so I've fixed my scripts, so hopefully this shouldn't happen again,
but I'd ask for 2 things:

1. Next time you should mail me, could you try and be more civil?

2. Consider fixing your scripts
.https://docs.kernel.org/process/submitting-patches.html#using-reported-by-tested-by-reviewed-by-suggested-by-and-fixes
is pretty explicit as to how "Fixes:" tags are supposed to look like.


-- 
Thanks,
Sasha

Re: fs/bcachefs/

[Kent Overstreet]

On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > has been tested by my CI.
> > > > > 
> > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > know about
> > > > 
> > > > Sure, we can ignore fs/bcachefs/ patches.
> > > 
> > > I see that you even acked this.
> > > 
> > > What the fuck?
> > 
> > Accidents happen, you were copied on those patches.  I'll go drop them
> > now, not a big deal.
> 
> Wait, why are you doing "Fixes:" with an empty tag in your commits like
> 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> 
> That's messing with scripts and doesn't make much sense.  Please put a
> real git id in there as the documentation suggests to.

There isn't always a clear-cut commit when a regression was introduced
(it might not have been a regresison at all). I could dig and make
something up, but that's slowing down your workflow, and I thought I was
going to be handling all the stable backports for fs/bcachefs/, so - ?

Re: fs/bcachefs/

[Greg KH]

On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > has been tested by my CI.
> > > > > > 
> > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > know about
> > > > > 
> > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > 
> > > > I see that you even acked this.
> > > > 
> > > > What the fuck?
> > > 
> > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > now, not a big deal.
> > 
> > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > 
> > That's messing with scripts and doesn't make much sense.  Please put a
> > real git id in there as the documentation suggests to.
> 
> There isn't always a clear-cut commit when a regression was introduced
> (it might not have been a regresison at all). I could dig and make
> something up, but that's slowing down your workflow, and I thought I was
> going to be handling all the stable backports for fs/bcachefs/, so - ?
> 

Doesn't matter, please do not put "fake" tags in commit messages like
this.  It hurts all of the people that parse commit logs.  Just don't
put a fixes tag at all as the documentation states that after "Fixes:" a
commit id belongs.

thanks,

greg k-h

Re: fs/bcachefs/

[Kent Overstreet]

On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
> On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> > On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > > has been tested by my CI.
> > > > > > > 
> > > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > > know about
> > > > > > 
> > > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > > 
> > > > > I see that you even acked this.
> > > > > 
> > > > > What the fuck?
> > > > 
> > > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > > now, not a big deal.
> > > 
> > > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > > 
> > > That's messing with scripts and doesn't make much sense.  Please put a
> > > real git id in there as the documentation suggests to.
> > 
> > There isn't always a clear-cut commit when a regression was introduced
> > (it might not have been a regresison at all). I could dig and make
> > something up, but that's slowing down your workflow, and I thought I was
> > going to be handling all the stable backports for fs/bcachefs/, so - ?
> > 
> 
> Doesn't matter, please do not put "fake" tags in commit messages like
> this.  It hurts all of the people that parse commit logs.  Just don't
> put a fixes tag at all as the documentation states that after "Fixes:" a
> commit id belongs.

Then there's a gap, because I need a tag that I can stick in a commit
message that says "this is a bugfix I need to consider backporting
later", and the way you want the Fixes: tag used doesn't meet my needs.

Re: fs/bcachefs/

[Greg KH]

On Tue, Feb 20, 2024 at 03:22:59PM -0500, Kent Overstreet wrote:
> On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
> > On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> > > On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > > > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > > > has been tested by my CI.
> > > > > > > > 
> > > > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > > > know about
> > > > > > > 
> > > > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > > > 
> > > > > > I see that you even acked this.
> > > > > > 
> > > > > > What the fuck?
> > > > > 
> > > > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > > > now, not a big deal.
> > > > 
> > > > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > > > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > > > 
> > > > That's messing with scripts and doesn't make much sense.  Please put a
> > > > real git id in there as the documentation suggests to.
> > > 
> > > There isn't always a clear-cut commit when a regression was introduced
> > > (it might not have been a regresison at all). I could dig and make
> > > something up, but that's slowing down your workflow, and I thought I was
> > > going to be handling all the stable backports for fs/bcachefs/, so - ?
> > > 
> > 
> > Doesn't matter, please do not put "fake" tags in commit messages like
> > this.  It hurts all of the people that parse commit logs.  Just don't
> > put a fixes tag at all as the documentation states that after "Fixes:" a
> > commit id belongs.
> 
> Then there's a gap, because I need a tag that I can stick in a commit
> message that says "this is a bugfix I need to consider backporting
> later", and the way you want the Fixes: tag used doesn't meet my needs.
> 

Then use patchwork or something else, but please do not override a 15+
year old tag for just one small portion of the kernel.

Or better yet, use the fixes: tag with the commit id you are fixing,
that way all other kernel workflows work properly with this filesystem.
No need to be unique here :)

thanks,

greg k-h

Re: fs/bcachefs/

[Kent Overstreet]

On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
> On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> > On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > > has been tested by my CI.
> > > > > > > 
> > > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > > know about
> > > > > > 
> > > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > > 
> > > > > I see that you even acked this.
> > > > > 
> > > > > What the fuck?
> > > > 
> > > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > > now, not a big deal.
> > > 
> > > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > > 
> > > That's messing with scripts and doesn't make much sense.  Please put a
> > > real git id in there as the documentation suggests to.
> > 
> > There isn't always a clear-cut commit when a regression was introduced
> > (it might not have been a regresison at all). I could dig and make
> > something up, but that's slowing down your workflow, and I thought I was
> > going to be handling all the stable backports for fs/bcachefs/, so - ?
> > 
> 
> Doesn't matter, please do not put "fake" tags in commit messages like
> this.  It hurts all of the people that parse commit logs.  Just don't
> put a fixes tag at all as the documentation states that after "Fixes:" a
> commit id belongs.

So you manually repicked a subset of my pull request, and of the two
patches you silently dropped, one was a security fix - and you _never
communicated_ what you were doing.

Greg, this isn't working. How are we going to fix this?

Re: fs/bcachefs/

[Kent Overstreet]

On Tue, Feb 20, 2024 at 09:39:03PM +0100, Greg KH wrote:
> On Tue, Feb 20, 2024 at 03:22:59PM -0500, Kent Overstreet wrote:
> > On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
> > > On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> > > > On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > > > > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > > > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > > > > has been tested by my CI.
> > > > > > > > > 
> > > > > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > > > > know about
> > > > > > > > 
> > > > > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > > > > 
> > > > > > > I see that you even acked this.
> > > > > > > 
> > > > > > > What the fuck?
> > > > > > 
> > > > > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > > > > now, not a big deal.
> > > > > 
> > > > > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > > > > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > > > > 
> > > > > That's messing with scripts and doesn't make much sense.  Please put a
> > > > > real git id in there as the documentation suggests to.
> > > > 
> > > > There isn't always a clear-cut commit when a regression was introduced
> > > > (it might not have been a regresison at all). I could dig and make
> > > > something up, but that's slowing down your workflow, and I thought I was
> > > > going to be handling all the stable backports for fs/bcachefs/, so - ?
> > > > 
> > > 
> > > Doesn't matter, please do not put "fake" tags in commit messages like
> > > this.  It hurts all of the people that parse commit logs.  Just don't
> > > put a fixes tag at all as the documentation states that after "Fixes:" a
> > > commit id belongs.
> > 
> > Then there's a gap, because I need a tag that I can stick in a commit
> > message that says "this is a bugfix I need to consider backporting
> > later", and the way you want the Fixes: tag used doesn't meet my needs.
> > 
> 
> Then use patchwork or something else, but please do not override a 15+
> year old tag for just one small portion of the kernel.
> 
> Or better yet, use the fixes: tag with the commit id you are fixing,
> that way all other kernel workflows work properly with this filesystem.
> No need to be unique here :)

And stop trying to blame this on your scripts or the fixes tag, there's
a real lack of communication there; you can't say you're going to do one
thing and then do the complete opposite.

Re: fs/bcachefs/

[Greg KH]

On Tue, Feb 20, 2024 at 03:39:16PM -0500, Kent Overstreet wrote:
> On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
> > On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> > > On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > > > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > > > has been tested by my CI.
> > > > > > > > 
> > > > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > > > know about
> > > > > > > 
> > > > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > > > 
> > > > > > I see that you even acked this.
> > > > > > 
> > > > > > What the fuck?
> > > > > 
> > > > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > > > now, not a big deal.
> > > > 
> > > > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > > > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > > > 
> > > > That's messing with scripts and doesn't make much sense.  Please put a
> > > > real git id in there as the documentation suggests to.
> > > 
> > > There isn't always a clear-cut commit when a regression was introduced
> > > (it might not have been a regresison at all). I could dig and make
> > > something up, but that's slowing down your workflow, and I thought I was
> > > going to be handling all the stable backports for fs/bcachefs/, so - ?
> > > 
> > 
> > Doesn't matter, please do not put "fake" tags in commit messages like
> > this.  It hurts all of the people that parse commit logs.  Just don't
> > put a fixes tag at all as the documentation states that after "Fixes:" a
> > commit id belongs.
> 
> So you manually repicked a subset of my pull request, and of the two
> patches you silently dropped, one was a security fix - and you _never
> communicated_ what you were doing.

I explicitly said "Not all of these applied properly, please send me the
remaining ones".  I can go back and get the message-id if you want
reciepts :)

> Greg, this isn't working. How are we going to fix this?

Please send a set of backported commits that you wish to have applied to
the stable trees.  All other subsystems do this fairly easily, it's no
different from sending a patch series out for anything else.

Worst case, I can take a git tree, BUT I will then turn that git tree
into individual commits as that is what we MUST deal with for the stable
trees, we can not work with direct pull requests for obvious reasons of
how the tree needs to be managed (i.e. rebasing all the time would never
work.)

thanks,

greg k-h

Re: fs/bcachefs/

[Kent Overstreet]

On Tue, Feb 20, 2024 at 09:51:20PM +0100, Greg KH wrote:
> On Tue, Feb 20, 2024 at 03:39:16PM -0500, Kent Overstreet wrote:
> > On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
> > > On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> > > > On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > > > > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > > > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > > > > has been tested by my CI.
> > > > > > > > > 
> > > > > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > > > > know about
> > > > > > > > 
> > > > > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > > > > 
> > > > > > > I see that you even acked this.
> > > > > > > 
> > > > > > > What the fuck?
> > > > > > 
> > > > > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > > > > now, not a big deal.
> > > > > 
> > > > > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > > > > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > > > > 
> > > > > That's messing with scripts and doesn't make much sense.  Please put a
> > > > > real git id in there as the documentation suggests to.
> > > > 
> > > > There isn't always a clear-cut commit when a regression was introduced
> > > > (it might not have been a regresison at all). I could dig and make
> > > > something up, but that's slowing down your workflow, and I thought I was
> > > > going to be handling all the stable backports for fs/bcachefs/, so - ?
> > > > 
> > > 
> > > Doesn't matter, please do not put "fake" tags in commit messages like
> > > this.  It hurts all of the people that parse commit logs.  Just don't
> > > put a fixes tag at all as the documentation states that after "Fixes:" a
> > > commit id belongs.
> > 
> > So you manually repicked a subset of my pull request, and of the two
> > patches you silently dropped, one was a security fix - and you _never
> > communicated_ what you were doing.
> 
> I explicitly said "Not all of these applied properly, please send me the
> remaining ones".  I can go back and get the message-id if you want
> reciepts :)

I gave you a _signed pull request_, and there were no merge conflicts.

> > Greg, this isn't working. How are we going to fix this?
> 
> Please send a set of backported commits that you wish to have applied to
> the stable trees.  All other subsystems do this fairly easily, it's no
> different from sending a patch series out for anything else.
> 
> Worst case, I can take a git tree, BUT I will then turn that git tree
> into individual commits as that is what we MUST deal with for the stable
> trees, we can not work with direct pull requests for obvious reasons of
> how the tree needs to be managed (i.e. rebasing all the time would never
> work.)

You rebase these trees? Why? Are they not public?

Look, I need to know that the code I send you is the same as the code
that gets published in stable releases. If you're going to be rebasing
the trees I send you, _all_ the mechanisms we have for doing that
validation break and I'm back to manual verification.

And given that we've got mechanisms for avoiding that - not rebasing so
that we can verify by the sha1, gpg signing - why is stable special
here?

Re: fs/bcachefs/

[Greg KH]

On Tue, Feb 20, 2024 at 04:00:18PM -0500, Kent Overstreet wrote:
> On Tue, Feb 20, 2024 at 09:51:20PM +0100, Greg KH wrote:
> > On Tue, Feb 20, 2024 at 03:39:16PM -0500, Kent Overstreet wrote:
> > > On Tue, Feb 20, 2024 at 09:19:01PM +0100, Greg KH wrote:
> > > > On Tue, Feb 20, 2024 at 03:06:14PM -0500, Kent Overstreet wrote:
> > > > > On Tue, Feb 20, 2024 at 07:53:04PM +0100, Greg KH wrote:
> > > > > > On Tue, Feb 20, 2024 at 07:03:23PM +0100, Greg KH wrote:
> > > > > > > On Tue, Feb 20, 2024 at 12:23:33PM -0500, Kent Overstreet wrote:
> > > > > > > > On Mon, Jan 15, 2024 at 06:03:11PM -0500, Sasha Levin wrote:
> > > > > > > > > On Mon, Jan 15, 2024 at 05:12:17PM -0500, Kent Overstreet wrote:
> > > > > > > > > > Hi stable team - please don't take patches for fs/bcachefs/ except from
> > > > > > > > > > myself; I'll be doing backports and sending pull requests after stuff
> > > > > > > > > > has been tested by my CI.
> > > > > > > > > > 
> > > > > > > > > > Thanks, and let me know if there's any other workflow things I should
> > > > > > > > > > know about
> > > > > > > > > 
> > > > > > > > > Sure, we can ignore fs/bcachefs/ patches.
> > > > > > > > 
> > > > > > > > I see that you even acked this.
> > > > > > > > 
> > > > > > > > What the fuck?
> > > > > > > 
> > > > > > > Accidents happen, you were copied on those patches.  I'll go drop them
> > > > > > > now, not a big deal.
> > > > > > 
> > > > > > Wait, why are you doing "Fixes:" with an empty tag in your commits like
> > > > > > 1a1c93e7f814 ("bcachefs: Fix missing bch2_err_class() calls")?
> > > > > > 
> > > > > > That's messing with scripts and doesn't make much sense.  Please put a
> > > > > > real git id in there as the documentation suggests to.
> > > > > 
> > > > > There isn't always a clear-cut commit when a regression was introduced
> > > > > (it might not have been a regresison at all). I could dig and make
> > > > > something up, but that's slowing down your workflow, and I thought I was
> > > > > going to be handling all the stable backports for fs/bcachefs/, so - ?
> > > > > 
> > > > 
> > > > Doesn't matter, please do not put "fake" tags in commit messages like
> > > > this.  It hurts all of the people that parse commit logs.  Just don't
> > > > put a fixes tag at all as the documentation states that after "Fixes:" a
> > > > commit id belongs.
> > > 
> > > So you manually repicked a subset of my pull request, and of the two
> > > patches you silently dropped, one was a security fix - and you _never
> > > communicated_ what you were doing.
> > 
> > I explicitly said "Not all of these applied properly, please send me the
> > remaining ones".  I can go back and get the message-id if you want
> > reciepts :)
> 
> I gave you a _signed pull request_, and there were no merge conflicts.
> 
> > > Greg, this isn't working. How are we going to fix this?
> > 
> > Please send a set of backported commits that you wish to have applied to
> > the stable trees.  All other subsystems do this fairly easily, it's no
> > different from sending a patch series out for anything else.
> > 
> > Worst case, I can take a git tree, BUT I will then turn that git tree
> > into individual commits as that is what we MUST deal with for the stable
> > trees, we can not work with direct pull requests for obvious reasons of
> > how the tree needs to be managed (i.e. rebasing all the time would never
> > work.)
> 
> You rebase these trees? Why? Are they not public?
> 
> Look, I need to know that the code I send you is the same as the code
> that gets published in stable releases. If you're going to be rebasing
> the trees I send you, _all_ the mechanisms we have for doing that
> validation break and I'm back to manual verification.
> 
> And given that we've got mechanisms for avoiding that - not rebasing so
> that we can verify by the sha1, gpg signing - why is stable special
> here?



Hi,

This is the friendly email-bot of the stable kernel team.  I've detected that
you are asking a question that has been discussed many times in the past.  If
you wish to contact the stable developers directly, please use their phone
hotline, 1-800-382-5633.

thanks,

stable email bot

----------------

*beep  beep beep beep  beep beep beep  beep beep beep beep*

*ring*  *ring*

	This is the Linux stable kernel team's phone line.

	Please listen to the following menu fully as the items have changed
	recently.

	To contact the Linux stable developer's directly, please press 1.

	To contact the Linux stable develop—
*1*

	All of the current stable developers are busy handling other kernel
	developer issues or reviewing properly tagged kernel patches.  Please
	wait for the next available developer, or press 0 to return to the main
	menu.

	You are caller number SEVENTY FIVE in the queue with an expected wait
	time of ELEVEN HUNDRED AND TWENTY FIVE MINUTES.

	Please enjoy the smooth sounds of Enya while you wait.

		"Who can say where the road goes?
		 Where the day flows?
		 Only time.
		 And who can say—
*0*

	This is the Linux stable kernel team's phone line.

	Please listen to the following menu fully as the items have changed
	recently.

	To contact the Linux stable developer's directly, please press 1.

	To contact the Linux stable developer's legal department to lodge a
	complaint, please press 2.

	To listen to the—
*2*

	You are now being transferred to the legal department, to return to the
	main menu, please press 0

	....

	Welcome to the law firm of Dewey Cheatem & Howe.  Unfortunately, due to
	a large outstanding bill, we no longer represent the Linux stable
	kernel team as it turned out they were not being paid for their
	services, and so didn't feel like paying for ours either.  If you wish
	to hire us to represent you in an accident claim, stay on the line and
	someone will be with you—
*0*

	This is the Linux stable kernel team's phone line.

	Please listen to the following menu fully as the items have changed
	recently.

	To contact the Linux stable developer's directly, please press 1.

	To contact the Linux stable developer's legal department to lodge a
	complaint, please press 2.

	To listen to the often asked question of "why don't you accept pull
	requests directly", please press 3.

	For all other questions, please write an email and it will be handled
	in the order in which it is received.

	This menu will now repeat for your enjoyment.

	This is the Linux stable—

*3*

	The reason the stable kernel team can not accept pull requests directly
	is that they accumulate many patches from many different developers and
	trees and combine them all into a set of patches to be applied on top
	of the last release.  If a git tree were used, the ordering,
	reshuffling, removing, and adding of patches in the list would not
	work properly at all as it would require constant rebasing.  This can
	be seen directly by following the often-rebased linux-stable-rc git
	tree, which is only present for CI systems to consume if they can not
	handle a set of quilt patches directly.

	The stable developer workflow has been working well for over 15 years
	as a patch queue, no convincing of "but you must accept my pull
	request" is going to work well, given the current requirements of how
	the stable trees are generated.

	A pull request can be handled, by the stable developer taking it,
	exporting the changes as individual patches (using 'git format-patch')
	and then importing them into the stable patch queue directly.  If a
	subsystem maintainer feels that this is the best way to coordinate with
	the stable team, that is fine, but note that individual patches will be
	the end result, so perhaps just using 'git send-email' in the first
	place would be simpler for everyone involved?

	Having to process a git pull takes additional time and energy and slows
	down the patch acceptance rate.  It is generally postponed as there are
	lots of properly tagged commits that are easier and simpler for them to
	handle.

	Given the huge patch volume that the stable tree manages (30-40 changes
	accepted a day, 7 days a week), any one kernel subsystem that wishes to
	do something different only slows down everyone else.  Yes, your
	subsystem is special and unique, just like everyone else's, we know,
	remember, we maintain kernel subsystems too.  Just use patches and
	email, it's the lowest common denominator here that works well given
	the often-disconnected nature of the stable developers as they travel
	the globe racking up frequent flyer points and talking in generic
	conference center meeting rooms about hardware roadmaps and their
	applicability to the kernel.

	Thank you for listening to the reasoning here, and as we generally
	assume this will not have calmed you down much at all, please feel free
	to stay on the line in the phone queue which currently has EIGHTY FIVE
	callers ahead of you, with a expected wait time of TWELVE HUNDRED AND
	SEVENTY MINUTES

	Please continue to enjoy the smooth sounds of Enya while you wait.

		"Let me sail, let me sail
		 Let the orinoco flow"
		 Let me reach, let me beach—

*CLICK*

Re: fs/bcachefs/

[Oleksandr Natalenko]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

On středa 21. února 2024 15:53:11 CET Greg KH wrote:
> 	Given the huge patch volume that the stable tree manages (30-40 changes
> 	accepted a day, 7 days a week), any one kernel subsystem that wishes to
> 	do something different only slows down everyone else.

Lower down the volume then? Raise the bar for what gets backported? Stable kernel releases got unnecessarily big [1] (Jiří is in Cc). Those 40 changes a day cannot get a proper review. Each stable release tries to mimic -rc except -rc is in consistent state while "stable" is just a bunch of changes picked here and there.

[1] https://lwn.net/Articles/962131/

-- 
Oleksandr Natalenko (post-factum)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: fs/bcachefs/

[Greg KH]

On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
> On středa 21. února 2024 15:53:11 CET Greg KH wrote:
> > 	Given the huge patch volume that the stable tree manages (30-40 changes
> > 	accepted a day, 7 days a week), any one kernel subsystem that wishes to
> > 	do something different only slows down everyone else.
> 
> Lower down the volume then? Raise the bar for what gets backported?
> Stable kernel releases got unnecessarily big [1] (Jiří is in Cc).
> Those 40 changes a day cannot get a proper review. Each stable release
> tries to mimic -rc except -rc is in consistent state while "stable" is
> just a bunch of changes picked here and there.

If you can point out any specific commits that we should not be taking,
please let us know.

Personally I think we are not taking enough, and are still missing real
fixes.  Overall, this is only a very small % of what goes into Linus's
tree every day, so by that measure alone, we know we are missing things.

thanks,

greg k-h

Re: fs/bcachefs/

[Vlastimil Babka]

On 2/21/24 18:57, Greg KH wrote:
> On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
>> On středa 21. února 2024 15:53:11 CET Greg KH wrote:
>> > 	Given the huge patch volume that the stable tree manages (30-40 changes
>> > 	accepted a day, 7 days a week), any one kernel subsystem that wishes to
>> > 	do something different only slows down everyone else.
>> 
>> Lower down the volume then? Raise the bar for what gets backported?
>> Stable kernel releases got unnecessarily big [1] (Jiří is in Cc).
>> Those 40 changes a day cannot get a proper review. Each stable release
>> tries to mimic -rc except -rc is in consistent state while "stable" is
>> just a bunch of changes picked here and there.
> 
> If you can point out any specific commits that we should not be taking,
> please let us know.
> 
> Personally I think we are not taking enough, and are still missing real
> fixes.  Overall, this is only a very small % of what goes into Linus's
> tree every day, so by that measure alone, we know we are missing things.

What % of what goes into Linus's tree do you think fits within the rules
stated in Documentation/process/stable-kernel-rules.rst ? I don't know but
"very small" would be my guess, so we should be fine as it is?

Or are the rules actually still being observed? I doubt e.g. many of the
AUTOSEL backports fit them? Should we rename the file to
stable-rules-nonsense.rst?

> thanks,
> 
> greg k-h

Re: fs/bcachefs/

[Kent Overstreet]

On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
> On 2/21/24 18:57, Greg KH wrote:
> > On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
> >> On středa 21. února 2024 15:53:11 CET Greg KH wrote:
> >> > 	Given the huge patch volume that the stable tree manages (30-40 changes
> >> > 	accepted a day, 7 days a week), any one kernel subsystem that wishes to
> >> > 	do something different only slows down everyone else.
> >> 
> >> Lower down the volume then? Raise the bar for what gets backported?
> >> Stable kernel releases got unnecessarily big [1] (Jiří is in Cc).
> >> Those 40 changes a day cannot get a proper review. Each stable release
> >> tries to mimic -rc except -rc is in consistent state while "stable" is
> >> just a bunch of changes picked here and there.
> > 
> > If you can point out any specific commits that we should not be taking,
> > please let us know.
> > 
> > Personally I think we are not taking enough, and are still missing real
> > fixes.  Overall, this is only a very small % of what goes into Linus's
> > tree every day, so by that measure alone, we know we are missing things.
> 
> What % of what goes into Linus's tree do you think fits within the rules
> stated in Documentation/process/stable-kernel-rules.rst ? I don't know but
> "very small" would be my guess, so we should be fine as it is?
> 
> Or are the rules actually still being observed? I doubt e.g. many of the
> AUTOSEL backports fit them? Should we rename the file to
> stable-rules-nonsense.rst?

Yeah, I'd say around half of the backports I see being done really had
no justification at all - i.e. were for fixes for bugs that weren't
present on the kernel being backported to, including most of the autosel
patches.

There's clearly a balance to be struck with what we backport - but it
doesn't seem like Greg and Sasha are trying to find that balance, it
seems to be all pedal-to-the-metal backport-everything.

And the "process" seems to be whatever makes things most convenient for
Greg and Sasha, and they _really_ want to be doing everything
themselves. That form letter response quite illustrates that.

This doesn't scale.

And Greg not taking signed pull requests is a _real_ what the fuck. If
we care about supply chain attacks at all, surely we care about them for
stable, because those are the kernels most people actually run!

Greg, you're doing an end run around a lot of the process the
_community_ has built up.

Re: fs/bcachefs/

[Sasha Levin]

On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
>On 2/21/24 18:57, Greg KH wrote:
>> On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
>>> On středa 21. února 2024 15:53:11 CET Greg KH wrote:
>>> > 	Given the huge patch volume that the stable tree manages (30-40 changes
>>> > 	accepted a day, 7 days a week), any one kernel subsystem that wishes to
>>> > 	do something different only slows down everyone else.
>>>
>>> Lower down the volume then? Raise the bar for what gets backported?
>>> Stable kernel releases got unnecessarily big [1] (Jiří is in Cc).
>>> Those 40 changes a day cannot get a proper review. Each stable release
>>> tries to mimic -rc except -rc is in consistent state while "stable" is
>>> just a bunch of changes picked here and there.
>>
>> If you can point out any specific commits that we should not be taking,
>> please let us know.
>>
>> Personally I think we are not taking enough, and are still missing real
>> fixes.  Overall, this is only a very small % of what goes into Linus's
>> tree every day, so by that measure alone, we know we are missing things.
>
>What % of what goes into Linus's tree do you think fits within the rules
>stated in Documentation/process/stable-kernel-rules.rst ? I don't know but
>"very small" would be my guess, so we should be fine as it is?
>
>Or are the rules actually still being observed? I doubt e.g. many of the
>AUTOSEL backports fit them? Should we rename the file to
>stable-rules-nonsense.rst?

Hey, I have an exercise for you which came up last week during the whole
CVE thing!

Take a look at a random LTS kernel (I picked 5.10), in particular at the
CVEs assigned to the kernel (in my case I relied on
https://github.com/nluedtke/linux_kernel_cves/blob/master/data/5.10/5.10_security.txt).

See how many of those actually have a stable@ tag to let us know that we
need to pull that commit. (spoiler alert: in the 5.10 case it was ~33%)

Do you have a better way for us to fish for the remaining 67%?

Yeah, some have a Fixes tag, (it's not in stable-kernel-rules.rst!), and
in the 5.10 case it would have helped with about half of the commits,
but even then - what do we do with the remaining half?

The argument you're making is in favor of just ignoring it until they
get a CVE assigned (and even then, would we take them if it goes against
stable-kernel-rules.rst?), but then we end up leaving users exposed for *years*
as evidenced by some CVEs.

So if we go with the current workflow, folks complain that we take too
many patches. If we were to lean strictly to what
stable-kernel-rules.rst says, we'd apparently miss most of the
(security) issues affecting users.

-- 
Thanks,
Sasha

Re: fs/bcachefs/

[Kent Overstreet]

On Wed, Feb 21, 2024 at 05:58:30PM -0500, Sasha Levin wrote:
> On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
> > On 2/21/24 18:57, Greg KH wrote:
> > > On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
> > > > On středa 21. února 2024 15:53:11 CET Greg KH wrote:
> > > > > 	Given the huge patch volume that the stable tree manages (30-40 changes
> > > > > 	accepted a day, 7 days a week), any one kernel subsystem that wishes to
> > > > > 	do something different only slows down everyone else.
> > > > 
> > > > Lower down the volume then? Raise the bar for what gets backported?
> > > > Stable kernel releases got unnecessarily big [1] (Jiří is in Cc).
> > > > Those 40 changes a day cannot get a proper review. Each stable release
> > > > tries to mimic -rc except -rc is in consistent state while "stable" is
> > > > just a bunch of changes picked here and there.
> > > 
> > > If you can point out any specific commits that we should not be taking,
> > > please let us know.
> > > 
> > > Personally I think we are not taking enough, and are still missing real
> > > fixes.  Overall, this is only a very small % of what goes into Linus's
> > > tree every day, so by that measure alone, we know we are missing things.
> > 
> > What % of what goes into Linus's tree do you think fits within the rules
> > stated in Documentation/process/stable-kernel-rules.rst ? I don't know but
> > "very small" would be my guess, so we should be fine as it is?
> > 
> > Or are the rules actually still being observed? I doubt e.g. many of the
> > AUTOSEL backports fit them? Should we rename the file to
> > stable-rules-nonsense.rst?
> 
> Hey, I have an exercise for you which came up last week during the whole
> CVE thing!
> 
> Take a look at a random LTS kernel (I picked 5.10), in particular at the
> CVEs assigned to the kernel (in my case I relied on
> https://github.com/nluedtke/linux_kernel_cves/blob/master/data/5.10/5.10_security.txt).
> 
> See how many of those actually have a stable@ tag to let us know that we
> need to pull that commit. (spoiler alert: in the 5.10 case it was ~33%)
> 
> Do you have a better way for us to fish for the remaining 67%?
> 
> Yeah, some have a Fixes tag, (it's not in stable-kernel-rules.rst!), and
> in the 5.10 case it would have helped with about half of the commits,
> but even then - what do we do with the remaining half?
> 
> The argument you're making is in favor of just ignoring it until they
> get a CVE assigned (and even then, would we take them if it goes against
> stable-kernel-rules.rst?), but then we end up leaving users exposed for *years*
> as evidenced by some CVEs.

No, I'm not in favor of ignoring things either. What I want is better
collaboration between subsystem maintainers and you guys. Awhile back I
was proposing some sort of shared dashboard where we could all track and
annotate which patches should be considered for backporting when,
because the current email based workflow sucks.

We need to be working together better, which is why I'm so _damn
annoyed_ over the crap with the Fixes: tag, and Greg not taking my
signed pull request as an atomic unit.

You guys shouldn't be doing all this work yourselves; the subsystem
maintainers, the people who know the code best, should be involved. But
your attitude 100% pushes people away.

I need a workflow that works for me, if I'm going to do the backporting
for bcachefs patches, as I intend to: fuckups in filesystem land can
have _far_ too high a cost for me to leave this work to anyone else.

What I need is:

 - A way to unambigiously tag a patch as something that I need to look
   at later when I'm doing backports; and I do _not_ want to have to go
   git blame spelunking when I'm doing that, distracting me from
   whatever else I'm working on; if you insist on the Fixes: tag
   conforming to your tools then this won't get done and bugfixes will
   get lost.

 - Signed pull requests to not get tweaked and rebased.

   Tweaking and hand editing patches is one thing when it's being done
   by the subsystem maintainer, the person who at least nominally knows
   the code best and is best qualified to review those changes.

   You guys should not be doing that, because then I have to go back and
   check your work (and even if you swear you aren't changing the code;
   how do I know unless I look at the diffs? Remember all the talk about
   supply chain attacks?) and I'm just not going to do that. That adds
   too much to my workload, and there's no reason for it.

And please, you guys, make a bit more of an effort to work with people,
and listen to and address their concerns. I hear _way_ too much grousing
in private about -stable bullshit, and despite that I went in to this
optimistic that we'd be able to cut past the bullshit and establish a
good working relationship. Let's see if we still can...

Re: fs/bcachefs/

[Oleksandr Natalenko]

[-- Attachment #1: Type: text/plain, Size: 3407 bytes --]

On středa 21. února 2024 23:58:30 CET Sasha Levin wrote:
> On Wed, Feb 21, 2024 at 07:10:02PM +0100, Vlastimil Babka wrote:
> >On 2/21/24 18:57, Greg KH wrote:
> >> On Wed, Feb 21, 2024 at 05:00:05PM +0100, Oleksandr Natalenko wrote:
> >>> On středa 21. února 2024 15:53:11 CET Greg KH wrote:
> >>> > 	Given the huge patch volume that the stable tree manages (30-40 changes
> >>> > 	accepted a day, 7 days a week), any one kernel subsystem that wishes to
> >>> > 	do something different only slows down everyone else.
> >>>
> >>> Lower down the volume then? Raise the bar for what gets backported?
> >>> Stable kernel releases got unnecessarily big [1] (Jiří is in Cc).
> >>> Those 40 changes a day cannot get a proper review. Each stable release
> >>> tries to mimic -rc except -rc is in consistent state while "stable" is
> >>> just a bunch of changes picked here and there.
> >>
> >> If you can point out any specific commits that we should not be taking,
> >> please let us know.
> >>
> >> Personally I think we are not taking enough, and are still missing real
> >> fixes.  Overall, this is only a very small % of what goes into Linus's
> >> tree every day, so by that measure alone, we know we are missing things.
> >
> >What % of what goes into Linus's tree do you think fits within the rules
> >stated in Documentation/process/stable-kernel-rules.rst ? I don't know but
> >"very small" would be my guess, so we should be fine as it is?
> >
> >Or are the rules actually still being observed? I doubt e.g. many of the
> >AUTOSEL backports fit them? Should we rename the file to
> >stable-rules-nonsense.rst?
> 
> Hey, I have an exercise for you which came up last week during the whole
> CVE thing!
> 
> Take a look at a random LTS kernel (I picked 5.10), in particular at the
> CVEs assigned to the kernel (in my case I relied on
> https://github.com/nluedtke/linux_kernel_cves/blob/master/data/5.10/5.10_security.txt).
> 
> See how many of those actually have a stable@ tag to let us know that we
> need to pull that commit. (spoiler alert: in the 5.10 case it was ~33%)
> 
> Do you have a better way for us to fish for the remaining 67%?

With all due respect, once a measure becomes a target, it ceases to be a good measure.

> Yeah, some have a Fixes tag, (it's not in stable-kernel-rules.rst!), and
> in the 5.10 case it would have helped with about half of the commits,
> but even then - what do we do with the remaining half?
> 
> The argument you're making is in favor of just ignoring it until they
> get a CVE assigned (and even then, would we take them if it goes against
> stable-kernel-rules.rst?), but then we end up leaving users exposed for *years*
> as evidenced by some CVEs.
> 
> So if we go with the current workflow, folks complain that we take too
> many patches. If we were to lean strictly to what
> stable-kernel-rules.rst says, we'd apparently miss most of the
> (security) issues affecting users.

It's not a catastrophic problem to miss fixes, you will never be able to reach 100% anyway, guaranteed. Introducing a new, untested and not reviewed code on scale is a bigger problem. Yes, backports should be considered as a new code that needs appropriate review. Regardless of how skilled you two are, it's not a task for you. There must be another way of doing this.

-- 
Oleksandr Natalenko (post-factum)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: fs/bcachefs/

[Greg KH]

On Wed, Feb 21, 2024 at 06:12:42PM -0500, Kent Overstreet wrote:
> > The argument you're making is in favor of just ignoring it until they
> > get a CVE assigned (and even then, would we take them if it goes against
> > stable-kernel-rules.rst?), but then we end up leaving users exposed for *years*
> > as evidenced by some CVEs.
> 
> No, I'm not in favor of ignoring things either. What I want is better
> collaboration between subsystem maintainers and you guys. Awhile back I
> was proposing some sort of shared dashboard where we could all track and
> annotate which patches should be considered for backporting when,
> because the current email based workflow sucks.

I too would like a pony, but for now, we work with what we have, right?

> We need to be working together better, which is why I'm so _damn
> annoyed_ over the crap with the Fixes: tag, and Greg not taking my
> signed pull request as an atomic unit.

Fixes: has a long long long history and to think that your subsystem can
ignore it and do something else with it feels very odd to me.  Your
subsystem lives in the same ecosystem as all other 4000+ developers.
There's a reason we have standards/rules, you can't just ignore them.

> You guys shouldn't be doing all this work yourselves; the subsystem
> maintainers, the people who know the code best, should be involved. But
> your attitude 100% pushes people away.

The goal of the stable/lts kernels is that no developer or maintainer
that does NOT want to help out, has to do anything more than just a
simple "cc: stable@vger.kernel.org" addition to a commit when they apply
it to their tree.

Anything above that is wonderful, but again, not something I can ever
tell someone to do.

> I need a workflow that works for me, if I'm going to do the backporting
> for bcachefs patches, as I intend to: fuckups in filesystem land can
> have _far_ too high a cost for me to leave this work to anyone else.
> 
> What I need is:
> 
>  - A way to unambigiously tag a patch as something that I need to look
>    at later when I'm doing backports; and I do _not_ want to have to go
>    git blame spelunking when I'm doing that, distracting me from
>    whatever else I'm working on; if you insist on the Fixes: tag
>    conforming to your tools then this won't get done and bugfixes will
>    get lost.

Why can't you use Fixes like all of the 10's of thousands of other
developers have over the past 15+ years?

And if you don't like it, that's fine, but again, you can not redefine
it to mean something else for your tiny subsystem, sorry, that's not how
projects of any size work well and survive.

>  - Signed pull requests to not get tweaked and rebased.

As-is, I can NOT take your signed pull request as it did not include the
needed information in it, as I said at the time (i.e. no reference to
the commits that you were backporting.)

What I DID do is dig through your pull request and take the individual
commits that DID apply after looking up, by hand, the proper upstream
git commit id.  I then did NOT take the 2 commits that you had modified
from their upstream version, as there was no indication as to why the
changes were made, or even that any change was made at all, from what is
in Linus's tree.  And at the time I told you all of this, so there was
no question of what happened, and what was expected.

And remember, Linus's tree is the "canonical" version here.  We rely on
signed tags for that, and then we can safely cherry-pick and compare to
those commits to verify that "yes, this is the exact same change that we
already approved and accepted".

>    Tweaking and hand editing patches is one thing when it's being done
>    by the subsystem maintainer, the person who at least nominally knows
>    the code best and is best qualified to review those changes.

Wonderful, then say you tweaked and hand-edited the patches when you
submit them to us and we are fine with it.  But you didn't do that, so
in the interest of saftey and stability, I did NOT take the changes that
were modified.  And here I would think you would be happy we do
validation and verification...

>    You guys should not be doing that, because then I have to go back and
>    check your work (and even if you swear you aren't changing the code;
>    how do I know unless I look at the diffs? Remember all the talk about
>    supply chain attacks?) and I'm just not going to do that. That adds
>    too much to my workload, and there's no reason for it.

If I was a more paranoid person, I would have thought that the modified
changes you sent us with no indication that the changes were modified,
was a "supply chain attack" that you were attempting to do on us.

> And please, you guys, make a bit more of an effort to work with people,
> and listen to and address their concerns. I hear _way_ too much grousing
> in private about -stable bullshit, and despite that I went in to this
> optimistic that we'd be able to cut past the bullshit and establish a
> good working relationship. Let's see if we still can...

This goes both ways please.

Again, I can not take pull requests, it does not work at all with our
workflow as we NEED and REQUIRE actual individual commits, both for
verification and validation, as well as to actually be able to apply to
our trees.

We NEED and REQUIRE the git commit ids of the changes you are asking for
including to be in the commit message itself (or somewhere that I can
then add it), as that is how we, and everyone else, tracks what gets
applied to where, and to be able to validate and ensure that the commit
really is what you say it is.

We NEED and REQUIRE you, if you do modify a commit from what is in
Linus's tree, to say "hey, this is modified because of X and Y", not to
just not say anything and assume that we should blindly take a modified
change.  You don't want us to do that, right?

So, to work with the stable trees, you have a number of simple options,
here they are going from least-amount-of-work to most-amount-of-work:
	- ignore the stable trees and insist that everyone only use
	  Linus's releases for your subsystem (some subsystems do want
	  this, for whatever reason, that's on them.)
	- add only "Fixes:" tags to commits, with the correct git id.
	  These will eventually get picked up by a stable developer and
	  a semi-best effort done to apply them were relevant, but we
	  can not guarantee it.
	- add a "Cc: stable@vger.kernel.org" tag to a commit when you
	  apply it to your tree.  This is the simplest, and recommended
	  way.  Bonus if you also include a "Fixes:" tag so we know how
	  far back to apply it, AND you will get an automated message
	  for when we can NOT apply it that far back successfully.
	- Send us a list of git ids you want to have us backport.
	- Send us a patch series that you want to have applied, with the
	  git id of the original commit in the commit body, and if the
	  change has been modified from the original, a small note
	  somewhere (usually in the signed-off-by area) saying it has
	  been modified and why.
	- Send us a pull request of patches built up the same way as the
	  above option.  This puts more work on us as we then need to
	  turn the pull request into a set of individual patches and
	  review them "by hand" in a tool other than our mail readers,
	  but eventually the patches will get applied.

Note that no where is a "send a pull request that consists of patches
with no references to upstream git ids or notification that anything has
changed" option.  That just will not work for all of the reasons
speficied above, sorry.

Hope this helps explain all of this better.

greg k-h

Re: fs/bcachefs/

[Kent Overstreet]

On Thu, Feb 22, 2024 at 06:48:58AM +0100, Greg KH wrote:
> >  - A way to unambigiously tag a patch as something that I need to look
> >    at later when I'm doing backports; and I do _not_ want to have to go
> >    git blame spelunking when I'm doing that, distracting me from
> >    whatever else I'm working on; if you insist on the Fixes: tag
> >    conforming to your tools then this won't get done and bugfixes will
> >    get lost.
> 
> Why can't you use Fixes like all of the 10's of thousands of other
> developers have over the past 15+ years?

I've explained that multiple times and you're not listening.

That's not how you work with people, Greg; cut the form letter response
bullshit and try and have a real conversation.

You've been repeatedly stonewalling and shutting down my every attempt
to find a solution here.

But honestly, this isn't even the main issue.

> And if you don't like it, that's fine, but again, you can not redefine
> it to mean something else for your tiny subsystem, sorry, that's not how
> projects of any size work well and survive.
> 
> >  - Signed pull requests to not get tweaked and rebased.
> 
> As-is, I can NOT take your signed pull request as it did not include the
> needed information in it, as I said at the time (i.e. no reference to
> the commits that you were backporting.)

You communicated that one, and I redid the cherry picks with -x...

...And they still showed up in your tree as completely different
commits.

> What I DID do is dig through your pull request and take the individual
> commits that DID apply after looking up, by hand, the proper upstream
> git commit id.  I then did NOT take the 2 commits that you had modified
> from their upstream version, as there was no indication as to why the
> changes were made, or even that any change was made at all, from what is
> in Linus's tree.  And at the time I told you all of this, so there was
> no question of what happened, and what was expected.

Why would you silently redo work that I sent you?

All you're doing is making more work for yourself. If I send you a pull
request that you can't use, kick it back to me and tell me why!

But silently redo it and drop a security fix? Can we please not?

> If I was a more paranoid person, I would have thought that the modified
> changes you sent us with no indication that the changes were modified,
> was a "supply chain attack" that you were attempting to do on us.

Of my own code?

Look, this isn't about not trusting _you_, it's that the further up the
food chain commits go the less it is possible and reasonable to expect
anything to be caught by review.

> > And please, you guys, make a bit more of an effort to work with people,
> > and listen to and address their concerns. I hear _way_ too much grousing
> > in private about -stable bullshit, and despite that I went in to this
> > optimistic that we'd be able to cut past the bullshit and establish a
> > good working relationship. Let's see if we still can...
> 
> This goes both ways please.
> 
> Again, I can not take pull requests, it does not work at all with our
> workflow as we NEED and REQUIRE actual individual commits, both for
> verification and validation, as well as to actually be able to apply to
> our trees.
> 
> We NEED and REQUIRE the git commit ids of the changes you are asking for
> including to be in the commit message itself (or somewhere that I can
> then add it), as that is how we, and everyone else, tracks what gets
> applied to where, and to be able to validate and ensure that the commit
> really is what you say it is.
> 
> We NEED and REQUIRE you, if you do modify a commit from what is in
> Linus's tree, to say "hey, this is modified because of X and Y", not to
> just not say anything and assume that we should blindly take a modified
> change.  You don't want us to do that, right?

I can provide commit messages in the format you need - but also: _none_
of this is documented in stable-kernel-rules.rst, so I'm going to want
something clear and specific I can go off of.

(And why are you not specifying the original sha1 in the format
cherry-pick -x produces? Why is that not documented?)

But not taking signed pull requests is going to be a sticking point, as
well as the fact that you only took part of the pull request.

We've had multiple bugs lately stemming from related patches not being
carried together - _nasty_ bugs, as in "I can't access my filesystem" or
"My data is being corrupted".

That was the case in this pull request too; one of the patches you
dropped, IIRC, was a locking fix for an earlier fix by Al; those patches
should have gone together.

This sort of thing is a good part of why I'm insisting on doing things
myself.

So: in the interest of avoiding issues like this, can I at least ask you
to start taking signed pull requests if the patch metadata is all in the
format you need?

Re: fs/bcachefs/

[Sasha Levin]

On Thu, Feb 22, 2024 at 01:30:01AM -0500, Kent Overstreet wrote:
>On Thu, Feb 22, 2024 at 06:48:58AM +0100, Greg KH wrote:
>> We NEED and REQUIRE you, if you do modify a commit from what is in
>> Linus's tree, to say "hey, this is modified because of X and Y", not to
>> just not say anything and assume that we should blindly take a modified
>> change.  You don't want us to do that, right?
>
>I can provide commit messages in the format you need - but also: _none_
>of this is documented in stable-kernel-rules.rst, so I'm going to want
>something clear and specific I can go off of.

This part is explicitly documented in stable-kernel-rules.rst. In
particular, it lists the 3 ways one could send us commits for inclusion
(https://docs.kernel.org/process/stable-kernel-rules.html#procedure-for-submitting-patches-to-the-stable-tree).
To summarize, it's your choice of:

1. Tag it with a stable@ tag. OR:
2. Send us a mail to the stable mailing list with commit IDs for us to
cherry pick. OR:
3. Send the patches themselves (which is the option you've picked), and
then the doc proceeds to call out how to include the upstream commit ID
in the patch
(https://docs.kernel.org/process/stable-kernel-rules.html#option-3).

-- 
Thanks,
Sasha

stable-kernel-rules was Re: fs/bcachefs/

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1666 bytes --]

Hi!

> > Personally I think we are not taking enough, and are still missing real
> > fixes.  Overall, this is only a very small % of what goes into Linus's
> > tree every day, so by that measure alone, we know we are missing things.
> 
> What % of what goes into Linus's tree do you think fits within the rules
> stated in Documentation/process/stable-kernel-rules.rst ? I don't know but
> "very small" would be my guess, so we should be fine as it is?
> 
> Or are the rules actually still being observed? I doubt e.g. many of the
> AUTOSEL backports fit them? Should we rename the file to
> stable-rules-nonsense.rst?

There seems to be just one rule being observed: "It or an equivalent
fix must already exist in Linus' tree (upstream).". Every other rule is
broken pretty much all the time.

AUTOSEL is a problem.

Plus there's problem with dependencies -- if a patch A is need for fix
B, the rules pretty much go out of the window, huge patches are
applied, whitespace fixes are applied, etc.

There are even known-bad patches being applied, and then
reverted. Greg explained that it heps his process somehow.

For example in 6.1.53 review, my notes say 30% of the patches did not
match the documented rules. 42% for v6.1.76.

OTOH ammount of patches that cause "real" problems is not that great,
and we seem to have enough testing. Still, updating the documentation
to match the reality would be good (perhaps explaining that stable
does not have manpower to re-do the dependencies, and how "apply bad
and revert" works).

Best regards,
								Pavel
-- 
People of Russia, stop Putin before his war on Ukraine escalates.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: stable-kernel-rules was Re: fs/bcachefs/

[Kent Overstreet]

On Thu, Feb 22, 2024 at 08:19:06PM +0100, Pavel Machek wrote:
> Hi!
> 
> > > Personally I think we are not taking enough, and are still missing real
> > > fixes.  Overall, this is only a very small % of what goes into Linus's
> > > tree every day, so by that measure alone, we know we are missing things.
> > 
> > What % of what goes into Linus's tree do you think fits within the rules
> > stated in Documentation/process/stable-kernel-rules.rst ? I don't know but
> > "very small" would be my guess, so we should be fine as it is?
> > 
> > Or are the rules actually still being observed? I doubt e.g. many of the
> > AUTOSEL backports fit them? Should we rename the file to
> > stable-rules-nonsense.rst?
> 
> There seems to be just one rule being observed: "It or an equivalent
> fix must already exist in Linus' tree (upstream).". Every other rule is
> broken pretty much all the time.
> 
> AUTOSEL is a problem.
> 
> Plus there's problem with dependencies -- if a patch A is need for fix
> B, the rules pretty much go out of the window, huge patches are
> applied, whitespace fixes are applied, etc.
> 
> There are even known-bad patches being applied, and then
> reverted. Greg explained that it heps his process somehow.

This seems to be a pretty consistent theme theme - thins are done baesd
on whatever makes Greg's process easier, not input from the people
stable ought to be working with. Pretty questionable set of priorities
if you ask me.

Re: stable-kernel-rules was Re: fs/bcachefs/

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

Hi!

> > There seems to be just one rule being observed: "It or an equivalent
> > fix must already exist in Linus' tree (upstream).". Every other rule is
> > broken pretty much all the time.
> > 
> > AUTOSEL is a problem.
> > 
> > Plus there's problem with dependencies -- if a patch A is need for fix
> > B, the rules pretty much go out of the window, huge patches are
> > applied, whitespace fixes are applied, etc.
> > 
> > There are even known-bad patches being applied, and then
> > reverted. Greg explained that it heps his process somehow.
> 
> This seems to be a pretty consistent theme theme - thins are done baesd
> on whatever makes Greg's process easier, not input from the people
> stable ought to be working with. Pretty questionable set of priorities
> if you ask me.

Well, I'd not mind stable process following the documented rules.

But fixing the documentation to match the reality would also be an
improvement, because some people actually read it and expect it to be
followed.

Best regards,
								Pavel
-- 
People of Russia, stop Putin before his war on Ukraine escalates.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]