* [PATCH AUTOSEL 6.1 07/21] fs: Use CHECK_DATA_CORRUPTION() when kernel bugs are detected [not found] <20230226034256.771769-1-sashal@kernel.org> @ 2023-02-26 3:42 ` Sasha Levin 2023-02-26 3:42 ` [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() Sasha Levin 1 sibling, 0 replies; 83+ messages in thread From: Sasha Levin @ 2023-02-26 3:42 UTC (permalink / raw) To: linux-kernel, stable Cc: Jann Horn, Christian Brauner, Kees Cook, Sasha Levin, viro, memxor, ast, brouer, davemarchevsky, linux-fsdevel From: Jann Horn <jannh@google.com> [ Upstream commit 47d586913f2abec4d240bae33417f537fda987ec ] Currently, filp_close() and generic_shutdown_super() use printk() to log messages when bugs are detected. This is problematic because infrastructure like syzkaller has no idea that this message indicates a bug. In addition, some people explicitly want their kernels to BUG() when kernel data corruption has been detected (CONFIG_BUG_ON_DATA_CORRUPTION). And finally, when generic_shutdown_super() detects remaining inodes on a system without CONFIG_BUG_ON_DATA_CORRUPTION, it would be nice if later accesses to a busy inode would at least crash somewhat cleanly rather than walking through freed memory. To address all three, use CHECK_DATA_CORRUPTION() when kernel bugs are detected. Signed-off-by: Jann Horn <jannh@google.com> Reviewed-by: Christian Brauner (Microsoft) <brauner@kernel.org> Reviewed-by: Kees Cook <keescook@chromium.org> Signed-off-by: Christian Brauner (Microsoft) <brauner@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org> --- fs/open.c | 5 +++-- fs/super.c | 21 +++++++++++++++++---- include/linux/poison.h | 3 +++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/fs/open.c b/fs/open.c index a81319b6177f6..7853deb6fcf47 100644 --- a/fs/open.c +++ b/fs/open.c @@ -1411,8 +1411,9 @@ int filp_close(struct file *filp, fl_owner_t id) { int retval = 0; - if (!file_count(filp)) { - printk(KERN_ERR "VFS: Close: file count is 0\n"); + if (CHECK_DATA_CORRUPTION(file_count(filp) == 0, + "VFS: Close: file count is 0 (f_op=%ps)", + filp->f_op)) { return 0; } diff --git a/fs/super.c b/fs/super.c index 8d39e4f11cfa3..4f8a626a35cd9 100644 --- a/fs/super.c +++ b/fs/super.c @@ -491,10 +491,23 @@ void generic_shutdown_super(struct super_block *sb) if (sop->put_super) sop->put_super(sb); - if (!list_empty(&sb->s_inodes)) { - printk("VFS: Busy inodes after unmount of %s. " - "Self-destruct in 5 seconds. Have a nice day...\n", - sb->s_id); + if (CHECK_DATA_CORRUPTION(!list_empty(&sb->s_inodes), + "VFS: Busy inodes after unmount of %s (%s)", + sb->s_id, sb->s_type->name)) { + /* + * Adding a proper bailout path here would be hard, but + * we can at least make it more likely that a later + * iput_final() or such crashes cleanly. + */ + struct inode *inode; + + spin_lock(&sb->s_inode_list_lock); + list_for_each_entry(inode, &sb->s_inodes, i_sb_list) { + inode->i_op = VFS_PTR_POISON; + inode->i_sb = VFS_PTR_POISON; + inode->i_mapping = VFS_PTR_POISON; + } + spin_unlock(&sb->s_inode_list_lock); } } spin_lock(&sb_lock); diff --git a/include/linux/poison.h b/include/linux/poison.h index 2d3249eb0e62d..0e8a1f2ceb2f1 100644 --- a/include/linux/poison.h +++ b/include/linux/poison.h @@ -84,4 +84,7 @@ /********** kernel/bpf/ **********/ #define BPF_PTR_POISON ((void *)(0xeB9FUL + POISON_POINTER_DELTA)) +/********** VFS **********/ +#define VFS_PTR_POISON ((void *)(0xF5 + POISON_POINTER_DELTA)) + #endif -- 2.39.0 ^ permalink raw reply related [flat|nested] 83+ messages in thread
* [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() [not found] <20230226034256.771769-1-sashal@kernel.org> 2023-02-26 3:42 ` [PATCH AUTOSEL 6.1 07/21] fs: Use CHECK_DATA_CORRUPTION() when kernel bugs are detected Sasha Levin @ 2023-02-26 3:42 ` Sasha Levin 2023-02-26 4:07 ` Eric Biggers 1 sibling, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-02-26 3:42 UTC (permalink / raw) To: linux-kernel, stable; +Cc: Eric Biggers, Sasha Levin, viro, linux-fsdevel From: Eric Biggers <ebiggers@google.com> [ Upstream commit ec64036e68634231f5891faa2b7a81cdc5dcd001 ] Now that the key associated with the "test_dummy_operation" mount option is added on-demand when it's needed, rather than immediately when the filesystem is mounted, fscrypt_destroy_keyring() no longer needs to be called from __put_super() to avoid a memory leak on mount failure. Remove this call, which was causing confusion because it appeared to be a sleep-in-atomic bug (though it wasn't, for a somewhat-subtle reason). Signed-off-by: Eric Biggers <ebiggers@google.com> Link: https://lore.kernel.org/r/20230208062107.199831-5-ebiggers@kernel.org Signed-off-by: Sasha Levin <sashal@kernel.org> --- fs/super.c | 1 - 1 file changed, 1 deletion(-) diff --git a/fs/super.c b/fs/super.c index 4f8a626a35cd9..76d47620b930d 100644 --- a/fs/super.c +++ b/fs/super.c @@ -291,7 +291,6 @@ static void __put_super(struct super_block *s) WARN_ON(s->s_inode_lru.node); WARN_ON(!list_empty(&s->s_mounts)); security_sb_free(s); - fscrypt_destroy_keyring(s); put_user_ns(s->s_user_ns); kfree(s->s_subtype); call_rcu(&s->rcu, destroy_super_rcu); -- 2.39.0 ^ permalink raw reply related [flat|nested] 83+ messages in thread
* Re: [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() 2023-02-26 3:42 ` [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() Sasha Levin @ 2023-02-26 4:07 ` Eric Biggers 2023-02-26 5:30 ` Eric Biggers 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-02-26 4:07 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Sat, Feb 25, 2023 at 10:42:47PM -0500, Sasha Levin wrote: > From: Eric Biggers <ebiggers@google.com> > > [ Upstream commit ec64036e68634231f5891faa2b7a81cdc5dcd001 ] > > Now that the key associated with the "test_dummy_operation" mount option > is added on-demand when it's needed, rather than immediately when the > filesystem is mounted, fscrypt_destroy_keyring() no longer needs to be > called from __put_super() to avoid a memory leak on mount failure. > > Remove this call, which was causing confusion because it appeared to be > a sleep-in-atomic bug (though it wasn't, for a somewhat-subtle reason). > > Signed-off-by: Eric Biggers <ebiggers@google.com> > Link: https://lore.kernel.org/r/20230208062107.199831-5-ebiggers@kernel.org > Signed-off-by: Sasha Levin <sashal@kernel.org> Why is this being backported? - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() 2023-02-26 4:07 ` Eric Biggers @ 2023-02-26 5:30 ` Eric Biggers 2023-02-26 19:24 ` Eric Biggers 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-02-26 5:30 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Sat, Feb 25, 2023 at 08:07:55PM -0800, Eric Biggers wrote: > On Sat, Feb 25, 2023 at 10:42:47PM -0500, Sasha Levin wrote: > > From: Eric Biggers <ebiggers@google.com> > > > > [ Upstream commit ec64036e68634231f5891faa2b7a81cdc5dcd001 ] > > > > Now that the key associated with the "test_dummy_operation" mount option > > is added on-demand when it's needed, rather than immediately when the > > filesystem is mounted, fscrypt_destroy_keyring() no longer needs to be > > called from __put_super() to avoid a memory leak on mount failure. > > > > Remove this call, which was causing confusion because it appeared to be > > a sleep-in-atomic bug (though it wasn't, for a somewhat-subtle reason). > > > > Signed-off-by: Eric Biggers <ebiggers@google.com> > > Link: https://lore.kernel.org/r/20230208062107.199831-5-ebiggers@kernel.org > > Signed-off-by: Sasha Levin <sashal@kernel.org> > > Why is this being backported? > > - Eric BTW, can you please permanently exclude all commits authored by me from AUTOSEL so that I don't have to repeatedly complain about every commit individually? Especially when these mails often come on weekends and holidays. I know how to use Cc stable, and how to ask explicitly for a stable backport if I find out after the fact that one is needed. (And other real people can always ask too... not counting AUTOSEL, even though you are sending the AUTOSEL emails, since clearly they go through no or very little human review.) Of course, it's not just me that AUTOSEL isn't working for. So, you'll still continue backporting random commits that I have to spend hours bisecting, e.g. https://lore.kernel.org/stable/20220921155332.234913-7-sashal@kernel.org. But at least I won't have to deal with this garbage for my own commits. Now, I'm not sure I'll get a response to this --- I received no response to my last AUTOSEL question at https://lore.kernel.org/stable/Y1DTFiP12ws04eOM@sol.localdomain. So to hopefully entice you to actually do something, I'm also letting you know that I won't be reviewing any AUTOSEL mails for my commits anymore. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() 2023-02-26 5:30 ` Eric Biggers @ 2023-02-26 19:24 ` Eric Biggers 2023-02-26 19:33 ` Slade Watkins 2023-02-27 14:18 ` Sasha Levin 0 siblings, 2 replies; 83+ messages in thread From: Eric Biggers @ 2023-02-26 19:24 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Sat, Feb 25, 2023 at 09:30:37PM -0800, Eric Biggers wrote: > On Sat, Feb 25, 2023 at 08:07:55PM -0800, Eric Biggers wrote: > > On Sat, Feb 25, 2023 at 10:42:47PM -0500, Sasha Levin wrote: > > > From: Eric Biggers <ebiggers@google.com> > > > > > > [ Upstream commit ec64036e68634231f5891faa2b7a81cdc5dcd001 ] > > > > > > Now that the key associated with the "test_dummy_operation" mount option > > > is added on-demand when it's needed, rather than immediately when the > > > filesystem is mounted, fscrypt_destroy_keyring() no longer needs to be > > > called from __put_super() to avoid a memory leak on mount failure. > > > > > > Remove this call, which was causing confusion because it appeared to be > > > a sleep-in-atomic bug (though it wasn't, for a somewhat-subtle reason). > > > > > > Signed-off-by: Eric Biggers <ebiggers@google.com> > > > Link: https://lore.kernel.org/r/20230208062107.199831-5-ebiggers@kernel.org > > > Signed-off-by: Sasha Levin <sashal@kernel.org> > > > > Why is this being backported? > > > > - Eric > > BTW, can you please permanently exclude all commits authored by me from AUTOSEL > so that I don't have to repeatedly complain about every commit individually? > Especially when these mails often come on weekends and holidays. > > I know how to use Cc stable, and how to ask explicitly for a stable backport if > I find out after the fact that one is needed. (And other real people can always > ask too... not counting AUTOSEL, even though you are sending the AUTOSEL emails, > since clearly they go through no or very little human review.) > > Of course, it's not just me that AUTOSEL isn't working for. So, you'll still > continue backporting random commits that I have to spend hours bisecting, e.g. > https://lore.kernel.org/stable/20220921155332.234913-7-sashal@kernel.org. > > But at least I won't have to deal with this garbage for my own commits. > > Now, I'm not sure I'll get a response to this --- I received no response to my > last AUTOSEL question at > https://lore.kernel.org/stable/Y1DTFiP12ws04eOM@sol.localdomain. So to > hopefully entice you to actually do something, I'm also letting you know that I > won't be reviewing any AUTOSEL mails for my commits anymore. > The really annoying thing is that someone even replied to your AUTOSEL email for that broken patch and told you it is broken (https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org), and ***you ignored it and applied the patch anyway***. Why are you even sending these emails if you are ignoring feedback anyway? How do I even get you to not apply a patch? Is it even possible? I guess I might as well just add an email filter that auto-deletes all AUTOSEL emails, as apparently there's no point in responding anyway? - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() 2023-02-26 19:24 ` Eric Biggers @ 2023-02-26 19:33 ` Slade Watkins 2023-02-27 14:18 ` Sasha Levin 1 sibling, 0 replies; 83+ messages in thread From: Slade Watkins @ 2023-02-26 19:33 UTC (permalink / raw) To: Eric Biggers; +Cc: Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On Sun, Feb 26, 2023 at 2:24 PM Eric Biggers <ebiggers@kernel.org> wrote: > > On Sat, Feb 25, 2023 at 09:30:37PM -0800, Eric Biggers wrote: > > On Sat, Feb 25, 2023 at 08:07:55PM -0800, Eric Biggers wrote: > > > On Sat, Feb 25, 2023 at 10:42:47PM -0500, Sasha Levin wrote: > > > > From: Eric Biggers <ebiggers@google.com> > > > > > > > > [ Upstream commit ec64036e68634231f5891faa2b7a81cdc5dcd001 ] > > > > > > > > Now that the key associated with the "test_dummy_operation" mount option > > > > is added on-demand when it's needed, rather than immediately when the > > > > filesystem is mounted, fscrypt_destroy_keyring() no longer needs to be > > > > called from __put_super() to avoid a memory leak on mount failure. > > > > > > > > Remove this call, which was causing confusion because it appeared to be > > > > a sleep-in-atomic bug (though it wasn't, for a somewhat-subtle reason). > > > > > > > > Signed-off-by: Eric Biggers <ebiggers@google.com> > > > > Link: https://lore.kernel.org/r/20230208062107.199831-5-ebiggers@kernel.org > > > > Signed-off-by: Sasha Levin <sashal@kernel.org> > > > > > > Why is this being backported? > > > > > > - Eric > > > > BTW, can you please permanently exclude all commits authored by me from AUTOSEL > > so that I don't have to repeatedly complain about every commit individually? > > Especially when these mails often come on weekends and holidays. > > > > I know how to use Cc stable, and how to ask explicitly for a stable backport if > > I find out after the fact that one is needed. (And other real people can always > > ask too... not counting AUTOSEL, even though you are sending the AUTOSEL emails, > > since clearly they go through no or very little human review.) > > > > Of course, it's not just me that AUTOSEL isn't working for. So, you'll still > > continue backporting random commits that I have to spend hours bisecting, e.g. > > https://lore.kernel.org/stable/20220921155332.234913-7-sashal@kernel.org. > > > > But at least I won't have to deal with this garbage for my own commits. > > > > Now, I'm not sure I'll get a response to this --- I received no response to my > > last AUTOSEL question at > > https://lore.kernel.org/stable/Y1DTFiP12ws04eOM@sol.localdomain. So to > > hopefully entice you to actually do something, I'm also letting you know that I > > won't be reviewing any AUTOSEL mails for my commits anymore. > > > > The really annoying thing is that someone even replied to your AUTOSEL email for > that broken patch and told you it is broken > (https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org), > and ***you ignored it and applied the patch anyway***. > > Why are you even sending these emails if you are ignoring feedback anyway? > > How do I even get you to not apply a patch? Is it even possible? > > I guess I might as well just add an email filter that auto-deletes all AUTOSEL > emails, as apparently there's no point in responding anyway? I test this branch for Greg but don't pay attention to these emails Sasha sends out (because there's just waaaaay too many of them to look through unless they get a reply; I find them quite annoying otherwise.) But if these commits automatically get applied to stable trees, even with objections from the committers, then I personally question the methodology for having AUTOSEL in the first place. Commits should be tested and backported with explicit purpose by their developers, IMO. -- Slade ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() 2023-02-26 19:24 ` Eric Biggers 2023-02-26 19:33 ` Slade Watkins @ 2023-02-27 14:18 ` Sasha Levin 2023-02-27 17:47 ` AUTOSEL process Eric Biggers 1 sibling, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-02-27 14:18 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-kernel, stable, viro, linux-fsdevel On Sun, Feb 26, 2023 at 11:24:36AM -0800, Eric Biggers wrote: >On Sat, Feb 25, 2023 at 09:30:37PM -0800, Eric Biggers wrote: >> On Sat, Feb 25, 2023 at 08:07:55PM -0800, Eric Biggers wrote: >> > On Sat, Feb 25, 2023 at 10:42:47PM -0500, Sasha Levin wrote: >> > > From: Eric Biggers <ebiggers@google.com> >> > > >> > > [ Upstream commit ec64036e68634231f5891faa2b7a81cdc5dcd001 ] >> > > >> > > Now that the key associated with the "test_dummy_operation" mount option >> > > is added on-demand when it's needed, rather than immediately when the >> > > filesystem is mounted, fscrypt_destroy_keyring() no longer needs to be >> > > called from __put_super() to avoid a memory leak on mount failure. >> > > >> > > Remove this call, which was causing confusion because it appeared to be >> > > a sleep-in-atomic bug (though it wasn't, for a somewhat-subtle reason). >> > > >> > > Signed-off-by: Eric Biggers <ebiggers@google.com> >> > > Link: https://lore.kernel.org/r/20230208062107.199831-5-ebiggers@kernel.org >> > > Signed-off-by: Sasha Levin <sashal@kernel.org> >> > >> > Why is this being backported? >> > >> > - Eric >> >> BTW, can you please permanently exclude all commits authored by me from AUTOSEL >> so that I don't have to repeatedly complain about every commit individually? >> Especially when these mails often come on weekends and holidays. Yup, no problem - I'll ignore any commits authored by you. >> I know how to use Cc stable, and how to ask explicitly for a stable backport if >> I find out after the fact that one is needed. (And other real people can always >> ask too... not counting AUTOSEL, even though you are sending the AUTOSEL emails, >> since clearly they go through no or very little human review.) One of the challanges here is that it's difficult to solicit reviews or really any interaction from authors after a commit lands upstream. Look at the response rates to Greg's "FAILED" emails that ask authors to provide backports to commits they tagged for stable. >> Of course, it's not just me that AUTOSEL isn't working for. So, you'll still >> continue backporting random commits that I have to spend hours bisecting, e.g. >> https://lore.kernel.org/stable/20220921155332.234913-7-sashal@kernel.org. >> >> But at least I won't have to deal with this garbage for my own commits. >> >> Now, I'm not sure I'll get a response to this --- I received no response to my >> last AUTOSEL question at >> https://lore.kernel.org/stable/Y1DTFiP12ws04eOM@sol.localdomain. So to >> hopefully entice you to actually do something, I'm also letting you know that I >> won't be reviewing any AUTOSEL mails for my commits anymore. >> > >The really annoying thing is that someone even replied to your AUTOSEL email for >that broken patch and told you it is broken >(https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org), >and ***you ignored it and applied the patch anyway***. > >Why are you even sending these emails if you are ignoring feedback anyway? I obviously didn't ignore it on purpose, right? -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 14:18 ` Sasha Levin @ 2023-02-27 17:47 ` Eric Biggers 2023-02-27 18:06 ` Eric Biggers 2023-03-13 3:37 ` Bagas Sanjaya 0 siblings, 2 replies; 83+ messages in thread From: Eric Biggers @ 2023-02-27 17:47 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 09:18:59AM -0500, Sasha Levin wrote: > On Sun, Feb 26, 2023 at 11:24:36AM -0800, Eric Biggers wrote: > > On Sat, Feb 25, 2023 at 09:30:37PM -0800, Eric Biggers wrote: > > > On Sat, Feb 25, 2023 at 08:07:55PM -0800, Eric Biggers wrote: > > > > On Sat, Feb 25, 2023 at 10:42:47PM -0500, Sasha Levin wrote: > > > > > From: Eric Biggers <ebiggers@google.com> > > > > > > > > > > [ Upstream commit ec64036e68634231f5891faa2b7a81cdc5dcd001 ] > > > > > > > > > > Now that the key associated with the "test_dummy_operation" mount option > > > > > is added on-demand when it's needed, rather than immediately when the > > > > > filesystem is mounted, fscrypt_destroy_keyring() no longer needs to be > > > > > called from __put_super() to avoid a memory leak on mount failure. > > > > > > > > > > Remove this call, which was causing confusion because it appeared to be > > > > > a sleep-in-atomic bug (though it wasn't, for a somewhat-subtle reason). > > > > > > > > > > Signed-off-by: Eric Biggers <ebiggers@google.com> > > > > > Link: https://lore.kernel.org/r/20230208062107.199831-5-ebiggers@kernel.org > > > > > Signed-off-by: Sasha Levin <sashal@kernel.org> > > > > > > > > Why is this being backported? > > > > > > > > - Eric > > > > > > BTW, can you please permanently exclude all commits authored by me from AUTOSEL > > > so that I don't have to repeatedly complain about every commit individually? > > > Especially when these mails often come on weekends and holidays. > > Yup, no problem - I'll ignore any commits authored by you. > > > > I know how to use Cc stable, and how to ask explicitly for a stable backport if > > > I find out after the fact that one is needed. (And other real people can always > > > ask too... not counting AUTOSEL, even though you are sending the AUTOSEL emails, > > > since clearly they go through no or very little human review.) > > One of the challanges here is that it's difficult to solicit reviews or > really any interaction from authors after a commit lands upstream. Look > at the response rates to Greg's "FAILED" emails that ask authors to > provide backports to commits they tagged for stable. Well, it doesn't help that most of the stable emails aren't sent to the subsystem's mailing list, but instead just to the individual people mentioned in the commit. So many people who would like to help never know about it. > > > Of course, it's not just me that AUTOSEL isn't working for. So, you'll still > > > continue backporting random commits that I have to spend hours bisecting, e.g. > > > https://lore.kernel.org/stable/20220921155332.234913-7-sashal@kernel.org. > > > > > > But at least I won't have to deal with this garbage for my own commits. > > > > > > Now, I'm not sure I'll get a response to this --- I received no response to my > > > last AUTOSEL question at > > > https://lore.kernel.org/stable/Y1DTFiP12ws04eOM@sol.localdomain. So to > > > hopefully entice you to actually do something, I'm also letting you know that I > > > won't be reviewing any AUTOSEL mails for my commits anymore. > > > > > > > The really annoying thing is that someone even replied to your AUTOSEL email for > > that broken patch and told you it is broken > > (https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org), > > and ***you ignored it and applied the patch anyway***. > > > > Why are you even sending these emails if you are ignoring feedback anyway? > > I obviously didn't ignore it on purpose, right? > I don't know, is it obvious? You've said in the past that sometimes you'd like to backport a commit even if the maintainer objects and/or it is known buggy. https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org also didn't explicitly say "Don't backport this" but instead "This patch has issues", so maybe that made a difference? Anyway, the fact is that it happened. And if it happened in the one bug that I happened to look at because it personally affected me and I spent hours bisecting, it probably is happening in lots of other cases too. So it seems the process is not working... Separately from responses to the AUTOSEL email, it also seems that you aren't checking for any reported regressions or pending fixes for a commit before backporting it. Simply searching lore for the commit title https://lore.kernel.org/all/?q=%22drm%2Famdgpu%3A+use+dirty+framebuffer+helper%22 would have turned up the bug report https://lore.kernel.org/dri-devel/20220918120926.10322-1-user@am64/ that bisected a regression to that commit, as well as a patch that Fixes that commit: https://lore.kernel.org/all/20220920130832.2214101-1-alexander.deucher@amd.com/ Both of these existed before you even sent the AUTOSEL email! So to summarize, that buggy commit was backported even though: * There were no indications that it was a bug fix (and thus potentially suitable for stable) in the first place. * On the AUTOSEL thread, someone told you the commit is broken. * There was already a thread that reported a regression caused by the commit. Easily findable via lore search. * There was also already a pending patch that Fixes the commit. Again easily findable via lore search. So it seems a *lot* of things went wrong, no? Why? If so many things can go wrong, it's not just a "mistake" but rather the process is the problem... - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 17:47 ` AUTOSEL process Eric Biggers @ 2023-02-27 18:06 ` Eric Biggers 2023-02-27 20:39 ` Sasha Levin 2023-03-07 21:18 ` Pavel Machek 2023-03-13 3:37 ` Bagas Sanjaya 1 sibling, 2 replies; 83+ messages in thread From: Eric Biggers @ 2023-02-27 18:06 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 09:47:48AM -0800, Eric Biggers wrote: > > > > Of course, it's not just me that AUTOSEL isn't working for. So, you'll still > > > > continue backporting random commits that I have to spend hours bisecting, e.g. > > > > https://lore.kernel.org/stable/20220921155332.234913-7-sashal@kernel.org. > > > > > > > > But at least I won't have to deal with this garbage for my own commits. > > > > > > > > Now, I'm not sure I'll get a response to this --- I received no response to my > > > > last AUTOSEL question at > > > > https://lore.kernel.org/stable/Y1DTFiP12ws04eOM@sol.localdomain. So to > > > > hopefully entice you to actually do something, I'm also letting you know that I > > > > won't be reviewing any AUTOSEL mails for my commits anymore. > > > > > > > > > > The really annoying thing is that someone even replied to your AUTOSEL email for > > > that broken patch and told you it is broken > > > (https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org), > > > and ***you ignored it and applied the patch anyway***. > > > > > > Why are you even sending these emails if you are ignoring feedback anyway? > > > > I obviously didn't ignore it on purpose, right? > > > > I don't know, is it obvious? You've said in the past that sometimes you'd like > to backport a commit even if the maintainer objects and/or it is known buggy. > https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org > also didn't explicitly say "Don't backport this" but instead "This patch has > issues", so maybe that made a difference? > > Anyway, the fact is that it happened. And if it happened in the one bug that I > happened to look at because it personally affected me and I spent hours > bisecting, it probably is happening in lots of other cases too. So it seems the > process is not working... > > Separately from responses to the AUTOSEL email, it also seems that you aren't > checking for any reported regressions or pending fixes for a commit before > backporting it. Simply searching lore for the commit title > https://lore.kernel.org/all/?q=%22drm%2Famdgpu%3A+use+dirty+framebuffer+helper%22 > would have turned up the bug report > https://lore.kernel.org/dri-devel/20220918120926.10322-1-user@am64/ that > bisected a regression to that commit, as well as a patch that Fixes that commit: > https://lore.kernel.org/all/20220920130832.2214101-1-alexander.deucher@amd.com/ > Both of these existed before you even sent the AUTOSEL email! > > So to summarize, that buggy commit was backported even though: > > * There were no indications that it was a bug fix (and thus potentially > suitable for stable) in the first place. > * On the AUTOSEL thread, someone told you the commit is broken. > * There was already a thread that reported a regression caused by the commit. > Easily findable via lore search. > * There was also already a pending patch that Fixes the commit. Again easily > findable via lore search. > > So it seems a *lot* of things went wrong, no? Why? If so many things can go > wrong, it's not just a "mistake" but rather the process is the problem... BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after only being in mainline for 4 days, and *released* in all LTS kernels after only being in mainline for 12 days. Surely that's a timeline befitting a critical security vulnerability, not some random neural-network-selected commit that wasn't even fixing anything? - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 18:06 ` Eric Biggers @ 2023-02-27 20:39 ` Sasha Levin 2023-02-27 21:38 ` Eric Biggers 2023-03-07 21:18 ` Pavel Machek 1 sibling, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-02-27 20:39 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 10:06:32AM -0800, Eric Biggers wrote: >On Mon, Feb 27, 2023 at 09:47:48AM -0800, Eric Biggers wrote: >> > > > Of course, it's not just me that AUTOSEL isn't working for. So, you'll still >> > > > continue backporting random commits that I have to spend hours bisecting, e.g. >> > > > https://lore.kernel.org/stable/20220921155332.234913-7-sashal@kernel.org. >> > > > >> > > > But at least I won't have to deal with this garbage for my own commits. >> > > > >> > > > Now, I'm not sure I'll get a response to this --- I received no response to my >> > > > last AUTOSEL question at >> > > > https://lore.kernel.org/stable/Y1DTFiP12ws04eOM@sol.localdomain. So to >> > > > hopefully entice you to actually do something, I'm also letting you know that I >> > > > won't be reviewing any AUTOSEL mails for my commits anymore. >> > > > >> > > >> > > The really annoying thing is that someone even replied to your AUTOSEL email for >> > > that broken patch and told you it is broken >> > > (https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org), >> > > and ***you ignored it and applied the patch anyway***. >> > > >> > > Why are you even sending these emails if you are ignoring feedback anyway? >> > >> > I obviously didn't ignore it on purpose, right? >> > >> >> I don't know, is it obvious? You've said in the past that sometimes you'd like >> to backport a commit even if the maintainer objects and/or it is known buggy. >> https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org >> also didn't explicitly say "Don't backport this" but instead "This patch has >> issues", so maybe that made a difference? From what I gather I missed the reply - I would not blindly ignore a maintainer. >> Anyway, the fact is that it happened. And if it happened in the one bug that I >> happened to look at because it personally affected me and I spent hours >> bisecting, it probably is happening in lots of other cases too. So it seems the >> process is not working... This one is tricky, becuase we also end up taking a lot of commits that do fix real bugs, and were never tagged for stable or even had a fixes tag. Maybe I should run the numbers again, but when we compared regression rates of stable tagged releases and AUTOSEL ones, it was fairly identical. >> Separately from responses to the AUTOSEL email, it also seems that you aren't >> checking for any reported regressions or pending fixes for a commit before >> backporting it. Simply searching lore for the commit title >> https://lore.kernel.org/all/?q=%22drm%2Famdgpu%3A+use+dirty+framebuffer+helper%22 >> would have turned up the bug report >> https://lore.kernel.org/dri-devel/20220918120926.10322-1-user@am64/ that >> bisected a regression to that commit, as well as a patch that Fixes that commit: >> https://lore.kernel.org/all/20220920130832.2214101-1-alexander.deucher@amd.com/ >> Both of these existed before you even sent the AUTOSEL email! I would love to have a way to automatically grep lore for reported issues that are pinpointed to a given commit. I'm hoping that Thorsten's regression tracker could be used that way soon enough. >> So to summarize, that buggy commit was backported even though: >> >> * There were no indications that it was a bug fix (and thus potentially >> suitable for stable) in the first place. >> * On the AUTOSEL thread, someone told you the commit is broken. >> * There was already a thread that reported a regression caused by the commit. >> Easily findable via lore search. >> * There was also already a pending patch that Fixes the commit. Again easily >> findable via lore search. >> >> So it seems a *lot* of things went wrong, no? Why? If so many things can go >> wrong, it's not just a "mistake" but rather the process is the problem... > >BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after >only being in mainline for 4 days, and *released* in all LTS kernels after only >being in mainline for 12 days. Surely that's a timeline befitting a critical >security vulnerability, not some random neural-network-selected commit that >wasn't even fixing anything? I would love to have a mechanism that tells me with 100% confidence if a given commit fixes a bug or not, could you provide me with one? w.r.t timelines, this is something that was discussed on the mailing list a few years ago where we decided that giving AUTOSEL commits 7 days of soaking time is sufficient, if anything changed we can have this discussion again. Note, however, that it's not enough to keep pointing at a tiny set and using it to suggest that the entire process is broken. How many AUTOSEL commits introduced a regression? How many -stable tagged ones did? How many bugs did AUTOSEL commits fix? -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 20:39 ` Sasha Levin @ 2023-02-27 21:38 ` Eric Biggers 2023-02-27 22:35 ` Sasha Levin 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-02-27 21:38 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 03:39:14PM -0500, Sasha Levin wrote: > > > So to summarize, that buggy commit was backported even though: > > > > > > * There were no indications that it was a bug fix (and thus potentially > > > suitable for stable) in the first place. > > > * On the AUTOSEL thread, someone told you the commit is broken. > > > * There was already a thread that reported a regression caused by the commit. > > > Easily findable via lore search. > > > * There was also already a pending patch that Fixes the commit. Again easily > > > findable via lore search. > > > > > > So it seems a *lot* of things went wrong, no? Why? If so many things can go > > > wrong, it's not just a "mistake" but rather the process is the problem... > > > > BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after > > only being in mainline for 4 days, and *released* in all LTS kernels after only > > being in mainline for 12 days. Surely that's a timeline befitting a critical > > security vulnerability, not some random neural-network-selected commit that > > wasn't even fixing anything? > > I would love to have a mechanism that tells me with 100% confidence if a > given commit fixes a bug or not, could you provide me with one? Just because you can't be 100% certain whether a commit is a fix doesn't mean you should be rushing to backport random commits that have no indications they are fixing anything. > w.r.t timelines, this is something that was discussed on the mailing > list a few years ago where we decided that giving AUTOSEL commits 7 days > of soaking time is sufficient, if anything changed we can have this > discussion again. Nothing has changed, but that doesn't mean that your process is actually working. 7 days might be appropriate for something that looks like a security fix, but not for a random commit with no indications it is fixing anything. BTW, based on that example it's not even 7 days between AUTOSEL and patch applied, but actually 7 days from AUTOSEL to *release*. So e.g. if someone takes just a 1 week vacation, in that time a commit they would have NAK'ed can be AUTOSEL'ed and pushed out across all LTS kernels... > Note, however, that it's not enough to keep pointing at a tiny set and > using it to suggest that the entire process is broken. How many AUTOSEL > commits introduced a regression? How many -stable tagged ones did? How > many bugs did AUTOSEL commits fix? So basically you don't accept feedback from individual people, as individual people don't have enough data? - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 21:38 ` Eric Biggers @ 2023-02-27 22:35 ` Sasha Levin 2023-02-27 22:59 ` Matthew Wilcox 2023-02-28 0:32 ` Eric Biggers 0 siblings, 2 replies; 83+ messages in thread From: Sasha Levin @ 2023-02-27 22:35 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 09:38:46PM +0000, Eric Biggers wrote: >On Mon, Feb 27, 2023 at 03:39:14PM -0500, Sasha Levin wrote: >> > > So to summarize, that buggy commit was backported even though: >> > > >> > > * There were no indications that it was a bug fix (and thus potentially >> > > suitable for stable) in the first place. >> > > * On the AUTOSEL thread, someone told you the commit is broken. >> > > * There was already a thread that reported a regression caused by the commit. >> > > Easily findable via lore search. >> > > * There was also already a pending patch that Fixes the commit. Again easily >> > > findable via lore search. >> > > >> > > So it seems a *lot* of things went wrong, no? Why? If so many things can go >> > > wrong, it's not just a "mistake" but rather the process is the problem... >> > >> > BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after >> > only being in mainline for 4 days, and *released* in all LTS kernels after only >> > being in mainline for 12 days. Surely that's a timeline befitting a critical >> > security vulnerability, not some random neural-network-selected commit that >> > wasn't even fixing anything? >> >> I would love to have a mechanism that tells me with 100% confidence if a >> given commit fixes a bug or not, could you provide me with one? > >Just because you can't be 100% certain whether a commit is a fix doesn't mean >you should be rushing to backport random commits that have no indications they >are fixing anything. The difference in opinion here is that I don't think it's rushing: the stable kernel rules say a commit must be in a released kernel, while the AUTOSEL timelines make it so a commit must have been in two released kernels. Should we extend it? Maybe, I just don't think we have enough data to make a decision either way. >> w.r.t timelines, this is something that was discussed on the mailing >> list a few years ago where we decided that giving AUTOSEL commits 7 days >> of soaking time is sufficient, if anything changed we can have this >> discussion again. > >Nothing has changed, but that doesn't mean that your process is actually >working. 7 days might be appropriate for something that looks like a security >fix, but not for a random commit with no indications it is fixing anything. How do we know if this is working or not though? How do you quantify the amount of useful commits? How do you know if a certain fix has security implications? Or even if it actually fixes anything? For every "security" commit tagged for stable I could probably list a "security" commit with no tags whatsoever. >BTW, based on that example it's not even 7 days between AUTOSEL and patch >applied, but actually 7 days from AUTOSEL to *release*. So e.g. if someone >takes just a 1 week vacation, in that time a commit they would have NAK'ed can >be AUTOSEL'ed and pushed out across all LTS kernels... Right, and same as above: what's "enough"? >> Note, however, that it's not enough to keep pointing at a tiny set and >> using it to suggest that the entire process is broken. How many AUTOSEL >> commits introduced a regression? How many -stable tagged ones did? How >> many bugs did AUTOSEL commits fix? > >So basically you don't accept feedback from individual people, as individual >people don't have enough data? I'd love to improve the process, but for that we need to figure out criteria for what we consider good or bad, collect data, and make decisions based on that data. What I'm getting from this thread is a few anecdotal examples and statements that the process isn't working at all. I took Jon's stablefixes script which he used for his previous articles around stable kernel regressions (here: https://lwn.net/Articles/812231/) and tried running it on the 5.15 stable tree (just a random pick). I've proceeded with ignoring the non-user-visible regressions as Jon defined in his article (basically issues that were introduced and fixed in the same releases) and ended up with 604 commits that caused a user visible regression. Out of those 604 commits: - 170 had an explicit stable tag. - 434 did not have a stable tag. Looking at the commits in the 5.15 tree: With stable tag: $ git log --oneline -i --grep "cc.*stable" v5.15..stable/linux-5.15.y | wc -l 3676 Without stable tag (-96 commits which are version bumps): $ git log --oneline --invert-grep -i --grep "cc.*stable" v5.15..stable/linux-5.15.y | wc -l 10649 Regression rate for commits with stable tag: 170 / 3676 = 4.62% Regression rate for commits without a stable tag: 434 / 10553 = 4.11% Is the analysis flawed somehow? Probably, and I'd happy take feedback on how/what I can do better, but this type of analysis is what I look for to know if the process is working well or not. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 22:35 ` Sasha Levin @ 2023-02-27 22:59 ` Matthew Wilcox 2023-02-28 0:52 ` Sasha Levin 2023-02-28 0:32 ` Eric Biggers 1 sibling, 1 reply; 83+ messages in thread From: Matthew Wilcox @ 2023-02-27 22:59 UTC (permalink / raw) To: Sasha Levin; +Cc: Eric Biggers, linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 05:35:30PM -0500, Sasha Levin wrote: > On Mon, Feb 27, 2023 at 09:38:46PM +0000, Eric Biggers wrote: > > Just because you can't be 100% certain whether a commit is a fix doesn't mean > > you should be rushing to backport random commits that have no indications they > > are fixing anything. > > The difference in opinion here is that I don't think it's rushing: the > stable kernel rules say a commit must be in a released kernel, while the > AUTOSEL timelines make it so a commit must have been in two released > kernels. Patches in -rc1 have been in _no_ released kernels. I'd feel a lot better about AUTOSEL if it didn't pick up changes until, say, -rc4, unless they were cc'd to stable. > > Nothing has changed, but that doesn't mean that your process is actually > > working. 7 days might be appropriate for something that looks like a security > > fix, but not for a random commit with no indications it is fixing anything. > > How do we know if this is working or not though? How do you quantify the > amount of useful commits? Sasha, 7 days is too short. People have to be allowed to take holiday. > I'd love to improve the process, but for that we need to figure out > criteria for what we consider good or bad, collect data, and make > decisions based on that data. > > What I'm getting from this thread is a few anecdotal examples and > statements that the process isn't working at all. > > I took Jon's stablefixes script which he used for his previous articles > around stable kernel regressions (here: > https://lwn.net/Articles/812231/) and tried running it on the 5.15 > stable tree (just a random pick). I've proceeded with ignoring the > non-user-visible regressions as Jon defined in his article (basically > issues that were introduced and fixed in the same releases) and ended up > with 604 commits that caused a user visible regression. > > Out of those 604 commits: > > - 170 had an explicit stable tag. > - 434 did not have a stable tag. I think a lot of people don't realise they have to _both_ put a Fixes tag _and_ add a Cc: stable. How many of those 604 commits had a Fixes tag? ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 22:59 ` Matthew Wilcox @ 2023-02-28 0:52 ` Sasha Levin 2023-02-28 1:25 ` Eric Biggers 2023-03-30 0:08 ` Eric Biggers 0 siblings, 2 replies; 83+ messages in thread From: Sasha Levin @ 2023-02-28 0:52 UTC (permalink / raw) To: Matthew Wilcox; +Cc: Eric Biggers, linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 10:59:27PM +0000, Matthew Wilcox wrote: >On Mon, Feb 27, 2023 at 05:35:30PM -0500, Sasha Levin wrote: >> On Mon, Feb 27, 2023 at 09:38:46PM +0000, Eric Biggers wrote: >> > Just because you can't be 100% certain whether a commit is a fix doesn't mean >> > you should be rushing to backport random commits that have no indications they >> > are fixing anything. >> >> The difference in opinion here is that I don't think it's rushing: the >> stable kernel rules say a commit must be in a released kernel, while the >> AUTOSEL timelines make it so a commit must have been in two released >> kernels. > >Patches in -rc1 have been in _no_ released kernels. I'd feel a lot >better about AUTOSEL if it didn't pick up changes until, say, -rc4, >unless they were cc'd to stable. This happened before my time, but -rc are considered releases. The counter point to your argument/ask is that if you run the numbers on regressions between -rc releases, it's the later one that tend to introduce (way) more issues. I've actually written about it a few years back to ksummit discuss (here: https://lwn.net/Articles/753329/) because the numbers I saw indicate that later -rc releases are 3x likely to introduce a regression. Linus pushed back on it saying that it is "by design" because those commits are way more complex than ones that land during the early -rc cycles. So yes, I don't mind modifying the release workflow to decrease the regressions we introduce, but I think that there's a difference between what folks see as "helpful" and the outcome it would have. >> > Nothing has changed, but that doesn't mean that your process is actually >> > working. 7 days might be appropriate for something that looks like a security >> > fix, but not for a random commit with no indications it is fixing anything. >> >> How do we know if this is working or not though? How do you quantify the >> amount of useful commits? > >Sasha, 7 days is too short. People have to be allowed to take holiday. That's true, and I don't have strong objections to making it longer. How often did it happen though? We don't end up getting too many replies past the 7 day window. I'll bump it to 14 days for a few months and see if it changes anything. >> I'd love to improve the process, but for that we need to figure out >> criteria for what we consider good or bad, collect data, and make >> decisions based on that data. >> >> What I'm getting from this thread is a few anecdotal examples and >> statements that the process isn't working at all. >> >> I took Jon's stablefixes script which he used for his previous articles >> around stable kernel regressions (here: >> https://lwn.net/Articles/812231/) and tried running it on the 5.15 >> stable tree (just a random pick). I've proceeded with ignoring the >> non-user-visible regressions as Jon defined in his article (basically >> issues that were introduced and fixed in the same releases) and ended up >> with 604 commits that caused a user visible regression. >> >> Out of those 604 commits: >> >> - 170 had an explicit stable tag. >> - 434 did not have a stable tag. > >I think a lot of people don't realise they have to _both_ put a Fixes >tag _and_ add a Cc: stable. How many of those 604 commits had a Fixes >tag? What do you mean? Just a cc: stable tag is enough to land it in stable, you don't have to do both. The numbers above reflect that. Running the numbers, there are 9422 commits with a Fixes tag in the 5.15 tree, out of which 360 had a regression, so 360 / 9422 = 3.82%. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 0:52 ` Sasha Levin @ 2023-02-28 1:25 ` Eric Biggers 2023-02-28 4:25 ` Willy Tarreau 2023-03-30 0:08 ` Eric Biggers 1 sibling, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-02-28 1:25 UTC (permalink / raw) To: Sasha Levin; +Cc: Matthew Wilcox, linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 07:52:39PM -0500, Sasha Levin wrote: > > > > Nothing has changed, but that doesn't mean that your process is actually > > > > working. 7 days might be appropriate for something that looks like a security > > > > fix, but not for a random commit with no indications it is fixing anything. > > > > > > How do we know if this is working or not though? How do you quantify the > > > amount of useful commits? > > > > Sasha, 7 days is too short. People have to be allowed to take holiday. > > That's true, and I don't have strong objections to making it longer. How > often did it happen though? We don't end up getting too many replies > past the 7 day window. > > I'll bump it to 14 days for a few months and see if it changes anything. It's not just for the review time, but also for the longer soak time in mainline. Of course, for that to work properly you have to actually take advantage of it, for example by re-checking for fixes when actually applying the patch, not just when sending the initial AUTOSEL email. Which I hope you're doing already, but who knows. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 1:25 ` Eric Biggers @ 2023-02-28 4:25 ` Willy Tarreau 0 siblings, 0 replies; 83+ messages in thread From: Willy Tarreau @ 2023-02-28 4:25 UTC (permalink / raw) To: Eric Biggers Cc: Sasha Levin, Matthew Wilcox, linux-kernel, stable, viro, linux-fsdevel On Tue, Feb 28, 2023 at 01:25:57AM +0000, Eric Biggers wrote: > On Mon, Feb 27, 2023 at 07:52:39PM -0500, Sasha Levin wrote: > > > > > Nothing has changed, but that doesn't mean that your process is actually > > > > > working. 7 days might be appropriate for something that looks like a security > > > > > fix, but not for a random commit with no indications it is fixing anything. > > > > > > > > How do we know if this is working or not though? How do you quantify the > > > > amount of useful commits? > > > > > > Sasha, 7 days is too short. People have to be allowed to take holiday. > > > > That's true, and I don't have strong objections to making it longer. How > > often did it happen though? We don't end up getting too many replies > > past the 7 day window. > > > > I'll bump it to 14 days for a few months and see if it changes anything. > > It's not just for the review time, but also for the longer soak time in > mainline. There's a tradeoff to find. I'm sure there are way many more stable users than mainline users. Does this mean we have to break stable from time to time to detect regressions ? Sadly, yes. And it's not specific to Linux, it's the same for virtually any other project that supports maintenance branches. I personally like the principle of delaying backports to older branches so that users relying on much older branches know they're having a much more stable experience because the rare regressions that happen in faster moving branches have more time to be caught. But that requires incredibly more complex management. On another project I'm sometimes seeing regressions caused by fixes pop up after 6 months of exposure in stable. And comparatively, regressions caused by new features tend to pop up faster, and users occasionally face such bugs in stable just because the backport got delayed. So there's no perfect balance, the problem is that any code change must be executed in field a few times to know if it's solid or not. The larger the exposure (i.e. stable) the faster regressions will be reported. The more frequent the code is triggered, the faster as well. Fixes for bugs that are very hard to trigger can cause regressions that will take ages to be reported. But nonetheless users want these fixes because most of the time they are correct. When I was maintaining extended LTS kernels such as 2.6.32 or 3.10, users were mostly upgrading when they were waiting for a specific fix. And even there, despite patches having being present in regular stable kernels for months, we faced regressions. Sometimes a fix was not suitable for that branch, sometimes it was incorrectly backported, etc. What's certain however, is that the longer you wait for a backport, the more difficult it becomes to find someone who still remembers well about that fix and its specificities, even during the review. This really has to be taken into account when suggesting increased delays. I really think that it's important to get backports early in recent stable branches. E.g. we could keep these 7 days till the last LTS branch, and let them cook one or two extra weeks before reaching older releases. But we wouldn't want to delay important fixes too much (which are still likely to cause regressions, especially security fixes which tend to focus on a specifically reported case). Maybe we could imagine that the Cc: stable could have a variant to mean "urgent" for important fixes that we want to bypass the wait time, and that by default other ones would flow a bit more slowly. This could satisfy most users by staying on the branch that brings them the update rate they need. But that would certainly be quite some extra work for Greg and for reviewers and that's certainly not cool, so we need to be reasonable here as well. Just my two cents, Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 0:52 ` Sasha Levin 2023-02-28 1:25 ` Eric Biggers @ 2023-03-30 0:08 ` Eric Biggers 2023-03-30 14:05 ` Sasha Levin 1 sibling, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-30 0:08 UTC (permalink / raw) To: Sasha Levin; +Cc: Matthew Wilcox, linux-kernel, stable, viro, linux-fsdevel Hi Sasha, On Mon, Feb 27, 2023 at 07:52:39PM -0500, Sasha Levin wrote: > > Sasha, 7 days is too short. People have to be allowed to take holiday. > > That's true, and I don't have strong objections to making it longer. How > often did it happen though? We don't end up getting too many replies > past the 7 day window. > > I'll bump it to 14 days for a few months and see if it changes anything. I see that for recent AUTOSEL patches you're still using 7 days. In fact, it seems you may have even decreased it further to 5 days: Sent Mar 14: https://lore.kernel.org/stable/20230314124435.471553-2-sashal@kernel.org Commited Mar 19: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?id=69aaf98f41593b95c012d91b3e5adeb8360b4b8d Any update on your plan to increase it to 14 days? I hope you can understand why I have to ask you --- you are the only person who can change your own policy. Thanks, - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-30 0:08 ` Eric Biggers @ 2023-03-30 14:05 ` Sasha Levin 2023-03-30 17:22 ` Eric Biggers 0 siblings, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-03-30 14:05 UTC (permalink / raw) To: Eric Biggers; +Cc: Matthew Wilcox, linux-kernel, stable, viro, linux-fsdevel On Thu, Mar 30, 2023 at 12:08:01AM +0000, Eric Biggers wrote: >Hi Sasha, > >On Mon, Feb 27, 2023 at 07:52:39PM -0500, Sasha Levin wrote: >> > Sasha, 7 days is too short. People have to be allowed to take holiday. >> >> That's true, and I don't have strong objections to making it longer. How >> often did it happen though? We don't end up getting too many replies >> past the 7 day window. >> >> I'll bump it to 14 days for a few months and see if it changes anything. > >I see that for recent AUTOSEL patches you're still using 7 days. In fact, it >seems you may have even decreased it further to 5 days: > > Sent Mar 14: https://lore.kernel.org/stable/20230314124435.471553-2-sashal@kernel.org > Commited Mar 19: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?id=69aaf98f41593b95c012d91b3e5adeb8360b4b8d > >Any update on your plan to increase it to 14 days? The commit you've pointed to was merged into Linus's tree on Feb 28th, and the first LTS tree that it was released in went out on March 22nd. Quoting the concern you've raised around the process: > BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after > only being in mainline for 4 days, and *released* in all LTS kernels after only > being in mainline for 12 days. Surely that's a timeline befitting a critical > security vulnerability, not some random neural-network-selected commit that > wasn't even fixing anything? So now there's at least 14 days between mainline inclusion and a release in an LTS kernel, does that not conform with what you thought I'd be doing? Most of that additional time comes in the form of me going over the tree and sending AUTOSEL mails a bit later, so I would be able to also pick follow-up fixes as they come in (and drop stuff that were reverted). As a side note, inclusion into the stable-queue which you've pointed to above is a few steps before a release - it's mostly a cheap way for us to get mileage out of bots that run on the queue and address issues - it doesn't mean that the commit is released. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-30 14:05 ` Sasha Levin @ 2023-03-30 17:22 ` Eric Biggers 2023-03-30 17:50 ` Sasha Levin 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-30 17:22 UTC (permalink / raw) To: Sasha Levin; +Cc: Matthew Wilcox, linux-kernel, stable, viro, linux-fsdevel On Thu, Mar 30, 2023 at 10:05:39AM -0400, Sasha Levin wrote: > On Thu, Mar 30, 2023 at 12:08:01AM +0000, Eric Biggers wrote: > > Hi Sasha, > > > > On Mon, Feb 27, 2023 at 07:52:39PM -0500, Sasha Levin wrote: > > > > Sasha, 7 days is too short. People have to be allowed to take holiday. > > > > > > That's true, and I don't have strong objections to making it longer. How > > > often did it happen though? We don't end up getting too many replies > > > past the 7 day window. > > > > > > I'll bump it to 14 days for a few months and see if it changes anything. > > > > I see that for recent AUTOSEL patches you're still using 7 days. In fact, it > > seems you may have even decreased it further to 5 days: > > > > Sent Mar 14: https://lore.kernel.org/stable/20230314124435.471553-2-sashal@kernel.org > > Commited Mar 19: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?id=69aaf98f41593b95c012d91b3e5adeb8360b4b8d > > > > Any update on your plan to increase it to 14 days? > > The commit you've pointed to was merged into Linus's tree on Feb 28th, > and the first LTS tree that it was released in went out on March 22nd. > > Quoting the concern you've raised around the process: > > > BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after > > only being in mainline for 4 days, and *released* in all LTS kernels after only > > being in mainline for 12 days. Surely that's a timeline befitting a critical > > security vulnerability, not some random neural-network-selected commit that > > wasn't even fixing anything? > > So now there's at least 14 days between mainline inclusion and a release > in an LTS kernel, does that not conform with what you thought I'd be > doing? Not quite. There are actually two different time periods: 1. The time from mainline to release 2. The time from AUTOSEL email to release (1) is a superset of (2), but concerns were raised about *both* time periods being too short. Especially (1), but also (2) because reviewers can miss the 7-day review e.g. if they are on vacation for a week. Yes, they can of course miss non-AUTOSEL patches too, *if* they happen to get merged quickly enough (most kernel patches take several weeks just to get to mainline). But, AUTOSEL patches are known to be low quality submissions that really need that review. I'm glad to hear that you've increased (1) to 14 days! However, that does not address (2). It also does not feel like much of a difference, since 12 days for (1) already seemed too short. To be honest, I hesitate a bit to give you a precise suggestion, as it's liable to be used to push back on future suggestions as "this is what people agreed on before". (Just as you did in this thread, with saying 7 days had been agreed on before.) And it's not like there are any magic numbers -- we just know that the current periods seem to be too short. But, for a simple change, I think increasing (2) to 14 days would be reasonable, as that automatically gives 14 days for (1) too. If it isn't too much trouble to separate the periods, though, it would also be reasonable to choose something a bit higher for (1), like 18-21 days, and something a bit lower for (2), like 10-12 days. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-30 17:22 ` Eric Biggers @ 2023-03-30 17:50 ` Sasha Levin 0 siblings, 0 replies; 83+ messages in thread From: Sasha Levin @ 2023-03-30 17:50 UTC (permalink / raw) To: Eric Biggers; +Cc: Matthew Wilcox, linux-kernel, stable, viro, linux-fsdevel On Thu, Mar 30, 2023 at 10:22:10AM -0700, Eric Biggers wrote: >On Thu, Mar 30, 2023 at 10:05:39AM -0400, Sasha Levin wrote: >> On Thu, Mar 30, 2023 at 12:08:01AM +0000, Eric Biggers wrote: >> > Hi Sasha, >> > >> > On Mon, Feb 27, 2023 at 07:52:39PM -0500, Sasha Levin wrote: >> > > > Sasha, 7 days is too short. People have to be allowed to take holiday. >> > > >> > > That's true, and I don't have strong objections to making it longer. How >> > > often did it happen though? We don't end up getting too many replies >> > > past the 7 day window. >> > > >> > > I'll bump it to 14 days for a few months and see if it changes anything. >> > >> > I see that for recent AUTOSEL patches you're still using 7 days. In fact, it >> > seems you may have even decreased it further to 5 days: >> > >> > Sent Mar 14: https://lore.kernel.org/stable/20230314124435.471553-2-sashal@kernel.org >> > Commited Mar 19: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?id=69aaf98f41593b95c012d91b3e5adeb8360b4b8d >> > >> > Any update on your plan to increase it to 14 days? >> >> The commit you've pointed to was merged into Linus's tree on Feb 28th, >> and the first LTS tree that it was released in went out on March 22nd. >> >> Quoting the concern you've raised around the process: >> >> > BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after >> > only being in mainline for 4 days, and *released* in all LTS kernels after only >> > being in mainline for 12 days. Surely that's a timeline befitting a critical >> > security vulnerability, not some random neural-network-selected commit that >> > wasn't even fixing anything? >> >> So now there's at least 14 days between mainline inclusion and a release >> in an LTS kernel, does that not conform with what you thought I'd be >> doing? > >Not quite. There are actually two different time periods: > >1. The time from mainline to release >2. The time from AUTOSEL email to release > >(1) is a superset of (2), but concerns were raised about *both* time periods >being too short. Especially (1), but also (2) because reviewers can miss the >7-day review e.g. if they are on vacation for a week. Yes, they can of course >miss non-AUTOSEL patches too, *if* they happen to get merged quickly enough >(most kernel patches take several weeks just to get to mainline). But, AUTOSEL >patches are known to be low quality submissions that really need that review. > >I'm glad to hear that you've increased (1) to 14 days! However, that does not >address (2). It also does not feel like much of a difference, since 12 days for >(1) already seemed too short. > >To be honest, I hesitate a bit to give you a precise suggestion, as it's liable >to be used to push back on future suggestions as "this is what people agreed on >before". (Just as you did in this thread, with saying 7 days had been agreed on >before.) And it's not like there are any magic numbers -- we just know that the >current periods seem to be too short. But, for a simple change, I think >increasing (2) to 14 days would be reasonable, as that automatically gives 14 >days for (1) too. If it isn't too much trouble to separate the periods, though, >it would also be reasonable to choose something a bit higher for (1), like 18-21 >days, and something a bit lower for (2), like 10-12 days. No objection on my end, I can enforce 18+ days for (1) and 14+ days for (2). I'd note that this isn't too far from what happened in the example in the previous mail: (1) happened in 23 days. (2) happened in 9 days. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 22:35 ` Sasha Levin 2023-02-27 22:59 ` Matthew Wilcox @ 2023-02-28 0:32 ` Eric Biggers 2023-02-28 1:53 ` Sasha Levin 1 sibling, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-02-28 0:32 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 05:35:30PM -0500, Sasha Levin wrote: > > > Note, however, that it's not enough to keep pointing at a tiny set and > > > using it to suggest that the entire process is broken. How many AUTOSEL > > > commits introduced a regression? How many -stable tagged ones did? How > > > many bugs did AUTOSEL commits fix? > > > > So basically you don't accept feedback from individual people, as individual > > people don't have enough data? > > I'd love to improve the process, but for that we need to figure out > criteria for what we consider good or bad, collect data, and make > decisions based on that data. > > What I'm getting from this thread is a few anecdotal examples and > statements that the process isn't working at all. > > I took Jon's stablefixes script which he used for his previous articles > around stable kernel regressions (here: > https://lwn.net/Articles/812231/) and tried running it on the 5.15 > stable tree (just a random pick). I've proceeded with ignoring the > non-user-visible regressions as Jon defined in his article (basically > issues that were introduced and fixed in the same releases) and ended up > with 604 commits that caused a user visible regression. > > Out of those 604 commits: > > - 170 had an explicit stable tag. > - 434 did not have a stable tag. > > Looking at the commits in the 5.15 tree: > > With stable tag: > > $ git log --oneline -i --grep "cc.*stable" v5.15..stable/linux-5.15.y | wc -l > 3676 > > Without stable tag (-96 commits which are version bumps): > > $ git log --oneline --invert-grep -i --grep "cc.*stable" v5.15..stable/linux-5.15.y | wc -l > 10649 > > Regression rate for commits with stable tag: 170 / 3676 = 4.62% > Regression rate for commits without a stable tag: 434 / 10553 = 4.11% > > Is the analysis flawed somehow? Probably, and I'd happy take feedback on > how/what I can do better, but this type of analysis is what I look for > to know if the process is working well or not. I'm shocked that these are the statistics you use to claim the current AUTOSEL process is working. I think they actually show quite the opposite! First, since many AUTOSEL commits aren't actually fixes but nearly all stable-tagged commits *are* fixes, the rate of regressions per commit would need to be lower for AUTOSEL commits than for stable-tagged commits in order for AUTOSEL commits to have the same rate of regressions *per fix*. Your numbers suggest a similar regression rate *per commit*. Thus, AUTOSEL probably introduces more regressions *per fix* than stable-tagged commits. Second, the way you're identifying regression-introducing commits seems to be excluding one of the most common, maybe *the* most common, cause of AUTOSEL regressions: missing prerequisite commits. A very common case that I've seen repeatedly is AUTOSEL picking just patch 2 or higher of a multi-patch series. For an example, see the patch that started this thread... If a missing prerequisite is backported later, my understanding is that it usually isn't given a Fixes tag, as the upstream commit didn't have it. I think such regressions aren't counted in your statistic, which only looks at Fixes tags. (Of course, stable-tagged commits sometimes have missing prerequisite bugs too. But it's expected to be at a lower rate, since the original developers and maintainers are directly involved in adding the stable tags. These are the people who are more familiar than anyone else with prerequisites.) Third, the category "commits without a stable tag" doesn't include just AUTOSEL commits, but also non-AUTOSEL commits that people asked to be added to stable because they fixed a problem for them. Such commits often have been in mainline for a long time, so naturally they're expected to have a lower regression rate than stable-tagged commits due to the longer soak time, on average. So if the regression rate of stable-tagged and non-stable-tagged commits is actually similar, that suggests the regression rate of non-stable-tagged commits is being brought up artifically by a high regression rate in AUTOSEL commits... So, I think your statistics actually reflect quite badly on AUTOSEL in its current form. By the way, to be clear, AUTOSEL is absolutely needed. The way you are doing it currently is not working well, though. I think it needs to be tuned to select fewer, higher-confidence fixes, and you need to do some basic checks against each one, like "does this commit have a pending fix" and "is this commit part of a multi-patch series, and if so are earlier patches needed as prerequisites". There also needs to be more soak time in mainline, and more review time. IMO you also need to take a hard look at whatever neural network thing you are using, as from what I've seen its results are quite poor... It does pick up some obvious fixes, but it seems they could have just as easily been found through some heuristics with grep. Beyond those obvious fixes, what it picks up seems to be barely distinguishable from a random selection. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 0:32 ` Eric Biggers @ 2023-02-28 1:53 ` Sasha Levin 2023-02-28 3:41 ` Eric Biggers 0 siblings, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-02-28 1:53 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-kernel, stable, viro, linux-fsdevel On Tue, Feb 28, 2023 at 12:32:20AM +0000, Eric Biggers wrote: >On Mon, Feb 27, 2023 at 05:35:30PM -0500, Sasha Levin wrote: >> > > Note, however, that it's not enough to keep pointing at a tiny set and >> > > using it to suggest that the entire process is broken. How many AUTOSEL >> > > commits introduced a regression? How many -stable tagged ones did? How >> > > many bugs did AUTOSEL commits fix? >> > >> > So basically you don't accept feedback from individual people, as individual >> > people don't have enough data? >> >> I'd love to improve the process, but for that we need to figure out >> criteria for what we consider good or bad, collect data, and make >> decisions based on that data. >> >> What I'm getting from this thread is a few anecdotal examples and >> statements that the process isn't working at all. >> >> I took Jon's stablefixes script which he used for his previous articles >> around stable kernel regressions (here: >> https://lwn.net/Articles/812231/) and tried running it on the 5.15 >> stable tree (just a random pick). I've proceeded with ignoring the >> non-user-visible regressions as Jon defined in his article (basically >> issues that were introduced and fixed in the same releases) and ended up >> with 604 commits that caused a user visible regression. >> >> Out of those 604 commits: >> >> - 170 had an explicit stable tag. >> - 434 did not have a stable tag. >> >> Looking at the commits in the 5.15 tree: >> >> With stable tag: >> >> $ git log --oneline -i --grep "cc.*stable" v5.15..stable/linux-5.15.y | wc -l >> 3676 >> >> Without stable tag (-96 commits which are version bumps): >> >> $ git log --oneline --invert-grep -i --grep "cc.*stable" v5.15..stable/linux-5.15.y | wc -l >> 10649 >> >> Regression rate for commits with stable tag: 170 / 3676 = 4.62% >> Regression rate for commits without a stable tag: 434 / 10553 = 4.11% >> >> Is the analysis flawed somehow? Probably, and I'd happy take feedback on >> how/what I can do better, but this type of analysis is what I look for >> to know if the process is working well or not. > >I'm shocked that these are the statistics you use to claim the current AUTOSEL >process is working. I think they actually show quite the opposite! > >First, since many AUTOSEL commits aren't actually fixes but nearly all >stable-tagged commits *are* fixes, the rate of regressions per commit would need >to be lower for AUTOSEL commits than for stable-tagged commits in order for >AUTOSEL commits to have the same rate of regressions *per fix*. Your numbers >suggest a similar regression rate *per commit*. Thus, AUTOSEL probably >introduces more regressions *per fix* than stable-tagged commits. Interesting claim. How many of the AUTOSEL commits are "actual" fixes? How do you know if a commit is a fix for anything or not? Could you try and back claims with some evidence? Yes, in a perfect world where we know if a commit is a fix we could avoid introducing regressions into the stable trees. Heck, maybe we could even stop writing buggy code to begin with? >Second, the way you're identifying regression-introducing commits seems to be >excluding one of the most common, maybe *the* most common, cause of AUTOSEL >regressions: missing prerequisite commits. A very common case that I've seen >repeatedly is AUTOSEL picking just patch 2 or higher of a multi-patch series. >For an example, see the patch that started this thread... If a missing >prerequisite is backported later, my understanding is that it usually isn't >given a Fixes tag, as the upstream commit didn't have it. I think such >regressions aren't counted in your statistic, which only looks at Fixes tags. It definitely happens, but we usually end up dropping the AUTOSEL-ed commit rather than bringing in complex dependency chains. Look at the stable-queue for a record of those. >(Of course, stable-tagged commits sometimes have missing prerequisite bugs too. >But it's expected to be at a lower rate, since the original developers and >maintainers are directly involved in adding the stable tags. These are the >people who are more familiar than anyone else with prerequisites.) You'd be surprised. There is documentation around how one would annotate dependencies for stable tagged commits, something along the lines of: cc: stable@kernel.org # dep1 dep2 Grep through the git log and see how often this is actually used. >Third, the category "commits without a stable tag" doesn't include just AUTOSEL >commits, but also non-AUTOSEL commits that people asked to be added to stable >because they fixed a problem for them. Such commits often have been in mainline >for a long time, so naturally they're expected to have a lower regression rate >than stable-tagged commits due to the longer soak time, on average. So if the >regression rate of stable-tagged and non-stable-tagged commits is actually >similar, that suggests the regression rate of non-stable-tagged commits is being >brought up artifically by a high regression rate in AUTOSEL commits... Yes, the numbers are pretty skewed up by different aspects of the process. >So, I think your statistics actually reflect quite badly on AUTOSEL in its >current form. > >By the way, to be clear, AUTOSEL is absolutely needed. The way you are doing it >currently is not working well, though. I think it needs to be tuned to select >fewer, higher-confidence fixes, and you need to do some basic checks against >each one, like "does this commit have a pending fix" and "is this commit part of Keep in mind that there's some lag time between when we do our thing vs when things appear upstream. Greg actually runs the "is there a pending fix" check even after I've pushed the patches, before he cuts releases. >a multi-patch series, and if so are earlier patches needed as prerequisites". >There also needs to be more soak time in mainline, and more review time. Tricky bit with mainline/review time is that very few of our users actually run -rc trees. We end up hitting many of the regressions because the commits actually end up in stable trees. Should it work that way? No, but our testing story around -rc releases is quite lacking. >IMO you also need to take a hard look at whatever neural network thing you are >using, as from what I've seen its results are quite poor... It does pick up >some obvious fixes, but it seems they could have just as easily been found >through some heuristics with grep. Beyond those obvious fixes, what it picks up >seems to be barely distinguishable from a random selection. I mean... patches welcome? Do you want to come up with a set of heuristics that performs better and give it a go? I'll happily switch over. I'm not sure how feedback in the form of "this sucks but I'm sure it could be much better" is useful. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 1:53 ` Sasha Levin @ 2023-02-28 3:41 ` Eric Biggers 2023-02-28 10:41 ` Amir Goldstein ` (2 more replies) 0 siblings, 3 replies; 83+ messages in thread From: Eric Biggers @ 2023-02-28 3:41 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 08:53:31PM -0500, Sasha Levin wrote: > > > > I'm shocked that these are the statistics you use to claim the current AUTOSEL > > process is working. I think they actually show quite the opposite! > > > > First, since many AUTOSEL commits aren't actually fixes but nearly all > > stable-tagged commits *are* fixes, the rate of regressions per commit would need > > to be lower for AUTOSEL commits than for stable-tagged commits in order for > > AUTOSEL commits to have the same rate of regressions *per fix*. Your numbers > > suggest a similar regression rate *per commit*. Thus, AUTOSEL probably > > introduces more regressions *per fix* than stable-tagged commits. > > Interesting claim. How many of the AUTOSEL commits are "actual" fixes? > How do you know if a commit is a fix for anything or not? > > Could you try and back claims with some evidence? > > Yes, in a perfect world where we know if a commit is a fix we could > avoid introducing regressions into the stable trees. Heck, maybe we could > even stop writing buggy code to begin with? Are you seriously trying to claim that a random commit your neural network picked up is just as likely to be a fix as a commit that the author explicitly tagged as a fix and/or for stable? That's quite an extraordinary claim, and it's not true from my experience. Lots of AUTOSEL patches that get Cc'ed to me, if I'm familiar enough with the area to understand fairly well whether the patch is a "fix", are not actually fixes. Or are very borderline "fixes" that don't meet stable criteria. (Note, I generally only bother responding to AUTOSEL if I think a patch is actually going to cause a problem. So a lack of response isn't necessarily agreement that a patch is really suitable for stable...) Oh sorry, personal experience is not "evidence". Please disregard my invalid non-evidence-based opinion. > > (Of course, stable-tagged commits sometimes have missing prerequisite bugs too. > > But it's expected to be at a lower rate, since the original developers and > > maintainers are directly involved in adding the stable tags. These are the > > people who are more familiar than anyone else with prerequisites.) > > You'd be surprised. There is documentation around how one would annotate > dependencies for stable tagged commits, something along the lines of: > > cc: stable@kernel.org # dep1 dep2 > > Grep through the git log and see how often this is actually used. Well, probably more common is that prerequisites are in the same patchset, and the prerequisites are tagged for stable too. Whereas AUTOSEL often just picks patch X of N. Also, developers and maintainers who tag patches for stable are probably more likely to help with the stable process in general and make sure patches are backported correctly... Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately cherry-picking patch X of N so often. > > a multi-patch series, and if so are earlier patches needed as prerequisites". > > There also needs to be more soak time in mainline, and more review time. > > Tricky bit with mainline/review time is that very few of our users > actually run -rc trees. > > We end up hitting many of the regressions because the commits actually > end up in stable trees. Should it work that way? No, but our testing > story around -rc releases is quite lacking. Well, in the bug that affected me, it *was* found on mainline almost immediately. It just took a bit longer than the extremely aggressive 7-day AUTOSEL period to be fixed. Oh sorry again, one example is not "evidence". Please disregard my invalid non-evidence-based opinion. > I'm not sure how feedback in the form of "this sucks but I'm sure it > could be much better" is useful. I've already given you some specific suggestions. I can't force you to listen to them, of course. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 3:41 ` Eric Biggers @ 2023-02-28 10:41 ` Amir Goldstein 2023-02-28 11:28 ` Greg KH 2023-02-28 17:03 ` Sasha Levin 2023-03-10 23:07 ` Eric Biggers 2 siblings, 1 reply; 83+ messages in thread From: Amir Goldstein @ 2023-02-28 10:41 UTC (permalink / raw) To: Eric Biggers; +Cc: Sasha Levin, linux-kernel, stable, viro, linux-fsdevel > > I'm not sure how feedback in the form of "this sucks but I'm sure it > > could be much better" is useful. > > I've already given you some specific suggestions. > > I can't force you to listen to them, of course. > Eric, As you probably know, this is not the first time that the subject of the AUTOSEL process has been discussed. Here is one example from fsdevel with a few other suggestions [1]. But just so you know, as a maintainer, you have the option to request that patches to your subsystem will not be selected by AUTOSEL and run your own process to select, test and submit fixes to stable trees. xfs maintainers have done that many years ago. This choice has consequences though - for years, no xfs fixes were flowing into stable trees at all, because no one was doing the backport work. It is hard to imagine that LTS kernel users were more happy about this situation than they would be from occasional regressions, but who knows... It has taken a long time until we found the resources and finally started a process of reviewing, testing and submitting xfs fixes to stable trees and this process involves a lot of resources (3 maintainers + $$$), so opting out of AUTOSEL is not a clear win. I will pencil down yet another discussion on fs and stable process at LSFMM23 to update on the current status with xfs, but it is hard to believe that this time we will be able to make significant changes to the AUTOSEL process. Thanks, Amir. [1] https://lore.kernel.org/linux-fsdevel/20201204160227.GA577125@mit.edu/#t ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 10:41 ` Amir Goldstein @ 2023-02-28 11:28 ` Greg KH 2023-03-01 2:05 ` Slade Watkins ` (2 more replies) 0 siblings, 3 replies; 83+ messages in thread From: Greg KH @ 2023-02-28 11:28 UTC (permalink / raw) To: Amir Goldstein Cc: Eric Biggers, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On Tue, Feb 28, 2023 at 12:41:07PM +0200, Amir Goldstein wrote: > > > I'm not sure how feedback in the form of "this sucks but I'm sure it > > > could be much better" is useful. > > > > I've already given you some specific suggestions. > > > > I can't force you to listen to them, of course. > > > > Eric, > > As you probably know, this is not the first time that the subject of the > AUTOSEL process has been discussed. > Here is one example from fsdevel with a few other suggestions [1]. > > But just so you know, as a maintainer, you have the option to request that > patches to your subsystem will not be selected by AUTOSEL and run your > own process to select, test and submit fixes to stable trees. Yes, and simply put, that's the answer for any subsystem or maintainer that does not want their patches picked using the AUTOSEL tool. The problem that the AUTOSEL tool is solving is real, we have whole major subsystems where no patches are ever marked as "for stable" and so real bugfixes are never backported properly. In an ideal world, all maintainers would properly mark their patches for stable backporting (as documented for the past 15+ years, with a cc: stable tag, NOT a Fixes: tag), but we do not live in that world, and hence, the need for the AUTOSEL work. thanks, greg k-h ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 11:28 ` Greg KH @ 2023-03-01 2:05 ` Slade Watkins 2023-03-01 5:13 ` Eric Biggers 2023-03-01 6:06 ` Greg KH 2023-03-01 10:31 ` Thorsten Leemhuis 2023-03-01 13:26 ` Mark Brown 2 siblings, 2 replies; 83+ messages in thread From: Slade Watkins @ 2023-03-01 2:05 UTC (permalink / raw) To: Greg KH, Sasha Levin Cc: Amir Goldstein, Eric Biggers, linux-kernel, stable, viro, linux-fsdevel On 2/28/23 06:28, Greg KH wrote: >> But just so you know, as a maintainer, you have the option to request that >> patches to your subsystem will not be selected by AUTOSEL and run your >> own process to select, test and submit fixes to stable trees. > > Yes, and simply put, that's the answer for any subsystem or maintainer > that does not want their patches picked using the AUTOSEL tool. > > The problem that the AUTOSEL tool is solving is real, we have whole > major subsystems where no patches are ever marked as "for stable" and so > real bugfixes are never backported properly. Yeah, I agree. And I'm throwing this out here [after having time to think about it due to an internet outage], but, would Cc'ing the patch's relevant subsystems on AUTOSEL emails help? This was sort of mentioned in this email[1] from Eric, and I think it _could_ help? I don't know, just something that crossed my mind earlier. > > In an ideal world, all maintainers would properly mark their patches for > stable backporting (as documented for the past 15+ years, with a cc: > stable tag, NOT a Fixes: tag), but we do not live in that world, and > hence, the need for the AUTOSEL work. (I wish we did... Oh well.) [1] https://lore.kernel.org/stable/Y%2Fzswi91axMN8OsA@sol.localdomain/ -- Slade ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 2:05 ` Slade Watkins @ 2023-03-01 5:13 ` Eric Biggers 2023-03-01 6:09 ` Greg KH 2023-03-01 6:06 ` Greg KH 1 sibling, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-01 5:13 UTC (permalink / raw) To: Slade Watkins Cc: Greg KH, Sasha Levin, Amir Goldstein, linux-kernel, stable, viro, linux-fsdevel On Tue, Feb 28, 2023 at 09:05:16PM -0500, Slade Watkins wrote: > On 2/28/23 06:28, Greg KH wrote: > >> But just so you know, as a maintainer, you have the option to request that > >> patches to your subsystem will not be selected by AUTOSEL and run your > >> own process to select, test and submit fixes to stable trees. > > > > Yes, and simply put, that's the answer for any subsystem or maintainer > > that does not want their patches picked using the AUTOSEL tool. > > > > The problem that the AUTOSEL tool is solving is real, we have whole > > major subsystems where no patches are ever marked as "for stable" and so > > real bugfixes are never backported properly. > > Yeah, I agree. > > And I'm throwing this out here [after having time to think about it due to an > internet outage], but, would Cc'ing the patch's relevant subsystems on AUTOSEL > emails help? This was sort of mentioned in this email[1] from Eric, and I > think it _could_ help? I don't know, just something that crossed my mind earlier. > AFAICT, that's already being done now, which is good. What I was talking about is that the subsystem lists aren't included on the *other* stable emails. Most importantly, the "FAILED: patch failed to apply to stable tree" emails. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 5:13 ` Eric Biggers @ 2023-03-01 6:09 ` Greg KH 2023-03-01 7:22 ` Eric Biggers 0 siblings, 1 reply; 83+ messages in thread From: Greg KH @ 2023-03-01 6:09 UTC (permalink / raw) To: Eric Biggers Cc: Slade Watkins, Sasha Levin, Amir Goldstein, linux-kernel, stable, viro, linux-fsdevel On Tue, Feb 28, 2023 at 09:13:56PM -0800, Eric Biggers wrote: > On Tue, Feb 28, 2023 at 09:05:16PM -0500, Slade Watkins wrote: > > On 2/28/23 06:28, Greg KH wrote: > > >> But just so you know, as a maintainer, you have the option to request that > > >> patches to your subsystem will not be selected by AUTOSEL and run your > > >> own process to select, test and submit fixes to stable trees. > > > > > > Yes, and simply put, that's the answer for any subsystem or maintainer > > > that does not want their patches picked using the AUTOSEL tool. > > > > > > The problem that the AUTOSEL tool is solving is real, we have whole > > > major subsystems where no patches are ever marked as "for stable" and so > > > real bugfixes are never backported properly. > > > > Yeah, I agree. > > > > And I'm throwing this out here [after having time to think about it due to an > > internet outage], but, would Cc'ing the patch's relevant subsystems on AUTOSEL > > emails help? This was sort of mentioned in this email[1] from Eric, and I > > think it _could_ help? I don't know, just something that crossed my mind earlier. > > > > AFAICT, that's already being done now, which is good. What I was talking about > is that the subsystem lists aren't included on the *other* stable emails. Most > importantly, the "FAILED: patch failed to apply to stable tree" emails. Why would the FAILED emails want to go to a mailing list? If the people that were part of making the patch don't want to respond to a FAILED email, why would anyone on the mailing list? But hey, I'll be glad to take a change to my script to add that functionality if you want to make it, it's here: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/scripts/bad_stable thanks, greg k-h ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 6:09 ` Greg KH @ 2023-03-01 7:22 ` Eric Biggers 2023-03-01 7:40 ` Willy Tarreau 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-01 7:22 UTC (permalink / raw) To: Greg KH Cc: Slade Watkins, Sasha Levin, Amir Goldstein, linux-kernel, stable, viro, linux-fsdevel On Wed, Mar 01, 2023 at 07:09:49AM +0100, Greg KH wrote: > > Why would the FAILED emails want to go to a mailing list? If the people > that were part of making the patch don't want to respond to a FAILED > email, why would anyone on the mailing list? The same reason we use mailing lists for kernel development at all instead of just sending patches directly to the MAINTAINERS. > > But hey, I'll be glad to take a change to my script to add that > functionality if you want to make it, it's here: > https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/scripts/bad_stable > Ah, the classic "patches welcome". Unfortunately, I don't think that can work very well for scripts that are only ever used by at most two people -- you and Sasha. Many even just by you, apparently, as I see your name, email address, home directory, etc. hardcoded in the scripts. (BTW, where are the scripts Sasha uses for AUTOSEL?) - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 7:22 ` Eric Biggers @ 2023-03-01 7:40 ` Willy Tarreau 2023-03-01 8:31 ` Eric Biggers 0 siblings, 1 reply; 83+ messages in thread From: Willy Tarreau @ 2023-03-01 7:40 UTC (permalink / raw) To: Eric Biggers Cc: Greg KH, Slade Watkins, Sasha Levin, Amir Goldstein, linux-kernel, stable, viro, linux-fsdevel On Tue, Feb 28, 2023 at 11:22:53PM -0800, Eric Biggers wrote: > On Wed, Mar 01, 2023 at 07:09:49AM +0100, Greg KH wrote: > > > > Why would the FAILED emails want to go to a mailing list? If the people > > that were part of making the patch don't want to respond to a FAILED > > email, why would anyone on the mailing list? > > The same reason we use mailing lists for kernel development at all instead of > just sending patches directly to the MAINTAINERS. I'm personally seeing a difference between reviewing patches on a mailing list to help someone polish it, and commenting on its suitability for stable once it's got merged, from someone not having participated to its inclusion. All such patches are already sent to stable@ which many of those of us interested in having a look are already subscribed to, and which most often just triggers a quick glance depending on areas of interest. I hardly see anyone just curious about a patch ask "are you sure ?". I could possibly understand the value of sending the failed ones to a list, in case someone with more time available than the author wants to give it a try. But again they're already sent to stable@. It's just that it might be possible that more interested parties are on other lists. But again I'm not fully convinced. > > But hey, I'll be glad to take a change to my script to add that > > functionality if you want to make it, it's here: > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/tree/scripts/bad_stable > > > > Ah, the classic "patches welcome". > > Unfortunately, I don't think that can work very well for scripts that are only > ever used by at most two people -- you and Sasha. Many even just by you, > apparently, as I see your name, email address, home directory, etc. hardcoded in > the scripts. But it's going into a dead end. You are the one saying that changes are easy, suggesting to use get_maintainers.pl, so easy that you can't try to adapt them in existing stuff. Even without modifying existing scripts, if you are really interested by such features, why not at least try to run your idea over a whole series, figure how long it takes, how accurate it seems to be, adjust the output to remove unwanted noise and propose for review a few lines that seem to do the job for you ? > (BTW, where are the scripts Sasha uses for AUTOSEL?) Maybe they're even uglier and he doesn't want to show them. The ones I was using to sort out 3.10 patches were totally ugly and full of heuristics, with lines that I would comment/uncomment along operations to refine the selection and I definitely wasn't going to post that anywhere. They were looking a bit like a commented extract of my .history. I'm sure over time Sasha has a much cleaner and repeatable process but maybe it involves parts that are not stabilized, that he's not proud of, or even tools contributed by coworkers that we don't necessarily need to know about and where he hardly sees how anyone could bring any help. Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 7:40 ` Willy Tarreau @ 2023-03-01 8:31 ` Eric Biggers 2023-03-01 8:43 ` Greg KH 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-01 8:31 UTC (permalink / raw) To: Willy Tarreau Cc: Greg KH, Slade Watkins, Sasha Levin, Amir Goldstein, linux-kernel, stable, viro, linux-fsdevel On Wed, Mar 01, 2023 at 08:40:03AM +0100, Willy Tarreau wrote: > But it's going into a dead end. You are the one saying that changes > are easy, suggesting to use get_maintainers.pl, so easy that you can't > try to adapt them in existing stuff. Even without modifying existing > scripts, if you are really interested by such features, why not at least > try to run your idea over a whole series, figure how long it takes, how > accurate it seems to be, adjust the output to remove unwanted noise and > propose for review a few lines that seem to do the job for you ? > As I said, Sasha *already does this for AUTOSEL*. So it seems this problem has already been solved, but Sasha and Greg are not coordinating with each other. Anyway, it's hard to have much motivation to try to contribute to scripts that not only can I not use or test myself, but before even getting to that point, pretty much any ideas for improvements are strongly pushed back on. Earlier I was actually going to go into more detail about some ideas for how to flag and review potential problems with backporting a commit, but I figured why bother given how this thread has been going. (Also I presume anything that would add *any* human time on the stable maintainers' end per patch, on average, would be strongly pushed back on too? But I have no visibility into what would be acceptable. I don't do their job.) - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 8:31 ` Eric Biggers @ 2023-03-01 8:43 ` Greg KH 0 siblings, 0 replies; 83+ messages in thread From: Greg KH @ 2023-03-01 8:43 UTC (permalink / raw) To: Eric Biggers Cc: Willy Tarreau, Slade Watkins, Sasha Levin, Amir Goldstein, linux-kernel, stable, viro, linux-fsdevel On Wed, Mar 01, 2023 at 12:31:22AM -0800, Eric Biggers wrote: > On Wed, Mar 01, 2023 at 08:40:03AM +0100, Willy Tarreau wrote: > > But it's going into a dead end. You are the one saying that changes > > are easy, suggesting to use get_maintainers.pl, so easy that you can't > > try to adapt them in existing stuff. Even without modifying existing > > scripts, if you are really interested by such features, why not at least > > try to run your idea over a whole series, figure how long it takes, how > > accurate it seems to be, adjust the output to remove unwanted noise and > > propose for review a few lines that seem to do the job for you ? > > > > As I said, Sasha *already does this for AUTOSEL*. So it seems this problem has > already been solved, but Sasha and Greg are not coordinating with each other. We do not share the same scripts for these tasks as we have different roles here. That's all, nothing malicious. thanks, greg k-h ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 2:05 ` Slade Watkins 2023-03-01 5:13 ` Eric Biggers @ 2023-03-01 6:06 ` Greg KH 2023-03-01 7:05 ` Eric Biggers 1 sibling, 1 reply; 83+ messages in thread From: Greg KH @ 2023-03-01 6:06 UTC (permalink / raw) To: Slade Watkins Cc: Sasha Levin, Amir Goldstein, Eric Biggers, linux-kernel, stable, viro, linux-fsdevel On Tue, Feb 28, 2023 at 09:05:16PM -0500, Slade Watkins wrote: > On 2/28/23 06:28, Greg KH wrote: > >> But just so you know, as a maintainer, you have the option to request that > >> patches to your subsystem will not be selected by AUTOSEL and run your > >> own process to select, test and submit fixes to stable trees. > > > > Yes, and simply put, that's the answer for any subsystem or maintainer > > that does not want their patches picked using the AUTOSEL tool. > > > > The problem that the AUTOSEL tool is solving is real, we have whole > > major subsystems where no patches are ever marked as "for stable" and so > > real bugfixes are never backported properly. > > Yeah, I agree. > > And I'm throwing this out here [after having time to think about it due to an > internet outage], but, would Cc'ing the patch's relevant subsystems on AUTOSEL > emails help? This was sort of mentioned in this email[1] from Eric, and I > think it _could_ help? I don't know, just something that crossed my mind earlier. I don't know, maybe? Note that determining a patch's "subsystem" at many times is difficult in an automated fashion, have any idea how to do that reliably that doesn't just hit lkml all the time? But again, how is that going to help much, the people who should be saying "no" are the ones on the signed-off-by and cc: lines in the patch itself. thanks, greg k-h ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-01 6:06 ` Greg KH @ 2023-03-01 7:05 ` Eric Biggers 0 siblings, 0 replies; 83+ messages in thread From: Eric Biggers @ 2023-03-01 7:05 UTC (permalink / raw) To: Greg KH Cc: Slade Watkins, Sasha Levin, Amir Goldstein, linux-kernel, stable, viro, linux-fsdevel On Wed, Mar 01, 2023 at 07:06:26AM +0100, Greg KH wrote: > On Tue, Feb 28, 2023 at 09:05:16PM -0500, Slade Watkins wrote: > > On 2/28/23 06:28, Greg KH wrote: > > >> But just so you know, as a maintainer, you have the option to request that > > >> patches to your subsystem will not be selected by AUTOSEL and run your > > >> own process to select, test and submit fixes to stable trees. > > > > > > Yes, and simply put, that's the answer for any subsystem or maintainer > > > that does not want their patches picked using the AUTOSEL tool. > > > > > > The problem that the AUTOSEL tool is solving is real, we have whole > > > major subsystems where no patches are ever marked as "for stable" and so > > > real bugfixes are never backported properly. > > > > Yeah, I agree. > > > > And I'm throwing this out here [after having time to think about it due to an > > internet outage], but, would Cc'ing the patch's relevant subsystems on AUTOSEL > > emails help? This was sort of mentioned in this email[1] from Eric, and I > > think it _could_ help? I don't know, just something that crossed my mind earlier. > > I don't know, maybe? Note that determining a patch's "subsystem" at > many times is difficult in an automated fashion, have any idea how to do > that reliably that doesn't just hit lkml all the time? As I said, it seems Sasha already does this for AUTOSEL (but not other stable emails). I assume he uses either get_maintainer.pl, or the lists the original patch is sent to (retrievable from lore). This is *not* a hard problem. > But again, how is that going to help much, the people who should be > saying "no" are the ones on the signed-off-by and cc: lines in the patch > itself. So that if one person does not respond, other people can help. You're basically arguing that mailing lists shouldn't exist at all... - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 11:28 ` Greg KH 2023-03-01 2:05 ` Slade Watkins @ 2023-03-01 10:31 ` Thorsten Leemhuis 2023-03-01 13:26 ` Mark Brown 2 siblings, 0 replies; 83+ messages in thread From: Thorsten Leemhuis @ 2023-03-01 10:31 UTC (permalink / raw) To: Greg KH, Amir Goldstein Cc: Eric Biggers, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On 28.02.23 12:28, Greg KH wrote: > On Tue, Feb 28, 2023 at 12:41:07PM +0200, Amir Goldstein wrote: >>>> I'm not sure how feedback in the form of "this sucks but I'm sure it >>>> could be much better" is useful. >>> I've already given you some specific suggestions. >>> I can't force you to listen to them, of course. >> >> As you probably know, this is not the first time that the subject of the >> AUTOSEL process has been discussed. >> Here is one example from fsdevel with a few other suggestions [1]. >> >> But just so you know, as a maintainer, you have the option to request that >> patches to your subsystem will not be selected by AUTOSEL and run your >> own process to select, test and submit fixes to stable trees. > [...] > In an ideal world, all maintainers would properly mark their patches for > stable backporting (as documented for the past 15+ years, with a cc: > stable tag, NOT a Fixes: tag), but we do not live in that world, and > hence, the need for the AUTOSEL work. Well, we could do something to get a bit closer to the ideal world: teach checkpatch.pl to help developers do the right thing in the first place. That's what I'm trying to do right now to make them add Link: tags more often (https://git.kernel.org/torvalds/c/d7f1d71e5ef6 ), as my regression tracking efforts heavily rely on them. Shouldn't be too hard to add a simple check along the lines of "this change has a Fixes: tag; either CC stable or do <foo> to suppress this warning" (<foo> could be a "nostable" tag or something else that we'd need to agree on first). In an ideal we'd maybe even have a "checkpatch bot" that looks at all patches posted and sends feedback to the list if it finds something to improve. Sure, some (a lot?) of what AUTOSEL does relies on data that is only available after a change was merged, but maybe some is available earlier already. Ciao, Thorsten ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 11:28 ` Greg KH 2023-03-01 2:05 ` Slade Watkins 2023-03-01 10:31 ` Thorsten Leemhuis @ 2023-03-01 13:26 ` Mark Brown 2 siblings, 0 replies; 83+ messages in thread From: Mark Brown @ 2023-03-01 13:26 UTC (permalink / raw) To: Greg KH Cc: Amir Goldstein, Eric Biggers, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel [-- Attachment #1: Type: text/plain, Size: 557 bytes --] On Tue, Feb 28, 2023 at 12:28:23PM +0100, Greg KH wrote: > In an ideal world, all maintainers would properly mark their patches for > stable backporting (as documented for the past 15+ years, with a cc: > stable tag, NOT a Fixes: tag), but we do not live in that world, and > hence, the need for the AUTOSEL work. Just as a datapoint here: I stopped making any effort to copy stable on things due to AUTOSEL, it's pulling back so much more than I would consider for stable that it no longer seems worthwhile to try to make decisions about what might fit. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 3:41 ` Eric Biggers 2023-02-28 10:41 ` Amir Goldstein @ 2023-02-28 17:03 ` Sasha Levin 2023-03-10 23:07 ` Eric Biggers 2 siblings, 0 replies; 83+ messages in thread From: Sasha Levin @ 2023-02-28 17:03 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote: >On Mon, Feb 27, 2023 at 08:53:31PM -0500, Sasha Levin wrote: >> > >> > I'm shocked that these are the statistics you use to claim the current AUTOSEL >> > process is working. I think they actually show quite the opposite! >> > >> > First, since many AUTOSEL commits aren't actually fixes but nearly all >> > stable-tagged commits *are* fixes, the rate of regressions per commit would need >> > to be lower for AUTOSEL commits than for stable-tagged commits in order for >> > AUTOSEL commits to have the same rate of regressions *per fix*. Your numbers >> > suggest a similar regression rate *per commit*. Thus, AUTOSEL probably >> > introduces more regressions *per fix* than stable-tagged commits. >> >> Interesting claim. How many of the AUTOSEL commits are "actual" fixes? >> How do you know if a commit is a fix for anything or not? >> >> Could you try and back claims with some evidence? >> >> Yes, in a perfect world where we know if a commit is a fix we could >> avoid introducing regressions into the stable trees. Heck, maybe we could >> even stop writing buggy code to begin with? > >Are you seriously trying to claim that a random commit your neural network >picked up is just as likely to be a fix as a commit that the author explicitly >tagged as a fix and/or for stable? I'd like to think that this is the case after the initial selection and multiple rounds of reviews, yes. >That's quite an extraordinary claim, and it's not true from my experience. Lots >of AUTOSEL patches that get Cc'ed to me, if I'm familiar enough with the area to >understand fairly well whether the patch is a "fix", are not actually fixes. Or >are very borderline "fixes" that don't meet stable criteria. (Note, I generally >only bother responding to AUTOSEL if I think a patch is actually going to cause >a problem. So a lack of response isn't necessarily agreement that a patch is >really suitable for stable...) > >Oh sorry, personal experience is not "evidence". Please disregard my invalid >non-evidence-based opinion. > >> > (Of course, stable-tagged commits sometimes have missing prerequisite bugs too. >> > But it's expected to be at a lower rate, since the original developers and >> > maintainers are directly involved in adding the stable tags. These are the >> > people who are more familiar than anyone else with prerequisites.) >> >> You'd be surprised. There is documentation around how one would annotate >> dependencies for stable tagged commits, something along the lines of: >> >> cc: stable@kernel.org # dep1 dep2 >> >> Grep through the git log and see how often this is actually used. > >Well, probably more common is that prerequisites are in the same patchset, and >the prerequisites are tagged for stable too. Whereas AUTOSEL often just picks >patch X of N. Also, developers and maintainers who tag patches for stable are >probably more likely to help with the stable process in general and make sure >patches are backported correctly... > >Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately >cherry-picking patch X of N so often. That's a fair point. >> > a multi-patch series, and if so are earlier patches needed as prerequisites". >> > There also needs to be more soak time in mainline, and more review time. >> >> Tricky bit with mainline/review time is that very few of our users >> actually run -rc trees. >> >> We end up hitting many of the regressions because the commits actually >> end up in stable trees. Should it work that way? No, but our testing >> story around -rc releases is quite lacking. > >Well, in the bug that affected me, it *was* found on mainline almost >immediately. It just took a bit longer than the extremely aggressive 7-day >AUTOSEL period to be fixed. > >Oh sorry again, one example is not "evidence". Please disregard my invalid >non-evidence-based opinion. I'm happy that we're in agreement that significant process changes can't happen because of opinions or anecdotal examples. In all seriousness, I will work on addressing the issues that happened around the commit(s) you've pointed out and improve our existing process. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-28 3:41 ` Eric Biggers 2023-02-28 10:41 ` Amir Goldstein 2023-02-28 17:03 ` Sasha Levin @ 2023-03-10 23:07 ` Eric Biggers 2023-03-11 13:41 ` Sasha Levin 2 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-10 23:07 UTC (permalink / raw) To: Sasha Levin; +Cc: linux-kernel, stable, viro, linux-fsdevel On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote: > > Well, probably more common is that prerequisites are in the same patchset, and > the prerequisites are tagged for stable too. Whereas AUTOSEL often just picks > patch X of N. Also, developers and maintainers who tag patches for stable are > probably more likely to help with the stable process in general and make sure > patches are backported correctly... > > Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately > cherry-picking patch X of N so often. > ... and AUTOSEL strikes again, with the 6.1 and 6.2 kernels currently crashing whenever a block device is removed, due to patches 1 and 3 of a 3-patch series being AUTOSEL'ed (on the same day I started this discussion, no less): https://lore.kernel.org/linux-block/CAOCAAm4reGhz400DSVrh0BetYD3Ljr2CZen7_3D4gXYYdB4SKQ@mail.gmail.com/T/#u Oh sorry, ignore this, it's just an anecdotal example. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-10 23:07 ` Eric Biggers @ 2023-03-11 13:41 ` Sasha Levin 2023-03-11 15:54 ` James Bottomley 0 siblings, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-03-11 13:41 UTC (permalink / raw) To: Eric Biggers; +Cc: linux-kernel, stable, viro, linux-fsdevel On Fri, Mar 10, 2023 at 03:07:04PM -0800, Eric Biggers wrote: >On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote: >> >> Well, probably more common is that prerequisites are in the same patchset, and >> the prerequisites are tagged for stable too. Whereas AUTOSEL often just picks >> patch X of N. Also, developers and maintainers who tag patches for stable are >> probably more likely to help with the stable process in general and make sure >> patches are backported correctly... >> >> Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately >> cherry-picking patch X of N so often. >> > >... and AUTOSEL strikes again, with the 6.1 and 6.2 kernels currently crashing >whenever a block device is removed, due to patches 1 and 3 of a 3-patch series >being AUTOSEL'ed (on the same day I started this discussion, no less): > >https://lore.kernel.org/linux-block/CAOCAAm4reGhz400DSVrh0BetYD3Ljr2CZen7_3D4gXYYdB4SKQ@mail.gmail.com/T/#u > >Oh sorry, ignore this, it's just an anecdotal example. Yes, clearly a problem with AUTOSEL and not with how sad the testing story is for stable releases. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 13:41 ` Sasha Levin @ 2023-03-11 15:54 ` James Bottomley 2023-03-11 18:07 ` Sasha Levin 0 siblings, 1 reply; 83+ messages in thread From: James Bottomley @ 2023-03-11 15:54 UTC (permalink / raw) To: Sasha Levin, Eric Biggers; +Cc: linux-kernel, stable, viro, linux-fsdevel On Sat, 2023-03-11 at 08:41 -0500, Sasha Levin wrote: > On Fri, Mar 10, 2023 at 03:07:04PM -0800, Eric Biggers wrote: > > On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote: > > > > > > Well, probably more common is that prerequisites are in the same > > > patchset, and the prerequisites are tagged for stable too. > > > Whereas AUTOSEL often just picks patch X of N. Also, developers > > > and maintainers who tag patches for stable are probably more > > > likely to help with the stable process in general and make sure > > > patches are backported correctly... > > > > > > Anyway, the point is, AUTOSEL needs to be fixed to stop > > > inappropriately cherry-picking patch X of N so often. > > > > > > > ... and AUTOSEL strikes again, with the 6.1 and 6.2 kernels > > currently crashing whenever a block device is removed, due to > > patches 1 and 3 of a 3-patch series being AUTOSEL'ed (on the same > > day I started this discussion, no less): > > > > https://lore.kernel.org/linux-block/CAOCAAm4reGhz400DSVrh0BetYD3Ljr2CZen7_3D4gXYYdB4SKQ@mail.gmail.com/T/#u > > > > Oh sorry, ignore this, it's just an anecdotal example. > > Yes, clearly a problem with AUTOSEL and not with how sad the testing > story is for stable releases. Hey, that's a completely circular argument: if we had perfect testing then, of course, it would pick up every bad patch before anything got released; but we don't, and everyone knows it. Therefore, we have to be discriminating about what patches we put in. And we have to acknowledge that zero bugs in patches isn't feasible in spite of all the checking we do do. I also think we have to acknowledge that users play a role in the testing process because some bugs simply aren't picked up until they try out a release. So discouraging users from running mainline -rc's means we do get bugs in the released kernel that we might not have had if they did. Likewise if everyone only runs stable kernels, the bugs in the released kernel don't get found until stable. So this blame game really isn't helping. I think the one thing everyone on this thread might agree on is that this bug wouldn't have happened if AUTOSEL could detect and backport series instead of individual patches. Sasha says that can't be done based on in information in Linus' tree[1] which is true but not a correct statement of the problem. The correct question is given all the information available, including lore, could we assist AUTOSEL in better detecting series and possibly making better decisions generally? I think that's the challenge for anyone who actually wants to help rather than complain. At least the series detection bit sounds like it could be a reasonable summer of code project. Regards, James [1] https://lore.kernel.org/linux-fsdevel/ZAyK0KM6JmVOvQWy@sashalap/ ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 15:54 ` James Bottomley @ 2023-03-11 18:07 ` Sasha Levin 2023-03-12 19:03 ` Theodore Ts'o 0 siblings, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-03-11 18:07 UTC (permalink / raw) To: James Bottomley; +Cc: Eric Biggers, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 10:54:36AM -0500, James Bottomley wrote: >On Sat, 2023-03-11 at 08:41 -0500, Sasha Levin wrote: >> On Fri, Mar 10, 2023 at 03:07:04PM -0800, Eric Biggers wrote: >> > On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote: >> > > >> > > Well, probably more common is that prerequisites are in the same >> > > patchset, and the prerequisites are tagged for stable too. >> > > Whereas AUTOSEL often just picks patch X of N. Also, developers >> > > and maintainers who tag patches for stable are probably more >> > > likely to help with the stable process in general and make sure >> > > patches are backported correctly... >> > > >> > > Anyway, the point is, AUTOSEL needs to be fixed to stop >> > > inappropriately cherry-picking patch X of N so often. >> > > >> > >> > ... and AUTOSEL strikes again, with the 6.1 and 6.2 kernels >> > currently crashing whenever a block device is removed, due to >> > patches 1 and 3 of a 3-patch series being AUTOSEL'ed (on the same >> > day I started this discussion, no less): >> > >> > https://lore.kernel.org/linux-block/CAOCAAm4reGhz400DSVrh0BetYD3Ljr2CZen7_3D4gXYYdB4SKQ@mail.gmail.com/T/#u >> > >> > Oh sorry, ignore this, it's just an anecdotal example. >> >> Yes, clearly a problem with AUTOSEL and not with how sad the testing >> story is for stable releases. > >Hey, that's a completely circular argument: if we had perfect testing >then, of course, it would pick up every bad patch before anything got >released; but we don't, and everyone knows it. Therefore, we have to >be discriminating about what patches we put in. And we have to >acknowledge that zero bugs in patches isn't feasible in spite of all >the checking we do do. I also think we have to acknowledge that users >play a role in the testing process because some bugs simply aren't >picked up until they try out a release. So discouraging users from >running mainline -rc's means we do get bugs in the released kernel that >we might not have had if they did. Likewise if everyone only runs >stable kernels, the bugs in the released kernel don't get found until >stable. So this blame game really isn't helping. > >I think the one thing everyone on this thread might agree on is that >this bug wouldn't have happened if AUTOSEL could detect and backport >series instead of individual patches. Sasha says that can't be done >based on in information in Linus' tree[1] which is true but not a >correct statement of the problem. The correct question is given all >the information available, including lore, could we assist AUTOSEL in >better detecting series and possibly making better decisions generally? My argument was that this type of issue is no AUTOSEL specific, and we saw it happening multiple times with stable tagged patches as well. It's something that needs to get solved, and I suspect that both Greg and myself will end up using it when it's there. >I think that's the challenge for anyone who actually wants to help >rather than complain. At least the series detection bit sounds like it >could be a reasonable summer of code project. Right - I was trying to reply directly to Willy's question: this is something very useful, somewhat hard, and I don't think I could do in the near future - so help is welcome here. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 18:07 ` Sasha Levin @ 2023-03-12 19:03 ` Theodore Ts'o 0 siblings, 0 replies; 83+ messages in thread From: Theodore Ts'o @ 2023-03-12 19:03 UTC (permalink / raw) To: Sasha Levin Cc: James Bottomley, Eric Biggers, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 01:07:09PM -0500, Sasha Levin wrote: > > I think the one thing everyone on this thread might agree on is that > > this bug wouldn't have happened if AUTOSEL could detect and backport > > series instead of individual patches. Sasha says that can't be done > > based on in information in Linus' tree[1] which is true but not a > > correct statement of the problem. The correct question is given all > > the information available, including lore, could we assist AUTOSEL in > > better detecting series and possibly making better decisions generally? > > My argument was that this type of issue is no AUTOSEL specific, and we > saw it happening multiple times with stable tagged patches as well. I suspect that it happens *less* with Greg's patches, since the people who add (and it's almost always remove) the Cc: tags have the dependency information fresh in their brains' caches, and are more likely to correctly tag which patches should and shouldn't have Cc: stable tags. Now, if after I've carefully annotated a patch series, some with Cc: stable tags, and some without, and AUTOSEL were to override my work and take some extra patches, that I had deliberately not tagged with Cc: stable, I can (and have) gotten mildly annoyed, but in general, it means that something which probably shouldn't and didn't need to go into an LTS kernel ended up going into an LTS kernel, and TBH, when this has happened, I don't worry about it *too* much. It probably has made LTS less sstable when this happens, but it's generally not a disaster. In any case, I think the problem of missing dependencies when they are in a patch series is *primarily* an AUTOSEL problem, and as I said, probably can be fixed by using the information in lore. I'll note that we can probably make the "was this part of the patch series" work even better by encouraging contributors not to take unrelated patches and put them unnecessarily into a patch series. So if I knew that the stable bots were taking patch series into account, or were about to start taking advantage of this signal, I'd be happy to push back on contributors to break up patch series that should be glommed together. - Ted ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 18:06 ` Eric Biggers 2023-02-27 20:39 ` Sasha Levin @ 2023-03-07 21:18 ` Pavel Machek 2023-03-07 21:45 ` Eric Biggers 1 sibling, 1 reply; 83+ messages in thread From: Pavel Machek @ 2023-03-07 21:18 UTC (permalink / raw) To: Eric Biggers; +Cc: Sasha Levin, linux-kernel, stable, viro, linux-fsdevel [-- Attachment #1: Type: text/plain, Size: 1319 bytes --] Hi! > > So to summarize, that buggy commit was backported even though: > > > > * There were no indications that it was a bug fix (and thus potentially > > suitable for stable) in the first place. > > * On the AUTOSEL thread, someone told you the commit is broken. > > * There was already a thread that reported a regression caused by the commit. > > Easily findable via lore search. > > * There was also already a pending patch that Fixes the commit. Again easily > > findable via lore search. > > > > So it seems a *lot* of things went wrong, no? Why? If so many things can go > > wrong, it's not just a "mistake" but rather the process is the problem... > > BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after > only being in mainline for 4 days, and *released* in all LTS kernels after only > being in mainline for 12 days. Surely that's a timeline befitting a critical > security vulnerability, not some random neural-network-selected commit that > wasn't even fixing anything? I see this problem, too, "-stable" is more experimental than Linus's releases. I believe that -stable would be more useful without AUTOSEL process. Best regards, Pavel -- People of Russia, stop Putin before his war on Ukraine escalates. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-07 21:18 ` Pavel Machek @ 2023-03-07 21:45 ` Eric Biggers 2023-03-11 6:25 ` Matthew Wilcox 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-07 21:45 UTC (permalink / raw) To: Pavel Machek; +Cc: Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On Tue, Mar 07, 2023 at 10:18:35PM +0100, Pavel Machek wrote: > Hi! > > > > So to summarize, that buggy commit was backported even though: > > > > > > * There were no indications that it was a bug fix (and thus potentially > > > suitable for stable) in the first place. > > > * On the AUTOSEL thread, someone told you the commit is broken. > > > * There was already a thread that reported a regression caused by the commit. > > > Easily findable via lore search. > > > * There was also already a pending patch that Fixes the commit. Again easily > > > findable via lore search. > > > > > > So it seems a *lot* of things went wrong, no? Why? If so many things can go > > > wrong, it's not just a "mistake" but rather the process is the problem... > > > > BTW, another cause of this is that the commit (66f99628eb24) was AUTOSEL'd after > > only being in mainline for 4 days, and *released* in all LTS kernels after only > > being in mainline for 12 days. Surely that's a timeline befitting a critical > > security vulnerability, not some random neural-network-selected commit that > > wasn't even fixing anything? > > I see this problem, too, "-stable" is more experimental than Linus's > releases. > > I believe that -stable would be more useful without AUTOSEL process. > There has to be a way to ensure that security fixes that weren't properly tagged make it to stable anyway. So, AUTOSEL is necessary, at least in some form. I think that debating *whether it should exist* is a distraction from what's actually important, which is that the current AUTOSEL process has some specific problems, and these specific problems need to be fixed... - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-07 21:45 ` Eric Biggers @ 2023-03-11 6:25 ` Matthew Wilcox 2023-03-11 8:11 ` Willy Tarreau 2023-03-11 14:06 ` Sasha Levin 0 siblings, 2 replies; 83+ messages in thread From: Matthew Wilcox @ 2023-03-11 6:25 UTC (permalink / raw) To: Eric Biggers Cc: Pavel Machek, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On Tue, Mar 07, 2023 at 09:45:24PM +0000, Eric Biggers wrote: > On Tue, Mar 07, 2023 at 10:18:35PM +0100, Pavel Machek wrote: > > I believe that -stable would be more useful without AUTOSEL process. > > There has to be a way to ensure that security fixes that weren't properly tagged > make it to stable anyway. So, AUTOSEL is necessary, at least in some form. I > think that debating *whether it should exist* is a distraction from what's > actually important, which is that the current AUTOSEL process has some specific > problems, and these specific problems need to be fixed... I agree with you, that we need autosel and we also need autosel to be better. I actually see Pavel's mail as a datapoint (or "anecdote", if you will) in support of that; the autosel process currently works so badly that a long-time contributor thinks it's worse than nothing. Sasha, what do you need to help you make this better? ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 6:25 ` Matthew Wilcox @ 2023-03-11 8:11 ` Willy Tarreau 2023-03-11 11:45 ` Pavel Machek 2023-03-11 14:06 ` Sasha Levin 1 sibling, 1 reply; 83+ messages in thread From: Willy Tarreau @ 2023-03-11 8:11 UTC (permalink / raw) To: Matthew Wilcox Cc: Eric Biggers, Pavel Machek, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 06:25:59AM +0000, Matthew Wilcox wrote: > On Tue, Mar 07, 2023 at 09:45:24PM +0000, Eric Biggers wrote: > > On Tue, Mar 07, 2023 at 10:18:35PM +0100, Pavel Machek wrote: > > > I believe that -stable would be more useful without AUTOSEL process. > > > > There has to be a way to ensure that security fixes that weren't properly tagged > > make it to stable anyway. So, AUTOSEL is necessary, at least in some form. I > > think that debating *whether it should exist* is a distraction from what's > > actually important, which is that the current AUTOSEL process has some specific > > problems, and these specific problems need to be fixed... > > I agree with you, that we need autosel and we also need autosel to > be better. I actually see Pavel's mail as a datapoint (or "anecdote", > if you will) in support of that; the autosel process currently works > so badly that a long-time contributor thinks it's worse than nothing. > > Sasha, what do you need to help you make this better? One would probably need to define "better" and "so badly". As a user of -stable kernels, I consider that they've got much better over the last years. A lot of processes have improved everywhere even before the release, but I do think that autosel is part of what generally gives a chance to some useful and desired fixed (e.g. in drivers) to be backported and save some users unneeded headaches. In fact I think that the reason for the negative perception is that patches that it picks are visible, and it's easy to think "WTF" when seeing one of them. Previously, these patches were not proposed, so nobody knew they were missing. It happened to plenty of us to spend some time trying to spot why a stable kernel would occasionally fail on a machine, and discovering in the process that mainline did work because it contained a fix that was never backported. This is frustrating but there's noone to blame for failing to pick that patch (and the patch's author should not be blamed either since for small compatibility stuff it's probably common to see first-timers who are not yet at ease with the process). Here the patches are CCed to their authors before being merged. They get a chance to be reviewed and rejected. Granted, maybe sometimes they could be subject to a longer delay or be sent to certain lists. Maybe. But I do think that the complaints in fact reflect a process that's not as broken as some think, precisely because it allows people to complain when something is going wrong. The previous process didn't permit that. For this alone it's a progress. Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 8:11 ` Willy Tarreau @ 2023-03-11 11:45 ` Pavel Machek 2023-03-11 12:29 ` Greg KH 0 siblings, 1 reply; 83+ messages in thread From: Pavel Machek @ 2023-03-11 11:45 UTC (permalink / raw) To: Willy Tarreau Cc: Matthew Wilcox, Eric Biggers, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel [-- Attachment #1: Type: text/plain, Size: 1570 bytes --] Hi! > > > > I believe that -stable would be more useful without AUTOSEL process. > > > > > > There has to be a way to ensure that security fixes that weren't properly tagged > > > make it to stable anyway. So, AUTOSEL is necessary, at least in some form. I > > > think that debating *whether it should exist* is a distraction from what's > > > actually important, which is that the current AUTOSEL process has some specific > > > problems, and these specific problems need to be fixed... > > > > I agree with you, that we need autosel and we also need autosel to > > be better. I actually see Pavel's mail as a datapoint (or "anecdote", > > if you will) in support of that; the autosel process currently works > > so badly that a long-time contributor thinks it's worse than nothing. > > > > Sasha, what do you need to help you make this better? > > One would probably need to define "better" and "so badly". As a user > of -stable kernels, I consider that they've got much better over the Well, we have Documentation/process/stable-kernel-rules.rst . If we wanted to define "better", we should start documenting what the real rules are for the patches in the stable tree. I agree that -stable works quite well, but the real rules are far away from what is documented. I don't think AUTOSEL works well. I believe we should require positive reply from patch author on relevant maintainer before merging such patch to -stable. Best regards, Pavel -- People of Russia, stop Putin before his war on Ukraine escalates. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 11:45 ` Pavel Machek @ 2023-03-11 12:29 ` Greg KH 2023-03-21 12:41 ` Maciej W. Rozycki 0 siblings, 1 reply; 83+ messages in thread From: Greg KH @ 2023-03-11 12:29 UTC (permalink / raw) To: Pavel Machek Cc: Willy Tarreau, Matthew Wilcox, Eric Biggers, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 12:45:20PM +0100, Pavel Machek wrote: > Hi! > > > > > > I believe that -stable would be more useful without AUTOSEL process. > > > > > > > > There has to be a way to ensure that security fixes that weren't properly tagged > > > > make it to stable anyway. So, AUTOSEL is necessary, at least in some form. I > > > > think that debating *whether it should exist* is a distraction from what's > > > > actually important, which is that the current AUTOSEL process has some specific > > > > problems, and these specific problems need to be fixed... > > > > > > I agree with you, that we need autosel and we also need autosel to > > > be better. I actually see Pavel's mail as a datapoint (or "anecdote", > > > if you will) in support of that; the autosel process currently works > > > so badly that a long-time contributor thinks it's worse than nothing. > > > > > > Sasha, what do you need to help you make this better? > > > > One would probably need to define "better" and "so badly". As a user > > of -stable kernels, I consider that they've got much better over the > > Well, we have Documentation/process/stable-kernel-rules.rst . If we > wanted to define "better", we should start documenting what the real > rules are for the patches in the stable tree. > > I agree that -stable works quite well, but the real rules are far away > from what is documented. > > I don't think AUTOSEL works well. I believe we should require positive > reply from patch author on relevant maintainer before merging such > patch to -stable. Again, for the people in the back so that everyone can hear it, that does not work as some subsystems refuse to tag ANY patches for stable commits, nor do they want to have anything to do with stable kernels at all. And that's fine, that's their option, but because of that, we have to have a way to actually get the real fixes in those subsystems to the users who use these stable kernels. Hence, the AUTOSEL work. So no, forcing a maintainer/author to ack a patch to get it into stable is not going to work UNLESS a maintainer/author explicitly asks for that, which some have, and that's wonderful. thanks, greg k-h ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 12:29 ` Greg KH @ 2023-03-21 12:41 ` Maciej W. Rozycki 0 siblings, 0 replies; 83+ messages in thread From: Maciej W. Rozycki @ 2023-03-21 12:41 UTC (permalink / raw) To: Greg KH Cc: Pavel Machek, Willy Tarreau, Matthew Wilcox, Eric Biggers, Sasha Levin, linux-kernel, stable, viro, linux-fsdevel On Sat, 11 Mar 2023, Greg KH wrote: > So no, forcing a maintainer/author to ack a patch to get it into stable > is not going to work UNLESS a maintainer/author explicitly asks for > that, which some have, and that's wonderful. FWIW I'm happy to ack patches for stable backporting (and I tag patches of mine for stable as I deem appropriate). Maciej ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 6:25 ` Matthew Wilcox 2023-03-11 8:11 ` Willy Tarreau @ 2023-03-11 14:06 ` Sasha Levin 2023-03-11 16:16 ` Theodore Ts'o 2023-03-11 22:38 ` David Laight 1 sibling, 2 replies; 83+ messages in thread From: Sasha Levin @ 2023-03-11 14:06 UTC (permalink / raw) To: Matthew Wilcox Cc: Eric Biggers, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 06:25:59AM +0000, Matthew Wilcox wrote: >On Tue, Mar 07, 2023 at 09:45:24PM +0000, Eric Biggers wrote: >> On Tue, Mar 07, 2023 at 10:18:35PM +0100, Pavel Machek wrote: >> > I believe that -stable would be more useful without AUTOSEL process. >> >> There has to be a way to ensure that security fixes that weren't properly tagged >> make it to stable anyway. So, AUTOSEL is necessary, at least in some form. I >> think that debating *whether it should exist* is a distraction from what's >> actually important, which is that the current AUTOSEL process has some specific >> problems, and these specific problems need to be fixed... > >I agree with you, that we need autosel and we also need autosel to >be better. I actually see Pavel's mail as a datapoint (or "anecdote", >if you will) in support of that; the autosel process currently works >so badly that a long-time contributor thinks it's worse than nothing. > >Sasha, what do you need to help you make this better? What could I do to avoid this? I suppose that if I had a way to know if a certain a commit is part of a series, I could either take all of it or none of it, but I don't think I have a way of doing that by looking at a commit in Linus' tree (suggestions welcome, I'm happy to implement them). Other than that, the commit at hand: 1. Describes a real problem that needs to be fixed, so while it was reverted for a quick fix, we'll need to go back and bring it in along with it's dependency. 2. Soaked for over two weeks between the AUTOSEL mails and the release, gone through multiple rounds of reviews. 3. Went through all the tests provided by all the individuals, bots, companies, etc who test the tree through multiple rounds of testing (we had to do a -rc2 for that releases). 4. Went through whatever tests distros run on the kernel before they package and release it. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 14:06 ` Sasha Levin @ 2023-03-11 16:16 ` Theodore Ts'o 2023-03-11 17:48 ` Eric Biggers 2023-03-11 22:38 ` David Laight 1 sibling, 1 reply; 83+ messages in thread From: Theodore Ts'o @ 2023-03-11 16:16 UTC (permalink / raw) To: Sasha Levin Cc: Matthew Wilcox, Eric Biggers, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 09:06:08AM -0500, Sasha Levin wrote: > > I suppose that if I had a way to know if a certain a commit is part of a > series, I could either take all of it or none of it, but I don't think I > have a way of doing that by looking at a commit in Linus' tree > (suggestions welcome, I'm happy to implement them). Well, this is why I think it is a good idea to have a link to the patch series in lore. I know Linus doesn't like it, claiming it doesn't add any value, but I have to disagree. It adds two bits of value. First, if there is any discussion on the review of the patch before it goes in, the lore link gives you access to that --- and if people have a back-lick in the cover letter of version N to the cover letter of version N-1, it allows someone doing code archeology to find all of the discussions around the patch series in the lore archives. Secondly, the lore link will allow you to figure out whether or not the patch is part of a series; b4 can figure this out by looking at the in-reply-to headers, and lore will chain the patch series together, so if the commit contains a lore link to the patch, the AUTOSEL script could use that to find out whether the patch is part of the series. And this is really easy to do. All you need is the following in .git/hooks/applypatch-msg: #!/bin/sh # For .git/hooks/applypatch-msg # . git-sh-setup perl -pi -e 's|^Message-Id:\s*<?([^>]+)>?$|Link: https://lore.kernel.org/r/$1|g;' "$1" test -x "$GIT_DIR/hooks/commit-msg" && exec "$GIT_DIR/hooks/commit-msg" ${1+"$@"} : Cheers, - Ted P.S. There was a recent patch series where I noticed that I would be screwed if AUTOSEL would only take patch 2/2 and not patch 1/2. I dealt with that though by adding an explicit "Cc: stable@kernel.org". So that's the other way to avoid breakage; if people were universally careful about adding "Cc: stable@kernel.org" tags, then we wouldn't need AUTOSEL at all. And this is another place where I break with commonly received wisdom about "Thou Shalt Never, Never Rewind The Git Branch". Personally, if I find that I missed a Cc: stable tag, rewinding the branch to add edit the trailers is *far* better a tradeoff than adhering to some religious rule about never rewinding git branches. Of course, I can get away with that since I don't have people basing their branches on my branch. But I've seen people who will self-righteously proclaim non-rewinding git branches as the One True Way to do git, and I profoundly disagree with that point of view. ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 16:16 ` Theodore Ts'o @ 2023-03-11 17:48 ` Eric Biggers 2023-03-11 18:26 ` Sasha Levin 2023-03-11 18:33 ` Willy Tarreau 0 siblings, 2 replies; 83+ messages in thread From: Eric Biggers @ 2023-03-11 17:48 UTC (permalink / raw) To: Theodore Ts'o Cc: Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 11:16:44AM -0500, Theodore Ts'o wrote: > On Sat, Mar 11, 2023 at 09:06:08AM -0500, Sasha Levin wrote: > > > > I suppose that if I had a way to know if a certain a commit is part of a > > series, I could either take all of it or none of it, but I don't think I > > have a way of doing that by looking at a commit in Linus' tree > > (suggestions welcome, I'm happy to implement them). > > Well, this is why I think it is a good idea to have a link to the > patch series in lore. I know Linus doesn't like it, claiming it > doesn't add any value, but I have to disagree. It adds two bits of > value. > So, earlier I was going to go into more detail about some of my ideas, before Sasha and Greg started stonewalling with "patches welcome" (i.e. "I'm refusing to do my job") and various silly arguments about why nothing should be changed. But I suppose the worst thing that can happen is that that just continues, so here it goes: One of the first things I would do if I was maintaining the stable kernels is to set up a way to automatically run searches on the mailing lists, and then take advantage of that in the stable process in various ways. Not having that is the root cause of a lot of the issues with the current process, IMO. Now that lore exists, this might be trivial: it could be done just by hammering lore.kernel.org with queries https://lore.kernel.org/linux-fsdevel/?q=query from a Python script. Of course, there's a chance that won't scale to multiple queries for each one of thousands of stable commits, or at least won't be friendly to the kernel.org admins. In that case, what can be done is to download down all emails from all lists, using lore's git mirrors or Atom feeds, and index them locally. (Note: if the complete history is inconveniently large, then just indexing the last year or so would work nearly as well.) Then once that is in place, that could be used in various ways. For example, given a git commit, it's possible to search by email subject to get to the original patch, *even if the git commit does not have a Link tag*. And it can be automatically checked whether it's part of a patch series, and if so, whether all the patches in the series are being backported or just some. This could also be used to check for mentions of a commit on the mailing list that potentially indicate a regression report, which is one of the issues we discussed earlier. I'm not sure what the optimal search criteria would be, but one option would be something like "messages that contain the commit title or commit ID and are dated to after the commit being committed". There might need to be some exclusions added to that. This could also be used to automatically find the AUTOSEL email, if one exists, and check whether it's been replied to or not. The purpose of all these mailing list searches would be to generate a list of potential issues with backporting each commit, which would then undergo brief human review. Once issues are reviewed, that state would be persisted, so that if the script gets run again, it would only show *new* information based on new mailing list emails that have not already been reviewed. That's needed because these issues need to be checked for when the patch is initially proposed for stable as well as slightly later, before the actual release happens. If the stable maintainers have no time for doing *any* human review themselves (again, I do not know what their requirements are on how much time they can spend per patch), then instead an email with the list of potential issues could be generated and sent to stable@vger.kernel.org for review by others. Anyway, that's my idea. I know the response will be either "that won't work" or "patches welcome", or a mix of both, but that's it. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 17:48 ` Eric Biggers @ 2023-03-11 18:26 ` Sasha Levin 2023-03-11 18:54 ` Eric Biggers 2023-03-11 20:17 ` Eric Biggers 2023-03-11 18:33 ` Willy Tarreau 1 sibling, 2 replies; 83+ messages in thread From: Sasha Levin @ 2023-03-11 18:26 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 09:48:13AM -0800, Eric Biggers wrote: >On Sat, Mar 11, 2023 at 11:16:44AM -0500, Theodore Ts'o wrote: >> On Sat, Mar 11, 2023 at 09:06:08AM -0500, Sasha Levin wrote: >> > >> > I suppose that if I had a way to know if a certain a commit is part of a >> > series, I could either take all of it or none of it, but I don't think I >> > have a way of doing that by looking at a commit in Linus' tree >> > (suggestions welcome, I'm happy to implement them). >> >> Well, this is why I think it is a good idea to have a link to the >> patch series in lore. I know Linus doesn't like it, claiming it >> doesn't add any value, but I have to disagree. It adds two bits of >> value. >> > >So, earlier I was going to go into more detail about some of my ideas, before >Sasha and Greg started stonewalling with "patches welcome" (i.e. "I'm refusing >to do my job") and various silly arguments about why nothing should be changed. >But I suppose the worst thing that can happen is that that just continues, so >here it goes: "job"? do you think I'm paid to do this work? Why would I stonewall improvements to the process? I'm getting a bunch of suggestions and complaints that I'm not implementing those suggestions fast enough on my spare time. >One of the first things I would do if I was maintaining the stable kernels is to >set up a way to automatically run searches on the mailing lists, and then take >advantage of that in the stable process in various ways. Not having that is the >root cause of a lot of the issues with the current process, IMO. "if I was maintaining the stable kernels" - why is this rellevant? give us the tool you've proposed below and we'll be happy to use it. Heck, don't give it to us, use it to review the patches we're sending out for review and let us know if we've missed anything. >Now that lore exists, this might be trivial: it could be done just by hammering >lore.kernel.org with queries https://lore.kernel.org/linux-fsdevel/?q=query from >a Python script. > >Of course, there's a chance that won't scale to multiple queries for each one of >thousands of stable commits, or at least won't be friendly to the kernel.org >admins. In that case, what can be done is to download down all emails from all >lists, using lore's git mirrors or Atom feeds, and index them locally. (Note: >if the complete history is inconveniently large, then just indexing the last >year or so would work nearly as well.) > >Then once that is in place, that could be used in various ways. For example, >given a git commit, it's possible to search by email subject to get to the >original patch, *even if the git commit does not have a Link tag*. And it can >be automatically checked whether it's part of a patch series, and if so, whether >all the patches in the series are being backported or just some. > >This could also be used to check for mentions of a commit on the mailing list >that potentially indicate a regression report, which is one of the issues we >discussed earlier. I'm not sure what the optimal search criteria would be, but >one option would be something like "messages that contain the commit title or >commit ID and are dated to after the commit being committed". There might need >to be some exclusions added to that. > >This could also be used to automatically find the AUTOSEL email, if one exists, >and check whether it's been replied to or not. > >The purpose of all these mailing list searches would be to generate a list of >potential issues with backporting each commit, which would then undergo brief >human review. Once issues are reviewed, that state would be persisted, so that >if the script gets run again, it would only show *new* information based on new >mailing list emails that have not already been reviewed. That's needed because >these issues need to be checked for when the patch is initially proposed for >stable as well as slightly later, before the actual release happens. > >If the stable maintainers have no time for doing *any* human review themselves >(again, I do not know what their requirements are on how much time they can >spend per patch), then instead an email with the list of potential issues could >be generated and sent to stable@vger.kernel.org for review by others. > >Anyway, that's my idea. I know the response will be either "that won't work" or >"patches welcome", or a mix of both, but that's it. I've been playing with this in the past - I had a bot that looks at the mailing lists for patches that are tagged for stable, and attempts to apply/build then on the multiple trees to verify that it works and send a reply back if something goes wrong, asking for a backport. It gets a bit tricky as there's no way to go back from a commit to the initial submission, you start hitting issues like: - Patches get re-sent multiple times (think stuff like tip trees, reviews from other maintainers, etc). - Different versions of patches - for example, v1 was a single patch and in v2 it became multiple patches. I'm not arguing against your idea, I'm just saying that it's not trivial. An incomplete work here simply won't scale to the thousands of patches that flow in the trees, and won't be as useful. I don't think that this is trivial as you suggest. If you disagree, and really think it's trivial, take 5 minutes to write something up? please? -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 18:26 ` Sasha Levin @ 2023-03-11 18:54 ` Eric Biggers 2023-03-11 19:01 ` Eric Biggers ` (2 more replies) 2023-03-11 20:17 ` Eric Biggers 1 sibling, 3 replies; 83+ messages in thread From: Eric Biggers @ 2023-03-11 18:54 UTC (permalink / raw) To: Sasha Levin Cc: Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 01:26:57PM -0500, Sasha Levin wrote: > > "job"? do you think I'm paid to do this work? > Why would I stonewall improvements to the process? > > I'm getting a bunch of suggestions and complaints that I'm not implementing > those suggestions fast enough on my spare time. > > > One of the first things I would do if I was maintaining the stable kernels is to > > set up a way to automatically run searches on the mailing lists, and then take > > advantage of that in the stable process in various ways. Not having that is the > > root cause of a lot of the issues with the current process, IMO. > > "if I was maintaining the stable kernels" - why is this rellevant? give > us the tool you've proposed below and we'll be happy to use it. Heck, > don't give it to us, use it to review the patches we're sending out for > review and let us know if we've missed anything. It's kind of a stretch to claim that maintaining the stable kernels is not part of your and Greg's jobs. But anyway, the real problem is that it's currently very hard for others to contribute, given the unique role the stable maintainers have and the lack of documentation about it. Each of the two maintainers has their own scripts, and it is not clear how they use them and what processes they follow. (Even just stable-kernel-rules.rst is totally incorrect these days.) Actually I still don't even know where your scripts are! They are not in stable-queue/scripts, it seems those are only Greg's scripts? And if I built something, how do I know you would even use it? You likely have all sorts of requirements that I don't even know about. > > I've been playing with this in the past - I had a bot that looks at the > mailing lists for patches that are tagged for stable, and attempts to > apply/build then on the multiple trees to verify that it works and send > a reply back if something goes wrong, asking for a backport. > > It gets a bit tricky as there's no way to go back from a commit to the > initial submission, you start hitting issues like: > > - Patches get re-sent multiple times (think stuff like tip trees, > reviews from other maintainers, etc). > - Different versions of patches - for example, v1 was a single patch > and in v2 it became multiple patches. > > I'm not arguing against your idea, I'm just saying that it's not > trivial. An incomplete work here simply won't scale to the thousands of > patches that flow in the trees, and won't be as useful. I don't think > that this is trivial as you suggest. There are obviously going to be edge cases; another one is commits that show up in git without ever having been sent to the mailing list. I don't think they actually matter very much, though. Worst case, we miss some things, but still find everything else. > > If you disagree, and really think it's trivial, take 5 minutes to write > something up? please? I never said that it's "trivial" or that it would take only 5 minutes; that's just silly. Just that this is possible and it's what needs to be done. If you don't have time, you should instead be helping ensure that the work gets done by someone else (internship, GSoC project, etc.). And yes, I am interested in contributing, but as I mentioned I think you need to first acknowledge that there is a problem, fix your attitude of immediately pushing back on everything, and make it easier for people to contribute. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 18:54 ` Eric Biggers @ 2023-03-11 19:01 ` Eric Biggers 2023-03-11 21:14 ` Sasha Levin 2023-03-13 17:41 ` Greg KH 2 siblings, 0 replies; 83+ messages in thread From: Eric Biggers @ 2023-03-11 19:01 UTC (permalink / raw) To: Sasha Levin Cc: Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 10:55:01AM -0800, Eric Biggers wrote: > > > > If you disagree, and really think it's trivial, take 5 minutes to write > > something up? please? > > I never said that it's "trivial" or that it would take only 5 minutes; that's > just silly. Just that this is possible and it's what needs to be done. (I did say that it would be trivial to query lore.kernel.org from a Python script, which is absolutely correct. I did *not* say that the whole thing, including the local index if querying lore.kernel.org directly is not scalable enough, would be trivial. Just that it's *possible*, and there do not seem to be any technical blockers...) - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 18:54 ` Eric Biggers 2023-03-11 19:01 ` Eric Biggers @ 2023-03-11 21:14 ` Sasha Levin 2023-03-12 8:04 ` Amir Goldstein 2023-03-13 17:41 ` Greg KH 2 siblings, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-03-11 21:14 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 10:54:59AM -0800, Eric Biggers wrote: >On Sat, Mar 11, 2023 at 01:26:57PM -0500, Sasha Levin wrote: >> >> "job"? do you think I'm paid to do this work? > >> Why would I stonewall improvements to the process? >> >> I'm getting a bunch of suggestions and complaints that I'm not implementing >> those suggestions fast enough on my spare time. >> >> > One of the first things I would do if I was maintaining the stable kernels is to >> > set up a way to automatically run searches on the mailing lists, and then take >> > advantage of that in the stable process in various ways. Not having that is the >> > root cause of a lot of the issues with the current process, IMO. >> >> "if I was maintaining the stable kernels" - why is this rellevant? give >> us the tool you've proposed below and we'll be happy to use it. Heck, >> don't give it to us, use it to review the patches we're sending out for >> review and let us know if we've missed anything. > >It's kind of a stretch to claim that maintaining the stable kernels is not part >of your and Greg's jobs. But anyway, the real problem is that it's currently >very hard for others to contribute, given the unique role the stable maintainers >have and the lack of documentation about it. Each of the two maintainers has >their own scripts, and it is not clear how they use them and what processes they >follow. (Even just stable-kernel-rules.rst is totally incorrect these days.) >Actually I still don't even know where your scripts are! They are not in >stable-queue/scripts, it seems those are only Greg's scripts? And if I built >something, how do I know you would even use it? You likely have all sorts of >requirements that I don't even know about. https://kernel.googlesource.com/pub/scm/linux/kernel/git/sashal/stable-tools/ I've last updated it about two years ago, but really it's not out of date - it just doesn't get that many changes at this point. This is a mess we want to solve too: having a single repository with tools for "maintaining stable kernel trees" would be ideal and awesome, but it's quite the lift. We ended up with different scripts because we started trying to solve different issues, and ended up converging into the same tree: even now, each of us handles different subsection of commits going into the kernel tree, we just end up pushing them into the same stable branch at the end. >> >> I've been playing with this in the past - I had a bot that looks at the >> mailing lists for patches that are tagged for stable, and attempts to >> apply/build then on the multiple trees to verify that it works and send >> a reply back if something goes wrong, asking for a backport. >> >> It gets a bit tricky as there's no way to go back from a commit to the >> initial submission, you start hitting issues like: >> >> - Patches get re-sent multiple times (think stuff like tip trees, >> reviews from other maintainers, etc). >> - Different versions of patches - for example, v1 was a single patch >> and in v2 it became multiple patches. >> >> I'm not arguing against your idea, I'm just saying that it's not >> trivial. An incomplete work here simply won't scale to the thousands of >> patches that flow in the trees, and won't be as useful. I don't think >> that this is trivial as you suggest. > >There are obviously going to be edge cases; another one is commits that show up >in git without ever having been sent to the mailing list. I don't think they >actually matter very much, though. Worst case, we miss some things, but still >find everything else. Consider the opposite, which I just saw earlier today with a commit that was tagged for stable: https://lore.kernel.org/all/20230217022200.3092987-1-yukuai1@huaweicloud.com/ Here, commit 1/2 reverts a previously broken fix, and is not marked for stable. Commit 2/2 applies the proper fix, but won't apply cleanly or correctly unless you have patch 1/2. In this case you need both commits in the series, rather than none of them, otherwise you leave the trees broken. >> >> If you disagree, and really think it's trivial, take 5 minutes to write >> something up? please? > >I never said that it's "trivial" or that it would take only 5 minutes; that's >just silly. Just that this is possible and it's what needs to be done. > >If you don't have time, you should instead be helping ensure that the work gets >done by someone else (internship, GSoC project, etc.). My personal experience with this approach was that: 1. It ends up being more effort mentoring someone who is unfamailiar with this work rather than doing it myself. 2. There are very *very* few people who want to be doing this: to begin with the kernel is one of the less popular areas to get into, and on top of that the stable tree work is even worse because you do "maintenance" rather than write new shiny features. >And yes, I am interested in contributing, but as I mentioned I think you need to >first acknowledge that there is a problem, fix your attitude of immediately >pushing back on everything, and make it easier for people to contribute. I don't think we disagree that the process is broken: this is one of the reasons we went away from trying to support 6 year LTS kernels. However, we are not pushing back on ideas, we are asking for a hand in improving the process: we've been getting drive-by comments quite often, but when it comes to be doing the actual work people are quite reluctant to help. If you want to sit down and scope out initial set of work around tooling to help here I'm more than happy to do that: I'm planning to be both in OSS and LPC if you want to do it in person, along with anyone else interested in helping out. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 21:14 ` Sasha Levin @ 2023-03-12 8:04 ` Amir Goldstein 2023-03-12 16:00 ` Sasha Levin 0 siblings, 1 reply; 83+ messages in thread From: Amir Goldstein @ 2023-03-12 8:04 UTC (permalink / raw) To: Sasha Levin Cc: Eric Biggers, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel, Leah Rumancik, Luis R. Chamberlain On Sat, Mar 11, 2023 at 11:25 PM Sasha Levin <sashal@kernel.org> wrote: > > On Sat, Mar 11, 2023 at 10:54:59AM -0800, Eric Biggers wrote: ... > >And yes, I am interested in contributing, but as I mentioned I think you need to > >first acknowledge that there is a problem, fix your attitude of immediately > >pushing back on everything, and make it easier for people to contribute. > > I don't think we disagree that the process is broken: this is one of the > reasons we went away from trying to support 6 year LTS kernels. > > However, we are not pushing back on ideas, we are asking for a hand in > improving the process: we've been getting drive-by comments quite often, > but when it comes to be doing the actual work people are quite reluctant > to help. > > If you want to sit down and scope out initial set of work around tooling > to help here I'm more than happy to do that: I'm planning to be both in > OSS and LPC if you want to do it in person, along with anyone else > interested in helping out. > Sasha, Will you be able to attend a session on AUTOSEL on the overlap day of LSFMM and OSS (May 10) or earlier? We were going to discuss the topic of filesystems and stable trees [1] anyway and I believe the discussion can be even more productive with you around. I realize that the scope of AUTOSEL is wider than backporting filesystem fixes, but somehow, most of the developers on this thread are fs developers. BTW, the story of filesystem testing in stable trees has also been improving since your last appearance in LSFMM. Thanks, Amir. [1] https://lore.kernel.org/linux-fsdevel/CACzhbgSZUCn-az1e9uCh0+AO314+yq6MJTTbFt0Hj8SGCiaWjw@mail.gmail.com/ ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 8:04 ` Amir Goldstein @ 2023-03-12 16:00 ` Sasha Levin 0 siblings, 0 replies; 83+ messages in thread From: Sasha Levin @ 2023-03-12 16:00 UTC (permalink / raw) To: Amir Goldstein Cc: Eric Biggers, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel, Leah Rumancik, Luis R. Chamberlain On Sun, Mar 12, 2023 at 10:04:23AM +0200, Amir Goldstein wrote: >On Sat, Mar 11, 2023 at 11:25 PM Sasha Levin <sashal@kernel.org> wrote: >> >> On Sat, Mar 11, 2023 at 10:54:59AM -0800, Eric Biggers wrote: >... >> >And yes, I am interested in contributing, but as I mentioned I think you need to >> >first acknowledge that there is a problem, fix your attitude of immediately >> >pushing back on everything, and make it easier for people to contribute. >> >> I don't think we disagree that the process is broken: this is one of the >> reasons we went away from trying to support 6 year LTS kernels. >> >> However, we are not pushing back on ideas, we are asking for a hand in >> improving the process: we've been getting drive-by comments quite often, >> but when it comes to be doing the actual work people are quite reluctant >> to help. >> >> If you want to sit down and scope out initial set of work around tooling >> to help here I'm more than happy to do that: I'm planning to be both in >> OSS and LPC if you want to do it in person, along with anyone else >> interested in helping out. >> > >Sasha, > >Will you be able to attend a session on AUTOSEL on the overlap day >of LSFMM and OSS (May 10) or earlier? > >We were going to discuss the topic of filesystems and stable trees [1] anyway >and I believe the discussion can be even more productive with you around. > >I realize that the scope of AUTOSEL is wider than backporting filesystem fixes, >but somehow, most of the developers on this thread are fs developers. > >BTW, the story of filesystem testing in stable trees has also been improving >since your last appearance in LSFMM. Happy to stop by and collaborate! I'll also be in Vancouver the whole week (though not in LSF/MM), so if you'd want to find time for a workshop around this topic with interested parties we can look into that too. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 18:54 ` Eric Biggers 2023-03-11 19:01 ` Eric Biggers 2023-03-11 21:14 ` Sasha Levin @ 2023-03-13 17:41 ` Greg KH 2023-03-13 18:54 ` Eric Biggers 2 siblings, 1 reply; 83+ messages in thread From: Greg KH @ 2023-03-13 17:41 UTC (permalink / raw) To: Eric Biggers Cc: Sasha Levin, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 10:54:59AM -0800, Eric Biggers wrote: > On Sat, Mar 11, 2023 at 01:26:57PM -0500, Sasha Levin wrote: > > > > "job"? do you think I'm paid to do this work? > > > Why would I stonewall improvements to the process? > > > > I'm getting a bunch of suggestions and complaints that I'm not implementing > > those suggestions fast enough on my spare time. > > > > > One of the first things I would do if I was maintaining the stable kernels is to > > > set up a way to automatically run searches on the mailing lists, and then take > > > advantage of that in the stable process in various ways. Not having that is the > > > root cause of a lot of the issues with the current process, IMO. > > > > "if I was maintaining the stable kernels" - why is this rellevant? give > > us the tool you've proposed below and we'll be happy to use it. Heck, > > don't give it to us, use it to review the patches we're sending out for > > review and let us know if we've missed anything. > > It's kind of a stretch to claim that maintaining the stable kernels is not part > of your and Greg's jobs. But anyway, the real problem is that it's currently > very hard for others to contribute, given the unique role the stable maintainers > have and the lack of documentation about it. Each of the two maintainers has > their own scripts, and it is not clear how they use them and what processes they > follow. Just a comment here about our scripts and process. Our scripts are different as we both currently do different things for the stable trees. I have almost no scripts for finding patches, all I use is a git hook that dumps emails into a mbox and then go through them and queue them up to the quilt trees based on if they are valid or not after review. My scripts primarily are for doing a release, not building the patches up. That being said, I do have 2 scripts I use to run on an existing tree or series to verify that the fixes are all present already (i.e. if we have fixes for the fixes), but that's not really relevant for this discussion. Those, and my big "treat the filesystem as a git database" hack can be found in this repo: https://git.sr.ht/~gregkh/linux-stable_commit_tree/ if you are curious, these are probably the relevant scripts if you are curious: https://git.sr.ht/~gregkh/linux-stable_commit_tree/tree/master/item/find_fixes_in_queue https://git.sr.ht/~gregkh/linux-stable_commit_tree/tree/master/item/find_fixes_in_range And I use: https://git.sr.ht/~gregkh/linux-stable_commit_tree/tree/master/item/id_found_in all the time to determine if a SHA1 is in any stable releases. > (Even just stable-kernel-rules.rst is totally incorrect these days.) I do not understand this, what is not correct? It's how to get patches merged into stable kernels, we go above-and-beyond that for those developers and maintainers that do NOT follow those rules. If everyone followed them, we wouldn't be having this discussion at all :) > Actually I still don't even know where your scripts are! They are not in > stable-queue/scripts, it seems those are only Greg's scripts? And if I built > something, how do I know you would even use it? You likely have all sorts of > requirements that I don't even know about. I think what you are talking about here would require new work. New tools to dig in the commits to extract "here's the whole series of patches" would be wonderful, but as others have pointed out, it is _very_ common to have a cc: stable as the first few commits in a series, and then the rest have nothing to do with a stable tree. But when doing something like what AUTOSEL does, digging up the whole series would be great. We have tools that can match up every commit in the tree to a specific email message (presentations on the tool and how it works have been a previous LinuxCon conferences), but if we can use lore.kernel.org for it, that would probably help everyone out. And that's why I use the Link: tag, as Ted pointed out, for everything that I apply to all of the subsystems I work with. While I know Linus doesn't like it, I think it is quite valuable as it makes it so that _anyone_ can instantly find the thread where the patch came from, and no external tools are required. Anyway, as always, I gladly accept help with figuring out what commits to apply to stable kernels. I've always said this, and Sasha has stepped up in an amazing way here over the years, creating tools based on collaboration with many others (see his presentations at conferences with Julia) on how to dig into the kernel repo to find patches that we all forget to tag for stable kernels and he sends them out for review. If you want to help out and do much the same thing using different sorts of tools, or come up with other ways of finding the bugfixes that are in there that are not properly tagged, wonderful, I will gladly accept them, I have never turned down help like this. And that's what I ask from companies all the time when they say "what can we do to help out?" A simple thing to do is dig in your vendor trees and send me the fixes that you have backported there. I know distros have this (and some distros help out and do this, I'll call out Debian for being very good here), and some companies do submit their backports as well (Amazon and Hawaii are good, Android also does a good job), but they are rare compared to all of the groups that I know use Linux. Anyway, if anyone noticed the big problems this weekend with the stable releases were due to patches that were actually tagged with "cc: stable" so that's kind of proof that we all are human and even when we think a fix is enough, it can cause problems when it hits real world testing. We are all human, the best we can do is when confronted with "hey, this fix causes a problem" is revert it and get the fix out to people as quick as possible. That includes fixes picked from tools like AUTOSEL as well as manual tags, there is no difference here in our response. thanks, greg k-h ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-13 17:41 ` Greg KH @ 2023-03-13 18:54 ` Eric Biggers 2023-03-14 18:26 ` Greg KH 0 siblings, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-13 18:54 UTC (permalink / raw) To: Greg KH Cc: Sasha Levin, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Mon, Mar 13, 2023 at 06:41:49PM +0100, Greg KH wrote: > > (Even just stable-kernel-rules.rst is totally incorrect these days.) > > I do not understand this, what is not correct? > > It's how to get patches merged into stable kernels, we go > above-and-beyond that for those developers and maintainers that do NOT > follow those rules. If everyone followed them, we wouldn't be having > this discussion at all :) The entire list of rules for what patches are accepted into stable. This is a longstanding issue that has been reiterated many times in the past, see https://lore.kernel.org/stable/20220924182124.GA19210@duo.ucw.cz for example. The fact is, many people *do* follow the rules exactly by *not* tagging commits for stable when they don't meet the documented eligibility criteria. But then the stable maintainers backport the commits anyway, as the real eligibility criteria are *much* more relaxed than what is documented. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-13 18:54 ` Eric Biggers @ 2023-03-14 18:26 ` Greg KH 0 siblings, 0 replies; 83+ messages in thread From: Greg KH @ 2023-03-14 18:26 UTC (permalink / raw) To: Eric Biggers Cc: Sasha Levin, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Mon, Mar 13, 2023 at 11:54:17AM -0700, Eric Biggers wrote: > > The fact is, many people *do* follow the rules exactly by *not* tagging commits > for stable when they don't meet the documented eligibility criteria. But then > the stable maintainers backport the commits anyway, as the real eligibility > criteria are *much* more relaxed than what is documented. Again, if you do NOT want your patches backported, because you always mark them properly for stable releases, just let us know and we will add you to the list of other subsystems and maintainers that have asked us for this in the past. We can't detect the absence of a tag as "I know exactly what I am doing" because of the huge number of developers who do NOT tag patches, and so, we have to dig through the tree to find fixes on our own. thanks, greg k-h ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 18:26 ` Sasha Levin 2023-03-11 18:54 ` Eric Biggers @ 2023-03-11 20:17 ` Eric Biggers 2023-03-11 21:02 ` Sasha Levin 1 sibling, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-11 20:17 UTC (permalink / raw) To: Sasha Levin Cc: Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 01:26:57PM -0500, Sasha Levin wrote: > I'm getting a bunch of suggestions and complaints that I'm not implementing > those suggestions fast enough on my spare time. BTW, the "I don't have enough time" argument is also a little frustrating because you are currently insisting on doing AUTOSEL at all, at the current sensitivity that picks up way too many commits. I can certainly imagine that that uses a lot of your time! But, many contributors are telling you that AUTOSEL is actually *worse than nothing* currently. So to some extent this is a self-inflicted problem. You are *choosing* to spend your precious time running in-place with something that is not working well, instead of putting AUTOSEL on pause or turning down the sensitivity to free up time while improvements to the process are worked on. (And yes, I know there are many stable patches besides AUTOSEL, and it's a lot of work, and I'm grateful for what you do. I am *just* talking about AUTOSEL here. And yes, I agree that AUTOSEL is needed in principle, so there's no need to re-hash the arguments for why it exists. It just needs some improvements.) - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 20:17 ` Eric Biggers @ 2023-03-11 21:02 ` Sasha Levin 2023-03-12 4:23 ` Willy Tarreau 0 siblings, 1 reply; 83+ messages in thread From: Sasha Levin @ 2023-03-11 21:02 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 12:17:50PM -0800, Eric Biggers wrote: >On Sat, Mar 11, 2023 at 01:26:57PM -0500, Sasha Levin wrote: >> I'm getting a bunch of suggestions and complaints that I'm not implementing >> those suggestions fast enough on my spare time. > >BTW, the "I don't have enough time" argument is also a little frustrating >because you are currently insisting on doing AUTOSEL at all, at the current >sensitivity that picks up way too many commits. I can certainly imagine that >that uses a lot of your time! But, many contributors are telling you that >AUTOSEL is actually *worse than nothing* currently. > >So to some extent this is a self-inflicted problem. You are *choosing* to spend >your precious time running in-place with something that is not working well, >instead of putting AUTOSEL on pause or turning down the sensitivity to free up >time while improvements to the process are worked on. > >(And yes, I know there are many stable patches besides AUTOSEL, and it's a lot >of work, and I'm grateful for what you do. I am *just* talking about AUTOSEL >here. And yes, I agree that AUTOSEL is needed in principle, so there's no need >to re-hash the arguments for why it exists. It just needs some improvements.) Just to make sure I'm sending the right message: I'd *love* to improve it, but I need help. I'm not pushing back on your ideas, I'm asking for help with their implementation. Maybe I'm putting words in Greg's mouth, but I think we both would ideally want to standardize around a single set of tools and scripts, it's just the case that both of us started with different set of problems we were trying to solve, and so our tooling evolved independently. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 21:02 ` Sasha Levin @ 2023-03-12 4:23 ` Willy Tarreau 0 siblings, 0 replies; 83+ messages in thread From: Willy Tarreau @ 2023-03-12 4:23 UTC (permalink / raw) To: Sasha Levin Cc: Eric Biggers, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 04:02:15PM -0500, Sasha Levin wrote: > Maybe I'm putting words in Greg's mouth, but I think we both would > ideally want to standardize around a single set of tools and scripts, > it's just the case that both of us started with different set of > problems we were trying to solve, and so our tooling evolved > independently. I'm not even sure this is strictly necessary. When I started jumping on 2.6 Greg told me "now you have to follow the new process involving a first preview release", and that completely changed the way I was dealing with fixes in 2.4. I then tried to adopt Greg's scripts, and when you do that you instantly figure that different people are at ease with different parts of the process, and over time I started to keep a collection of hacked scripts that would simplify certain parts of the process for me (stuff as stupid as creating directories having the kernel name in hard-coded paths in my work directory, or converting the "cherry-picked from" to the "upstream commit" format, etc). Your biggest difficulty probably is to find people willing and able to help, and while you should definitely show them your collection of tools to help them get started, these can also be frightening for newcomers or possibly not much useful to them if they come from a different background. Cheers, Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 17:48 ` Eric Biggers 2023-03-11 18:26 ` Sasha Levin @ 2023-03-11 18:33 ` Willy Tarreau 2023-03-11 19:24 ` Eric Biggers 1 sibling, 1 reply; 83+ messages in thread From: Willy Tarreau @ 2023-03-11 18:33 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 09:48:13AM -0800, Eric Biggers wrote: > The purpose of all these mailing list searches would be to generate a list of > potential issues with backporting each commit, which would then undergo brief > human review. This is one big part that I suspect is underestimated. I'll speak from my past experience maintaining extended LTS for 3.10. I couldn't produce as many releases as I would have liked to because despite the scripts that helped me figure some series, some dependencies, origin branches etc, the whole process of reviewing ~600 patches to end up with ~200 at the end (and adapting some of them to fit) required ~16 hours a day for a full week-end, and I didn't always have that amount of time available. Any my choices were far from being perfect, as during the reviews I got a number of "please don't backport this there" and "if you take this one you also need these ones". Also I used to intentionally drop what had nothing to do on old LTS stuff so even from that perspective my work could have been perceived as insufficient. The reviewing process is overwhelming, really. There is a point where you start to fail and make choices that are not better than a machine's. But is a mistake once in a while dramatic if on the other hand it fixes 200 other issues ? I think not as long as it's transparent and accepted by the users, because for one user that could experience a regression (one that escaped all the testing in place), thousands get fixes for existing problems. I'm not saying that regressions are good, I hate them, but as James said, we have to accept that user are part of the quality process. My approach on another project I maintain is to announce upfront my own level of trust in my backport work, saying "I had a difficult week fixing that problem, do not rush on it or be extra careful", or "nothing urgent, no need to upgrade if you have no problem" or also "just upgrade, it's almost riskless". Users love that, because they know they're part of the quality assurance process, and they will either take small risks when they can, or wait for others to take risks. But thinking that having one person review patches affecting many subsystem after pre-selection and extra info regarding discussions on each individual patch could result in more reliable stable releases is just an illusion IMHO, because the root of problem is that there are not enough humans to fix all the problems that humans introduce in the first place, and despite this we need to fix them. Just like automated scripts scraping lore, AUTOSEL does bring some value if it offloads some work from the available humans, even in its current state. And I hope that more of the selection and review work in the future will be automated and even less dependent on humans, because it does have a chance to be more reliable in front of that vast amount of work. And in any case I've seen you use the word "trivial" several times in this thread, and for having been through a little bit of this process in the past, I wouldn't use that word anywhere in a description of what my experience had been. You really seem to underestimate the difficulty here. Regards, Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 18:33 ` Willy Tarreau @ 2023-03-11 19:24 ` Eric Biggers 2023-03-11 19:46 ` Eric Biggers 2023-03-11 20:11 ` Willy Tarreau 0 siblings, 2 replies; 83+ messages in thread From: Eric Biggers @ 2023-03-11 19:24 UTC (permalink / raw) To: Willy Tarreau Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 07:33:58PM +0100, Willy Tarreau wrote: > On Sat, Mar 11, 2023 at 09:48:13AM -0800, Eric Biggers wrote: > > The purpose of all these mailing list searches would be to generate a list of > > potential issues with backporting each commit, which would then undergo brief > > human review. > > This is one big part that I suspect is underestimated. I'll speak from my > past experience maintaining extended LTS for 3.10. I couldn't produce as > many releases as I would have liked to because despite the scripts that > helped me figure some series, some dependencies, origin branches etc, the > whole process of reviewing ~600 patches to end up with ~200 at the end > (and adapting some of them to fit) required ~16 hours a day for a full > week-end, and I didn't always have that amount of time available. Any my > choices were far from being perfect, as during the reviews I got a number > of "please don't backport this there" and "if you take this one you also > need these ones". Also I used to intentionally drop what had nothing to > do on old LTS stuff so even from that perspective my work could have been > perceived as insufficient. > > The reviewing process is overwhelming, really. There is a point where you > start to fail and make choices that are not better than a machine's. But > is a mistake once in a while dramatic if on the other hand it fixes 200 > other issues ? I think not as long as it's transparent and accepted by > the users, because for one user that could experience a regression (one > that escaped all the testing in place), thousands get fixes for existing > problems. I'm not saying that regressions are good, I hate them, but as > James said, we have to accept that user are part of the quality process. > > My approach on another project I maintain is to announce upfront my own > level of trust in my backport work, saying "I had a difficult week fixing > that problem, do not rush on it or be extra careful", or "nothing urgent, > no need to upgrade if you have no problem" or also "just upgrade, it's > almost riskless". Users love that, because they know they're part of the > quality assurance process, and they will either take small risks when > they can, or wait for others to take risks. > > But thinking that having one person review patches affecting many > subsystem after pre-selection and extra info regarding discussions on > each individual patch could result in more reliable stable releases is > just an illusion IMHO, because the root of problem is that there are not > enough humans to fix all the problems that humans introduce in the first > place, and despite this we need to fix them. Just like automated scripts > scraping lore, AUTOSEL does bring some value if it offloads some work > from the available humans, even in its current state. And I hope that > more of the selection and review work in the future will be automated > and even less dependent on humans, because it does have a chance to be > more reliable in front of that vast amount of work. As I said in a part of my email which you did not quote, the fallback option is to send the list of issues to the mailing list for others to review. If even that fails, then it could be cut down to the *just the most useful* heuristics and decisions made automatically based on those... "Don't AUTOSEL patch N of a series without 1...N-1" might be a good one. But again, this comes back to one of the core issues here which is how does one even build something for the stable maintainers if their requirements are unknown to others? > And in any case I've seen you use the word "trivial" several times in > this thread, and for having been through a little bit of this process > in the past, I wouldn't use that word anywhere in a description of what > my experience had been. You really seem to underestimate the difficulty > here. I checked the entire email thread (https://lore.kernel.org/stable/?q=f%3Aebiggers+trivial). The only place I used the word "trivial" was mentioning that querying lore.kernel.org from a Python script might be trivial, which is true. And also in my response to Sasha's similar false claim that I was saying everything would be trivial. I'm not sure why you're literally just making things up; it's not a very good way to have a productive discussion... - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 19:24 ` Eric Biggers @ 2023-03-11 19:46 ` Eric Biggers 2023-03-11 20:19 ` Willy Tarreau 2023-03-11 20:11 ` Willy Tarreau 1 sibling, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-11 19:46 UTC (permalink / raw) To: Willy Tarreau Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 11:24:31AM -0800, Eric Biggers wrote: > On Sat, Mar 11, 2023 at 07:33:58PM +0100, Willy Tarreau wrote: > > On Sat, Mar 11, 2023 at 09:48:13AM -0800, Eric Biggers wrote: > > > The purpose of all these mailing list searches would be to generate a list of > > > potential issues with backporting each commit, which would then undergo brief > > > human review. > > > > This is one big part that I suspect is underestimated. I'll speak from my > > past experience maintaining extended LTS for 3.10. I couldn't produce as > > many releases as I would have liked to because despite the scripts that > > helped me figure some series, some dependencies, origin branches etc, the > > whole process of reviewing ~600 patches to end up with ~200 at the end > > (and adapting some of them to fit) required ~16 hours a day for a full > > week-end, and I didn't always have that amount of time available. Any my > > choices were far from being perfect, as during the reviews I got a number > > of "please don't backport this there" and "if you take this one you also > > need these ones". Also I used to intentionally drop what had nothing to > > do on old LTS stuff so even from that perspective my work could have been > > perceived as insufficient. > > > > The reviewing process is overwhelming, really. There is a point where you > > start to fail and make choices that are not better than a machine's. But > > is a mistake once in a while dramatic if on the other hand it fixes 200 > > other issues ? I think not as long as it's transparent and accepted by > > the users, because for one user that could experience a regression (one > > that escaped all the testing in place), thousands get fixes for existing > > problems. I'm not saying that regressions are good, I hate them, but as > > James said, we have to accept that user are part of the quality process. > > > > My approach on another project I maintain is to announce upfront my own > > level of trust in my backport work, saying "I had a difficult week fixing > > that problem, do not rush on it or be extra careful", or "nothing urgent, > > no need to upgrade if you have no problem" or also "just upgrade, it's > > almost riskless". Users love that, because they know they're part of the > > quality assurance process, and they will either take small risks when > > they can, or wait for others to take risks. > > > > But thinking that having one person review patches affecting many > > subsystem after pre-selection and extra info regarding discussions on > > each individual patch could result in more reliable stable releases is > > just an illusion IMHO, because the root of problem is that there are not > > enough humans to fix all the problems that humans introduce in the first > > place, and despite this we need to fix them. Just like automated scripts > > scraping lore, AUTOSEL does bring some value if it offloads some work > > from the available humans, even in its current state. And I hope that > > more of the selection and review work in the future will be automated > > and even less dependent on humans, because it does have a chance to be > > more reliable in front of that vast amount of work. > > As I said in a part of my email which you did not quote, the fallback option is > to send the list of issues to the mailing list for others to review. > > If even that fails, then it could be cut down to the *just the most useful* > heuristics and decisions made automatically based on those... "Don't AUTOSEL > patch N of a series without 1...N-1" might be a good one. > > But again, this comes back to one of the core issues here which is how does one > even build something for the stable maintainers if their requirements are > unknown to others? > Another issue that I'd like to reiterate is that AUTOSEL is currently turned up to 11. It's simply selecting too much. It should be made less sensitive and select higher confidence commits only. That would cut down on the workload slightly. (And please note, the key word here is *confidence*. We all agree that it's never possible to be absolutely 100% sure whether a commit is appropriate for stable or not. That's a red herring. And I would assume, or at least hope, that the neural network thing being used for AUTOSEL outputs a confidence rating and not just a yes/no answer. If it actually just outputs yes/no, well how is anyone supposed to know that and fix that, given that it does not seem to be an open source project?) - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 19:46 ` Eric Biggers @ 2023-03-11 20:19 ` Willy Tarreau 2023-03-11 20:59 ` Sasha Levin 0 siblings, 1 reply; 83+ messages in thread From: Willy Tarreau @ 2023-03-11 20:19 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 11:46:05AM -0800, Eric Biggers wrote: > (And please note, the key word here is *confidence*. We all agree that it's > never possible to be absolutely 100% sure whether a commit is appropriate for > stable or not. That's a red herring. In fact even developers themselves sometimes don't know, and even when they know, sometimes they know after committing it. Many times we've found that a bug was accidently resolved by a small change. Just for this it's important to support a post-merge analysis. > And I would assume, or at least hope, that the neural network thing being used > for AUTOSEL outputs a confidence rating and not just a yes/no answer. If it > actually just outputs yes/no, well how is anyone supposed to know that and fix > that, given that it does not seem to be an open source project?) Honestly I don't know. I ran a few experiments with natural language processors such as GPT-3 on commit messages which contained human-readable instructions, and asking "what am I expected to do with these patches", and seeing the bot respond "you should backport them to this version, change this and that in that version, and preliminary take that patch". It summarized extremely well the instructions delivered by the developer, which is awesome, but was not able to provide any form of confidence level. I don't know what Sasha uses but wouldn't be surprised it shares some such mechanisms and that it might not always be easy to get such a confidence level. But I could be wrong. Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 20:19 ` Willy Tarreau @ 2023-03-11 20:59 ` Sasha Levin 0 siblings, 0 replies; 83+ messages in thread From: Sasha Levin @ 2023-03-11 20:59 UTC (permalink / raw) To: Willy Tarreau Cc: Eric Biggers, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 09:19:54PM +0100, Willy Tarreau wrote: >On Sat, Mar 11, 2023 at 11:46:05AM -0800, Eric Biggers wrote: >> (And please note, the key word here is *confidence*. We all agree that it's >> never possible to be absolutely 100% sure whether a commit is appropriate for >> stable or not. That's a red herring. > >In fact even developers themselves sometimes don't know, and even when they >know, sometimes they know after committing it. Many times we've found that >a bug was accidently resolved by a small change. Just for this it's important >to support a post-merge analysis. >> And I would assume, or at least hope, that the neural network thing being used >> for AUTOSEL outputs a confidence rating and not just a yes/no answer. If it >> actually just outputs yes/no, well how is anyone supposed to know that and fix >> that, given that it does not seem to be an open source project?) > >Honestly I don't know. I ran a few experiments with natural language >processors such as GPT-3 on commit messages which contained human-readable >instructions, and asking "what am I expected to do with these patches", and >seeing the bot respond "you should backport them to this version, change >this and that in that version, and preliminary take that patch". It >summarized extremely well the instructions delivered by the developer, >which is awesome, but was not able to provide any form of confidence >level. I don't know what Sasha uses but wouldn't be surprised it shares >some such mechanisms and that it might not always be easy to get such a >confidence level. But I could be wrong. It's actually pretty stupid: it uses the existence of ~10k of the most common words in commit messages + metrics from cqmetrics (github.com/dspinellis/cqmetrics) as input. Although I get a score, which is already set pretty high, confidence is really non-existant here: at the end it depends mostly on the writing style of said commit author more than anything. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 19:24 ` Eric Biggers 2023-03-11 19:46 ` Eric Biggers @ 2023-03-11 20:11 ` Willy Tarreau 2023-03-11 20:53 ` Eric Biggers 2023-03-12 13:55 ` Mark Brown 1 sibling, 2 replies; 83+ messages in thread From: Willy Tarreau @ 2023-03-11 20:11 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 11:24:31AM -0800, Eric Biggers wrote: > > But thinking that having one person review patches affecting many > > subsystem after pre-selection and extra info regarding discussions on > > each individual patch could result in more reliable stable releases is > > just an illusion IMHO, because the root of problem is that there are not > > enough humans to fix all the problems that humans introduce in the first > > place, and despite this we need to fix them. Just like automated scripts > > scraping lore, AUTOSEL does bring some value if it offloads some work > > from the available humans, even in its current state. And I hope that > > more of the selection and review work in the future will be automated > > and even less dependent on humans, because it does have a chance to be > > more reliable in front of that vast amount of work. > > As I said in a part of my email which you did not quote, the fallback option is > to send the list of issues to the mailing list for others to review. Honestly, patches are already being delivered publicly tagged AUTOSEL, then published again as part of the stable review process. Have you seen the amount of feedback ? Once in a while there are responses, but aside Guenter reporting build successes or failures, it's a bit quiet. What makes you think that sending more detailed stuff that require even more involvement and decision would trigger more participation ? > If even that fails, then it could be cut down to the *just the most useful* > heuristics and decisions made automatically based on those... "Don't AUTOSEL > patch N of a series without 1...N-1" might be a good one. I do think that this one would be an improvement. But it needs to push harder. Not "don't autosel", but sending the message to relevant parties (all those involved in the patch being reviewed and merged) indicating "we are going to merge this patch, but it's part of the following series, should any/all/none of them be picked ? barring any response only this patch will be picked". And of course, ideally all selected ones from a series should be proposed at once to ease the review. > But again, this comes back to one of the core issues here which is how does one > even build something for the stable maintainers if their requirements are > unknown to others? Well, the description of the commit message is there for anyone to consume in the first place. A commit message is an argument for a patch to get adopted and resist any temptations to revert it. So it must be descriptive enough and give instructions. Dependencies should be clear there. When you seen github-like one-liners there's no hope to get much info, and it's not a matter of requirements, but of respect for a team development process where some facing your patch might miss the skills required to grasp the details. With a sufficiently clear commit message, even a bot can find (or suggest) dependencies. And this is not specific to -stable: if one of the dependencies is found to break stuff, how do you know it must not be reverted without reverting the whole series if that's not described anywhere ? > > And in any case I've seen you use the word "trivial" several times in > > this thread, and for having been through a little bit of this process > > in the past, I wouldn't use that word anywhere in a description of what > > my experience had been. You really seem to underestimate the difficulty > > here. > > I checked the entire email thread > (https://lore.kernel.org/stable/?q=f%3Aebiggers+trivial). The only place I used > the word "trivial" was mentioning that querying lore.kernel.org from a Python > script might be trivial, which is true. I'm unable to do it, so at best it's trivial for someone at ease with Python and the lore API. And parsing the results and classifying them might not be trivial at all either. Getting information is one part, processing it is another thing. > And also in my response to Sasha's > similar false claim that I was saying everything would be trivial. > > I'm not sure why you're literally just making things up; it's not a very good > way to have a productive discussion... I'm not making things up. Maybe you wrote "trivial" only once but the tone of your suggestions from the beginning was an exact description of something called trivial and made me feel you find all of this "trivial", which you finally confirmed in that excerpt above. Quite frankly, I'm not part of this process anymore and am really thankful that the current maintainers are doing that work. But it makes me feel really uneasy to read suggestions basically sounding like "why don't you fix your broken selection process" or "it should just be as trivial as collecting the missing info from lore". Had I received such contemptuous "suggestions" when I was doing that job, I would just have resigned. And just saying things like "I will not start helping before you change your attitude" you appear infantilizing at best and in no way looking like you're really willing to help. Sasha said he was open to receive proposals and suddenly the trivial job gets conditions. Just do your part of the work that seems poorly done to you, and everyone will see if your ideas and work finally helped or not. Nobody will even care if it was trivial or if it ended up taking 4 months of refining, as long as it helps in the end. But I suspect that you're not interested in helping, just in complaining. One thing I think that could be within reach and could very slightly improve the process would be to indicate in a stable announce the amount of patches coming from autosel. I think that it could help either refining the selection by making users more conscious about the importance of this source, or encourage more developers to Cc stable to reduce that ratio. Just an idea. Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 20:11 ` Willy Tarreau @ 2023-03-11 20:53 ` Eric Biggers 2023-03-12 4:32 ` Willy Tarreau 2023-03-12 13:55 ` Mark Brown 1 sibling, 1 reply; 83+ messages in thread From: Eric Biggers @ 2023-03-11 20:53 UTC (permalink / raw) To: Willy Tarreau Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 09:11:51PM +0100, Willy Tarreau wrote: > On Sat, Mar 11, 2023 at 11:24:31AM -0800, Eric Biggers wrote: > > > But thinking that having one person review patches affecting many > > > subsystem after pre-selection and extra info regarding discussions on > > > each individual patch could result in more reliable stable releases is > > > just an illusion IMHO, because the root of problem is that there are not > > > enough humans to fix all the problems that humans introduce in the first > > > place, and despite this we need to fix them. Just like automated scripts > > > scraping lore, AUTOSEL does bring some value if it offloads some work > > > from the available humans, even in its current state. And I hope that > > > more of the selection and review work in the future will be automated > > > and even less dependent on humans, because it does have a chance to be > > > more reliable in front of that vast amount of work. > > > > As I said in a part of my email which you did not quote, the fallback option is > > to send the list of issues to the mailing list for others to review. > > Honestly, patches are already being delivered publicly tagged AUTOSEL, > then published again as part of the stable review process. Have you seen > the amount of feedback ? Once in a while there are responses, but aside > Guenter reporting build successes or failures, it's a bit quiet. What > makes you think that sending more detailed stuff that require even more > involvement and decision would trigger more participation ? Well, there is not much people can do with 1000 patches with no context, but if there's a much shorter list of potential issues to pay attention to, I'd hope that would be helpful. > > I checked the entire email thread > > (https://lore.kernel.org/stable/?q=f%3Aebiggers+trivial). The only place I used > > the word "trivial" was mentioning that querying lore.kernel.org from a Python > > script might be trivial, which is true. > > I'm unable to do it, so at best it's trivial for someone at ease with > Python and the lore API. And parsing the results and classifying them > might not be trivial at all either. Getting information is one part, > processing it is another thing. > > > And also in my response to Sasha's > > similar false claim that I was saying everything would be trivial. > > > > I'm not sure why you're literally just making things up; it's not a very good > > way to have a productive discussion... > > I'm not making things up. Maybe you wrote "trivial" only once but the tone > of your suggestions from the beginning was an exact description of something > called trivial and made me feel you find all of this "trivial", which you > finally confirmed in that excerpt above. You are indeed making things up, which is annoying as it makes it hard to have a real discussion. Anyway, hopefully we can get past that. > Quite frankly, I'm not part of this process anymore and am really thankful > that the current maintainers are doing that work. But it makes me feel > really uneasy to read suggestions basically sounding like "why don't you > fix your broken selection process" or "it should just be as trivial as > collecting the missing info from lore". Had I received such contemptuous > "suggestions" when I was doing that job, I would just have resigned. And > just saying things like "I will not start helping before you change your > attitude" you appear infantilizing at best and in no way looking like > you're really willing to help. Sasha said he was open to receive proposals > and suddenly the trivial job gets conditions. Just do your part of the > work that seems poorly done to you, and everyone will see if your ideas > and work finally helped or not. Nobody will even care if it was trivial > or if it ended up taking 4 months of refining, as long as it helps in > the end. But I suspect that you're not interested in helping, just in > complaining. > > One thing I think that could be within reach and could very slightly > improve the process would be to indicate in a stable announce the amount > of patches coming from autosel. I think that it could help either > refining the selection by making users more conscious about the importance > of this source, or encourage more developers to Cc stable to reduce that > ratio. Just an idea. I'll try to put something together, despite all the pushback I'm getting. But by necessity it will be totally separate from the current stable scripts, as it seems there is no practical way for me to do it otherwise, given that the current stable process is not properly open and lacks proper leadership. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 20:53 ` Eric Biggers @ 2023-03-12 4:32 ` Willy Tarreau 2023-03-12 5:21 ` Eric Biggers 0 siblings, 1 reply; 83+ messages in thread From: Willy Tarreau @ 2023-03-12 4:32 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 12:53:29PM -0800, Eric Biggers wrote: > I'll try to put something together, despite all the pushback I'm getting. Thanks. > But > by necessity it will be totally separate from the current stable scripts, as it > seems there is no practical way for me to do it otherwise, It's better that way anyway. Adding diversity to the process is important if we want to experiment with multiple approaches. What matters is to have multiple inputs on list of patches. > given that the > current stable process is not properly open and lacks proper leadership. Please, really please, stop looping on this. I think it was already explained quite a few times that the process is mostly human, and that it's very difficult to document what has to be done. It's a lot of work based on common sense, intuition and experience which helps solving each an every individual case. The scripts that help are public, the rest is just experience. It's not fair to say that some people do not follow an open process while they're using their experience and intuition. They're not machines. Thanks, Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 4:32 ` Willy Tarreau @ 2023-03-12 5:21 ` Eric Biggers 2023-03-12 5:48 ` Willy Tarreau 2023-03-12 7:42 ` Amir Goldstein 0 siblings, 2 replies; 83+ messages in thread From: Eric Biggers @ 2023-03-12 5:21 UTC (permalink / raw) To: Willy Tarreau Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sun, Mar 12, 2023 at 05:32:32AM +0100, Willy Tarreau wrote: > On Sat, Mar 11, 2023 at 12:53:29PM -0800, Eric Biggers wrote: > > I'll try to put something together, despite all the pushback I'm getting. > > Thanks. > > > But > > by necessity it will be totally separate from the current stable scripts, as it > > seems there is no practical way for me to do it otherwise, > > It's better that way anyway. Adding diversity to the process is important > if we want to experiment with multiple approaches. What matters is to > have multiple inputs on list of patches. > > > given that the > > current stable process is not properly open and lacks proper leadership. > > Please, really please, stop looping on this. I think it was already > explained quite a few times that the process is mostly human, and that > it's very difficult to document what has to be done. It's a lot of work > based on common sense, intuition and experience which helps solving each > an every individual case. The scripts that help are public, the rest is > just experience. It's not fair to say that some people do not follow an > open process while they're using their experience and intuition. They're > not machines. > I mean, "patches welcome" is a bit pointless when there is nothing to patch, is it not? Even Sasha's stable-tools, which he finally gave a link to, does not include anything related to AUTOSEL. It seems AUTOSEL is still closed source. BTW, I already did something similar "off to the side" a few years ago when I wrote a script to keep track of and prioritize syzbot reports from https://syzkaller.appspot.com/, and generate per-subsystem reminder emails. I eventually ended up abandoning that, because doing something off to the side is not very effective and is hard to keep up with. The right approach is to make improvements to the "upstream" process (which was syzbot in that case), not to bolt something on to the side to try to fix it after the fact. So I hope people can understand where I'm coming from, with hoping that what the stable maintainers are doing can just be improved directly, without first building something from scratch off to the side as that is just not a good way to do things. But sure, if that's the only option to get anything nontrivial changed, I'll try to do it. - Eric ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 5:21 ` Eric Biggers @ 2023-03-12 5:48 ` Willy Tarreau 2023-03-12 7:42 ` Amir Goldstein 1 sibling, 0 replies; 83+ messages in thread From: Willy Tarreau @ 2023-03-12 5:48 UTC (permalink / raw) To: Eric Biggers Cc: Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 09:21:58PM -0800, Eric Biggers wrote: > I mean, "patches welcome" is a bit pointless when there is nothing to patch, is > it not? Maybe it's because they're used to regularly receive complaints suggesting to improve the process without knowing where to start from. > Even Sasha's stable-tools, which he finally gave a link to, does not > include anything related to AUTOSEL. It seems AUTOSEL is still closed source. I don't know. > BTW, I already did something similar "off to the side" a few years ago when I > wrote a script to keep track of and prioritize syzbot reports from > https://syzkaller.appspot.com/, and generate per-subsystem reminder emails. > > I eventually ended up abandoning that, because doing something off to the side > is not very effective and is hard to keep up with. The right approach is to > make improvements to the "upstream" process (which was syzbot in that case), not > to bolt something on to the side to try to fix it after the fact. I think that improving the upstreaming process does have some value, of course, especially when it's done from tools that can reliably and durably be improved. But it's not rocket science when input comes from humans. We still occasionally see patches missing an s-o-b. Humans can't be fixed, and the more the constraints that are added on them, the higher the failure rate they will show. However, regardless of the detailed knowledge of how to format this or that tag, it should still be possible for any patch author to answer the question "what do you want us to do with that patch". If most of the patches at least contain such info, it already becomes possible to figure after it gets merged whether the intent was to get it backported or not. It's not trivial but NLP and code analysis definitely help on this. And there will always be some patches whose need for backporting will be detected after the merge. That's why I'm seeing a lot of value in the post-processing part, because for me trying to fix the input will have a very limited effect. And let's face it, if a patch series gets merged, it means that at some point someone understood what to do with it, so all the needed information was already there, given a certain context. The cost is thousands of brains needed to decode that. But with the improvements in language processing, we should at some point be able to release some brains with more-or-less similar results and try to lower the barrier to contribution by relaxing patch submission rules instead of adding more. My feeling is that it's what we should aim for given that the number of maintainers and contributors doesn't seem to grow as fast as the code size and complexity. That's where I'm seeing a lot of value in the AUTOSEL work. If it can show the direction to something better so that in 3 or 4 years we can think back and say "remember the garbage it was compared to what we have now", I think it will have been a fantastic success. > So I hope people can understand where I'm coming from, with hoping that what the > stable maintainers are doing can just be improved directly, without first > building something from scratch off to the side as that is just not a good way > to do things. It's generally not good but here there's little to start from, and experimenting with your ideas on LKML threads or commit series can be a nice way to explore completely different approaches without being limited by what currently exists. Maybe you'll end up with something completely orthogonal and the combination of both of your solutions will already be a nice improvement. who knows. > But sure, if that's the only option to get anything nontrivial > changed, I'll try to do it. Thanks! Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 5:21 ` Eric Biggers 2023-03-12 5:48 ` Willy Tarreau @ 2023-03-12 7:42 ` Amir Goldstein 2023-03-12 13:34 ` Mark Brown 2023-03-12 15:57 ` Sasha Levin 1 sibling, 2 replies; 83+ messages in thread From: Amir Goldstein @ 2023-03-12 7:42 UTC (permalink / raw) To: Eric Biggers Cc: Willy Tarreau, Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel, Konstantin Ryabitsev On Sun, Mar 12, 2023 at 7:41 AM Eric Biggers <ebiggers@kernel.org> wrote: > ... > I mean, "patches welcome" is a bit pointless when there is nothing to patch, is > it not? Even Sasha's stable-tools, which he finally gave a link to, does not > include anything related to AUTOSEL. It seems AUTOSEL is still closed source. > > BTW, I already did something similar "off to the side" a few years ago when I > wrote a script to keep track of and prioritize syzbot reports from > https://syzkaller.appspot.com/, and generate per-subsystem reminder emails. > > I eventually ended up abandoning that, because doing something off to the side > is not very effective and is hard to keep up with. The right approach is to > make improvements to the "upstream" process (which was syzbot in that case), not > to bolt something on to the side to try to fix it after the fact. > > So I hope people can understand where I'm coming from, with hoping that what the > stable maintainers are doing can just be improved directly, without first > building something from scratch off to the side as that is just not a good way > to do things. But sure, if that's the only option to get anything nontrivial > changed, I'll try to do it. > Eric, Did you consider working to improve or add functionality to b4? Some of the things that you proposed sound like a natural extension of b4 that stable maintainers could use if they turn out to be useful. Some of the things in your wish list, b4 already does - it has a local cache of messages from query results, it can find the latest submission of a patch series. When working on backporting xfs patches to LTS, I ended up trying to improve and extend b4 [1]. My motivation was to retrieve the information provided in the pull request and cover letters which often have much better context to understand whether patches are stable material. My motivation was NOT to improve automatic scripts, but I did make some improvement to b4 that could benefit automatic scripts as well. Alas, despite sending a pull request via github and advertising my work and its benefits on several occasions, I got no feedback from Konstantin nor from any other developers, so I did not pursue upstreaming. If you find any part of this work relevant, I can try to rebase and post my b4 patches. Thanks, Amir. [1] https://github.com/mricon/b4/pull/1 ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 7:42 ` Amir Goldstein @ 2023-03-12 13:34 ` Mark Brown 2023-03-12 15:57 ` Sasha Levin 1 sibling, 0 replies; 83+ messages in thread From: Mark Brown @ 2023-03-12 13:34 UTC (permalink / raw) To: Amir Goldstein Cc: Eric Biggers, Willy Tarreau, Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel, Konstantin Ryabitsev [-- Attachment #1: Type: text/plain, Size: 861 bytes --] On Sun, Mar 12, 2023 at 09:42:59AM +0200, Amir Goldstein wrote: > Alas, despite sending a pull request via github and advertising my work > and its benefits on several occasions, I got no feedback from Konstantin > nor from any other developers, so I did not pursue upstreaming. > If you find any part of this work relevant, I can try to rebase and > post my b4 patches. ... > [1] https://github.com/mricon/b4/pull/1 b4 development is mainly done via email on the tools@linux.kernel.org list, and https://git.kernel.org/pub/scm/utils/b4/b4.git rather than that github repository (note that yours is the first and only pull request there) is the main repo. I suspect that github repo is just an automatically maintained mirror and nobody's seen your pull request, you'd be much more likely to get a response sending your patches to the list CC Konstantin. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 7:42 ` Amir Goldstein 2023-03-12 13:34 ` Mark Brown @ 2023-03-12 15:57 ` Sasha Levin 1 sibling, 0 replies; 83+ messages in thread From: Sasha Levin @ 2023-03-12 15:57 UTC (permalink / raw) To: Amir Goldstein Cc: Eric Biggers, Willy Tarreau, Theodore Ts'o, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel, Konstantin Ryabitsev On Sun, Mar 12, 2023 at 09:42:59AM +0200, Amir Goldstein wrote: >On Sun, Mar 12, 2023 at 7:41 AM Eric Biggers <ebiggers@kernel.org> wrote: >> >... >> I mean, "patches welcome" is a bit pointless when there is nothing to patch, is >> it not? Even Sasha's stable-tools, which he finally gave a link to, does not "finally" being all the way back in 2015 (https://lkml.org/lkml/2015/11/9/422), and getting no contributions from 3rd parties for the next 7 years. Really, we're not pushing back on ideas, we're just saying that this is something has happened before ("open up your scripts so we can reuse/improve") and getting nowhere. >> include anything related to AUTOSEL. It seems AUTOSEL is still closed source. Not as much as closed source as a complete mess on a VM I'm afraid to lose. I didn't really want to try and invest the effort into extracting it out becuase: 1. It's one of the first things I did when I started learning about neural networks, and in practice it's inefficient and could use a massive overhaul (or rewrite). For example, it currently has ~15k inputs, which means that it needs a beefy GPU to be able to run on (it won't run on your home GPU).. 2. At that time I wasn't familiar with coding for NN either, so it's a mess of LUA code to run under Torch (yes, not even pytorch). 3. It's very tied to how VMs on Azure operate, and could use a lot of abstraction. So really, I'd much rather invest effort into rewriting this mess rather than digging it out of the crevices of the VM it's sitting in. >> BTW, I already did something similar "off to the side" a few years ago when I >> wrote a script to keep track of and prioritize syzbot reports from >> https://syzkaller.appspot.com/, and generate per-subsystem reminder emails. >> >> I eventually ended up abandoning that, because doing something off to the side >> is not very effective and is hard to keep up with. The right approach is to >> make improvements to the "upstream" process (which was syzbot in that case), not >> to bolt something on to the side to try to fix it after the fact. >> >> So I hope people can understand where I'm coming from, with hoping that what the >> stable maintainers are doing can just be improved directly, without first >> building something from scratch off to the side as that is just not a good way >> to do things. But sure, if that's the only option to get anything nontrivial >> changed, I'll try to do it. >> > >Eric, > >Did you consider working to improve or add functionality to b4? FTR, I'm happy to shift investment into tooling for stable maintenance into b4. -- Thanks, Sasha ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 20:11 ` Willy Tarreau 2023-03-11 20:53 ` Eric Biggers @ 2023-03-12 13:55 ` Mark Brown 1 sibling, 0 replies; 83+ messages in thread From: Mark Brown @ 2023-03-12 13:55 UTC (permalink / raw) To: Willy Tarreau Cc: Eric Biggers, Theodore Ts'o, Sasha Levin, Matthew Wilcox, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel [-- Attachment #1: Type: text/plain, Size: 3065 bytes --] On Sat, Mar 11, 2023 at 09:11:51PM +0100, Willy Tarreau wrote: > On Sat, Mar 11, 2023 at 11:24:31AM -0800, Eric Biggers wrote: > > As I said in a part of my email which you did not quote, the fallback option is > > to send the list of issues to the mailing list for others to review. > Honestly, patches are already being delivered publicly tagged AUTOSEL, > then published again as part of the stable review process. Have you seen > the amount of feedback ? Once in a while there are responses, but aside > Guenter reporting build successes or failures, it's a bit quiet. What > makes you think that sending more detailed stuff that require even more > involvement and decision would trigger more participation ? TBH as someone getting copied on the AUTOSEL mails I think if the volume of backports is going to say the same what I'd really like is something that mitigates the volume of mail, or at least makes the mails that are being sent more readily parseable. Things that add more context to what's being sent would probably help a lot, right now I'm not really able to do much more than scan for obviously harmful things. > > But again, this comes back to one of the core issues here which is how does one > > even build something for the stable maintainers if their requirements are > > unknown to others? > Well, the description of the commit message is there for anyone to > consume in the first place. A commit message is an argument for a > patch to get adopted and resist any temptations to revert it. So > it must be descriptive enough and give instructions. Dependencies > should be clear there. When you seen github-like one-liners there's > no hope to get much info, and it's not a matter of requirements, > but of respect for a team development process where some facing your > patch might miss the skills required to grasp the details. With a > sufficiently clear commit message, even a bot can find (or suggest) > dependencies. And this is not specific to -stable: if one of the > dependencies is found to break stuff, how do you know it must not be > reverted without reverting the whole series if that's not described > anywhere ? I'd say that the most common class of missing dependency I've seen is on previously applied code which is much less likely to be apparent in the commit message and probably not noticed unless it causes a cherry pick or build issue. > One thing I think that could be within reach and could very slightly > improve the process would be to indicate in a stable announce the amount > of patches coming from autosel. I think that it could help either > refining the selection by making users more conscious about the importance > of this source, or encourage more developers to Cc stable to reduce that > ratio. Just an idea. I'm not sure if it's the ratio that's the issue here, if anything I'd expect that lowering the ratio would make people more stressed by AUTOSEL since assuming a similar volume of patches get picked overall it would increase the percentage of the AUTOSEL patches that have problems. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
* RE: AUTOSEL process 2023-03-11 14:06 ` Sasha Levin 2023-03-11 16:16 ` Theodore Ts'o @ 2023-03-11 22:38 ` David Laight 2023-03-12 4:41 ` Willy Tarreau 1 sibling, 1 reply; 83+ messages in thread From: David Laight @ 2023-03-11 22:38 UTC (permalink / raw) To: 'Sasha Levin', Matthew Wilcox Cc: Eric Biggers, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel ... > I suppose that if I had a way to know if a certain a commit is part of a > series, I could either take all of it or none of it, but I don't think I > have a way of doing that by looking at a commit in Linus' tree > (suggestions welcome, I'm happy to implement them). Could something in git (eg when patches get applied) add dependencies between the patches in a series. While that won't force the entire series be added, it would ensure that all earlier patches are added. David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-11 22:38 ` David Laight @ 2023-03-12 4:41 ` Willy Tarreau 2023-03-12 5:09 ` Theodore Ts'o 0 siblings, 1 reply; 83+ messages in thread From: Willy Tarreau @ 2023-03-12 4:41 UTC (permalink / raw) To: David Laight Cc: 'Sasha Levin', Matthew Wilcox, Eric Biggers, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sat, Mar 11, 2023 at 10:38:10PM +0000, David Laight wrote: > ... > > I suppose that if I had a way to know if a certain a commit is part of a > > series, I could either take all of it or none of it, but I don't think I > > have a way of doing that by looking at a commit in Linus' tree > > (suggestions welcome, I'm happy to implement them). > > Could something in git (eg when patches get applied) add dependencies > between the patches in a series. > While that won't force the entire series be added, it would ensure > that all earlier patches are added. I suspect that having an option to keep the message ID in the footer (a bit like the "cherry-picked from" tag but instead "blongs to series") could possibly help. And when no such info is present we could have one ID generated per "git am" execution since usually if you apply an mbox, it constitutes a series (but not always of course, though it's not difficult to arrange series like this). Willy ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 4:41 ` Willy Tarreau @ 2023-03-12 5:09 ` Theodore Ts'o 2023-03-14 14:12 ` Jan Kara 0 siblings, 1 reply; 83+ messages in thread From: Theodore Ts'o @ 2023-03-12 5:09 UTC (permalink / raw) To: Willy Tarreau Cc: David Laight, 'Sasha Levin', Matthew Wilcox, Eric Biggers, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sun, Mar 12, 2023 at 05:41:07AM +0100, Willy Tarreau wrote: > > I suspect that having an option to keep the message ID in the footer (a > bit like the "cherry-picked from" tag but instead "blongs to series") > could possibly help. And when no such info is present we could have > one ID generated per "git am" execution since usually if you apply an > mbox, it constitutes a series (but not always of course, though it's > not difficult to arrange series like this). As I pointed out earlier, some of us are adding the message ID in the footer alrady, using a Link tag. This is even documented already in the Kernel Maintainer's Handbook, so I'm pretty sure it's not just me. :-) https://www.kernel.org/doc/html/latest/maintainer/configure-git.html#creating-commit-links-to-lore-kernel-org This is quite sufficient to extract out the full patch series given a particular patch. The b4 python script does this this; given a single Message-Id, it can find all of the other patches in the series. I won't say that it it's "trivial", but the code already exists, and you can copy and paste it from b4. Or just have your script shell out to "b4 am -o /tmp/scratchdir $MSGID" - Ted ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-03-12 5:09 ` Theodore Ts'o @ 2023-03-14 14:12 ` Jan Kara 0 siblings, 0 replies; 83+ messages in thread From: Jan Kara @ 2023-03-14 14:12 UTC (permalink / raw) To: Theodore Ts'o Cc: Willy Tarreau, David Laight, 'Sasha Levin', Matthew Wilcox, Eric Biggers, Pavel Machek, linux-kernel, stable, viro, linux-fsdevel On Sun 12-03-23 00:09:47, Theodore Ts'o wrote: > On Sun, Mar 12, 2023 at 05:41:07AM +0100, Willy Tarreau wrote: > > > > I suspect that having an option to keep the message ID in the footer (a > > bit like the "cherry-picked from" tag but instead "blongs to series") > > could possibly help. And when no such info is present we could have > > one ID generated per "git am" execution since usually if you apply an > > mbox, it constitutes a series (but not always of course, though it's > > not difficult to arrange series like this). > > As I pointed out earlier, some of us are adding the message ID in the > footer alrady, using a Link tag. This is even documented already in > the Kernel Maintainer's Handbook, so I'm pretty sure it's not just me. :-) Yeah, given Linus' rants about links pointing to patch posting, what I'm currently doing is that I add Message-ID: tag to the patch instead of a Link: tag. It preserves the information as well and it is obvious to human reader what are links to reports / discussions and what is just a link to patch posting. Honza -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 83+ messages in thread
* Re: AUTOSEL process 2023-02-27 17:47 ` AUTOSEL process Eric Biggers 2023-02-27 18:06 ` Eric Biggers @ 2023-03-13 3:37 ` Bagas Sanjaya 1 sibling, 0 replies; 83+ messages in thread From: Bagas Sanjaya @ 2023-03-13 3:37 UTC (permalink / raw) To: Eric Biggers, Sasha Levin Cc: linux-kernel, stable, viro, linux-fsdevel, John Harrison, intel-gfx [-- Attachment #1: Type: text/plain, Size: 4251 bytes --] On Mon, Feb 27, 2023 at 09:47:46AM -0800, Eric Biggers wrote: > > > One of the challanges here is that it's difficult to solicit reviews or > > really any interaction from authors after a commit lands upstream. Look > > at the response rates to Greg's "FAILED" emails that ask authors to > > provide backports to commits they tagged for stable. > > Well, it doesn't help that most of the stable emails aren't sent to the > subsystem's mailing list, but instead just to the individual people mentioned in > the commit. So many people who would like to help never know about it. Let me joining in the discussion. In this case, Greg send these FAILED emails to upstream commit authors, but some of them have expectation that their commits can be backported automatically (without conflicts), so when Greg ask them to "manually" do the backport (and with resolving conflicts between), they have little to no idea on what should they do. That's why there was a doc patch sent [1] specifically on this matter. Fortunately, some others are aware on this and send the backport. > I don't know, is it obvious? You've said in the past that sometimes you'd like > to backport a commit even if the maintainer objects and/or it is known buggy. > https://lore.kernel.org/stable/d91aaff1-470f-cfdf-41cf-031eea9d6aca@mailbox.org > also didn't explicitly say "Don't backport this" but instead "This patch has > issues", so maybe that made a difference? > > Anyway, the fact is that it happened. And if it happened in the one bug that I > happened to look at because it personally affected me and I spent hours > bisecting, it probably is happening in lots of other cases too. So it seems the > process is not working... > > Separately from responses to the AUTOSEL email, it also seems that you aren't > checking for any reported regressions or pending fixes for a commit before > backporting it. Simply searching lore for the commit title > https://lore.kernel.org/all/?q=%22drm%2Famdgpu%3A+use+dirty+framebuffer+helper%22 > would have turned up the bug report > https://lore.kernel.org/dri-devel/20220918120926.10322-1-user@am64/ that > bisected a regression to that commit, as well as a patch that Fixes that commit: > https://lore.kernel.org/all/20220920130832.2214101-1-alexander.deucher@amd.com/ > Both of these existed before you even sent the AUTOSEL email! > > So to summarize, that buggy commit was backported even though: > > * There were no indications that it was a bug fix (and thus potentially > suitable for stable) in the first place. > * On the AUTOSEL thread, someone told you the commit is broken. > * There was already a thread that reported a regression caused by the commit. > Easily findable via lore search. > * There was also already a pending patch that Fixes the commit. Again easily > findable via lore search. Recently there was a regression in linux-5.15.y that was caused by faulty backport of upstream 85636167e3206c ("drm/i915: Don't use BAR mappings for ring buffers with LLC") as 4eb6789f9177a5 ("drm/i915: Don't use BAR mappings for ring buffers with LLC"). Fortunately, the upstream commit wasn't AUTOSEL'd by Sasha's scripts and the culprit backport got reverted. [CC'ed the upstream commit author and corresponding ML.] > > So it seems a *lot* of things went wrong, no? Why? If so many things can go > wrong, it's not just a "mistake" but rather the process is the problem... Testing the backports are also important, hence before a stable release is made, there is -rc review when testers can test linux-x.y trees as published in linux-stable-rc.git tree. The testing quality depends on how many testers test and send their report, as well as the type of test. For example, I do basic cross-compile test for arm64 and ppc64. Did they spot the regression I mentioned earlier? Nope, until after the release, where production users reported the regression. If they do, they would have replied to appropriate backported patch that caused the problem. Thanks. [1]: https://lore.kernel.org/linux-doc/20230303162553.17212-1-vegard.nossum@oracle.com/ -- An old man doll... just what I always wanted! - Clara [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 228 bytes --] ^ permalink raw reply [flat|nested] 83+ messages in thread
end of thread, other threads:[~2023-03-30 17:50 UTC | newest] Thread overview: 83+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <20230226034256.771769-1-sashal@kernel.org> 2023-02-26 3:42 ` [PATCH AUTOSEL 6.1 07/21] fs: Use CHECK_DATA_CORRUPTION() when kernel bugs are detected Sasha Levin 2023-02-26 3:42 ` [PATCH AUTOSEL 6.1 12/21] fs/super.c: stop calling fscrypt_destroy_keyring() from __put_super() Sasha Levin 2023-02-26 4:07 ` Eric Biggers 2023-02-26 5:30 ` Eric Biggers 2023-02-26 19:24 ` Eric Biggers 2023-02-26 19:33 ` Slade Watkins 2023-02-27 14:18 ` Sasha Levin 2023-02-27 17:47 ` AUTOSEL process Eric Biggers 2023-02-27 18:06 ` Eric Biggers 2023-02-27 20:39 ` Sasha Levin 2023-02-27 21:38 ` Eric Biggers 2023-02-27 22:35 ` Sasha Levin 2023-02-27 22:59 ` Matthew Wilcox 2023-02-28 0:52 ` Sasha Levin 2023-02-28 1:25 ` Eric Biggers 2023-02-28 4:25 ` Willy Tarreau 2023-03-30 0:08 ` Eric Biggers 2023-03-30 14:05 ` Sasha Levin 2023-03-30 17:22 ` Eric Biggers 2023-03-30 17:50 ` Sasha Levin 2023-02-28 0:32 ` Eric Biggers 2023-02-28 1:53 ` Sasha Levin 2023-02-28 3:41 ` Eric Biggers 2023-02-28 10:41 ` Amir Goldstein 2023-02-28 11:28 ` Greg KH 2023-03-01 2:05 ` Slade Watkins 2023-03-01 5:13 ` Eric Biggers 2023-03-01 6:09 ` Greg KH 2023-03-01 7:22 ` Eric Biggers 2023-03-01 7:40 ` Willy Tarreau 2023-03-01 8:31 ` Eric Biggers 2023-03-01 8:43 ` Greg KH 2023-03-01 6:06 ` Greg KH 2023-03-01 7:05 ` Eric Biggers 2023-03-01 10:31 ` Thorsten Leemhuis 2023-03-01 13:26 ` Mark Brown 2023-02-28 17:03 ` Sasha Levin 2023-03-10 23:07 ` Eric Biggers 2023-03-11 13:41 ` Sasha Levin 2023-03-11 15:54 ` James Bottomley 2023-03-11 18:07 ` Sasha Levin 2023-03-12 19:03 ` Theodore Ts'o 2023-03-07 21:18 ` Pavel Machek 2023-03-07 21:45 ` Eric Biggers 2023-03-11 6:25 ` Matthew Wilcox 2023-03-11 8:11 ` Willy Tarreau 2023-03-11 11:45 ` Pavel Machek 2023-03-11 12:29 ` Greg KH 2023-03-21 12:41 ` Maciej W. Rozycki 2023-03-11 14:06 ` Sasha Levin 2023-03-11 16:16 ` Theodore Ts'o 2023-03-11 17:48 ` Eric Biggers 2023-03-11 18:26 ` Sasha Levin 2023-03-11 18:54 ` Eric Biggers 2023-03-11 19:01 ` Eric Biggers 2023-03-11 21:14 ` Sasha Levin 2023-03-12 8:04 ` Amir Goldstein 2023-03-12 16:00 ` Sasha Levin 2023-03-13 17:41 ` Greg KH 2023-03-13 18:54 ` Eric Biggers 2023-03-14 18:26 ` Greg KH 2023-03-11 20:17 ` Eric Biggers 2023-03-11 21:02 ` Sasha Levin 2023-03-12 4:23 ` Willy Tarreau 2023-03-11 18:33 ` Willy Tarreau 2023-03-11 19:24 ` Eric Biggers 2023-03-11 19:46 ` Eric Biggers 2023-03-11 20:19 ` Willy Tarreau 2023-03-11 20:59 ` Sasha Levin 2023-03-11 20:11 ` Willy Tarreau 2023-03-11 20:53 ` Eric Biggers 2023-03-12 4:32 ` Willy Tarreau 2023-03-12 5:21 ` Eric Biggers 2023-03-12 5:48 ` Willy Tarreau 2023-03-12 7:42 ` Amir Goldstein 2023-03-12 13:34 ` Mark Brown 2023-03-12 15:57 ` Sasha Levin 2023-03-12 13:55 ` Mark Brown 2023-03-11 22:38 ` David Laight 2023-03-12 4:41 ` Willy Tarreau 2023-03-12 5:09 ` Theodore Ts'o 2023-03-14 14:12 ` Jan Kara 2023-03-13 3:37 ` Bagas Sanjaya
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).