RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Thorsten Leemhuis]

Lo! I want to provide users with an easier way to search our multitude
of mailing lists for reports about issues (aka bugs), as reporting the
same kernel problem multiple times has known downsides for everyone
involved. That's why I propose to create this new mailing list:

linux-issues@lists.linux.dev

Developers and users reporting or handling issues then can CC it or
search it via lore. But this will only fly if the idea has buy-in from
at least the core kernel maintainers, to make sure they and the
developers actually use it. That's why I'm looking for feedback with
this mail and also CCed ksummit-discuss, as that's the easiest way to
make sure maintainers get aware of this idea and can raise their voice.


Note, there is a second reason why ksummit-discuss is CCed: another
reason why I want to create this new list is making it easier to find
and track regressions reported to our various mailing lists (often
without LKML in CC, as for some subsystems it's seems to be custom to
not CC it). Back on the maintainers summit in 2017 it was agreed to
create a dedicated list for this purpose
(https://lwn.net/Articles/738216/). I even requested a
"linux-regressions@vger.kernel.org" a while later, but didn't hear
anything back; and, sadly, about the same time I started having trouble
finding spare time for working on regression tracking. :-/

But I soon will get back into that area:
https://linux-regtracking.leemhuis.info/post/hello-world/ Hence it's a
good time to prepare some groundwork for that. But these days I think
having something like linux-regressions@lists.linux.dev might be over
engineered, at least for now: a linux-issues@lists.linux.dev with a
simple "[regressions]" in the subject will suffice, as that tag is
something a lot of people are used to already. And if we think we need
that list we can still create it in the future. Or what do you folks
think about it?



We can obviously bikeshed about the name for the list. I'm sure some
people will prefer to use "bugs" instead of "issues" there. I propose
"issues" for now, because the new text I've written about reporting
kernels issues/bugs uses the word "issues" in the filename, the title,
and the body while avoiding "bug" (see
Documentation/admin-guide/reporting-issues.rst or
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
). I chose this approach as users are dealing with issues that might or
might not be bugs in the kernel. We discussed this before above text was
merged, but in the end stayed with issues:
https://lore.kernel.org/linux-doc/b5f5dfad-07bb-b518-0dff-3aa340333046@infradead.org/
BTW, creating this list will partly solve the second of the "FIXME
warning boxes" currently left in that text (two others are solved by
patches that are under review currently).


The question "Why not simply LKML" will likely pop up, but the thing is:
searching for reports there will often turn up patches that improve the
kernel and don't fix anything. That makes it hard to find issue reports,
especially for users that are not used to deal with mailing lists and
their archives.

And yes, I'm quite aware that searching linux-issues@lists.linux.dev
list obviously won't turn up reports that are filed in
bugzilla.kernel.org or some other bug tracking tool. That's okay, as the
reporting-issues.rst tells users to look in those places as well.

Another "and yes, I'm quite aware" note: sure reporting issues/bugs by
mail has downsides and maybe instead of creating yet another mailing
list it would be better if all the kernel issues would be reported to a
central place like bugzilla.kernel.org. But that tracker doesn't work
that well currently, as quite a few of the issues filed there afaics
never reach the people that need to be handle them. I don't see that
changing any time soon (we had a discussion about this recently:
https://lore.kernel.org/linux-doc/20210111194822.4kvl2tx24anyu23k@chatter.i7.local/
).

Creating a new mailing list for issues OTOH is something that can be
done quickly and easily to improve the situation without too much
hassle. That's why that's my plan currently, unless the discussion that
hopefully evolved due to this mail leads to something better. :-D

Ciao, Thorsten

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Lukas Bulwahn]

On Mon, Mar 22, 2021 at 4:38 PM Thorsten Leemhuis <linux@leemhuis.info> wrote:
>
>
> Lo! I want to provide users with an easier way to search our multitude
> of mailing lists for reports about issues (aka bugs), as reporting the
> same kernel problem multiple times has known downsides for everyone
> involved. That's why I propose to create this new mailing list:
>
> linux-issues@lists.linux.dev
>
> Developers and users reporting or handling issues then can CC it or
> search it via lore. But this will only fly if the idea has buy-in from
> at least the core kernel maintainers, to make sure they and the
> developers actually use it. That's why I'm looking for feedback with
> this mail and also CCed ksummit-discuss, as that's the easiest way to
> make sure maintainers get aware of this idea and can raise their voice.
>
>
> Note, there is a second reason why ksummit-discuss is CCed: another
> reason why I want to create this new list is making it easier to find
> and track regressions reported to our various mailing lists (often
> without LKML in CC, as for some subsystems it's seems to be custom to
> not CC it). Back on the maintainers summit in 2017 it was agreed to
> create a dedicated list for this purpose
> (https://lwn.net/Articles/738216/). I even requested a
> "linux-regressions@vger.kernel.org" a while later, but didn't hear
> anything back; and, sadly, about the same time I started having trouble
> finding spare time for working on regression tracking. :-/
>
> But I soon will get back into that area:
> https://linux-regtracking.leemhuis.info/post/hello-world/ Hence it's a
> good time to prepare some groundwork for that. But these days I think
> having something like linux-regressions@lists.linux.dev might be over
> engineered, at least for now: a linux-issues@lists.linux.dev with a
> simple "[regressions]" in the subject will suffice, as that tag is
> something a lot of people are used to already. And if we think we need
> that list we can still create it in the future. Or what do you folks
> think about it?
>

Thorsten, I generally support this initiative. I am just wondering:

What is the definition of an issue for you?

Just four examples that come to my mind:

- all the warnings that Stephen Rothwell reports on linux-next, such
as https://lore.kernel.org/linux-next/20210322170452.726525e8@canb.auug.org.au/
or https://lore.kernel.org/linux-next/20210322075108.3967d5d1@canb.auug.org.au/?
- all the issues that the kernel test robot reports?
- all the errors and warnings that kernel ci reports? Basically, each
"issue" that is already aggregated in this email,
https://lore.kernel.org/linux-next/6058787c.1c69fb81.d3f10.8c6d@mx.google.com/?
- all the syzbot reports?

Are you including all those automatic testing and checking efforts as
reporting valid "issues"? Or would you like to keep this list only for
reports from single individual human users that need to detect the
"issue" without using one of the tools above?

Lukas

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Konstantin Ryabitsev]

On Mon, Mar 22, 2021 at 04:18:14PM +0100, Thorsten Leemhuis wrote:
> Note, there is a second reason why ksummit-discuss is CCed: another
> reason why I want to create this new list is making it easier to find
> and track regressions reported to our various mailing lists (often
> without LKML in CC, as for some subsystems it's seems to be custom to
> not CC it). 

FYI, there will soon be a unified "search all of lore.kernel.org regardless of
the list/feed source" capability that may make it unnecessary to create a
separate list for this purpose. There's active ongoing work in the
public-inbox project to provide parallel ways to follow aggregate topics,
including query-based subscriptions (i.e. "put a thread into my inbox whenever
someone mentions my favourite file/function/device name"). This work is not
complete yet, but I have great hopes that it will become available in the next
little while.

Once we have this ability, we should be able to plug in multiple sources
beyond just mailing lists, including a feed of all bugzilla.kernel.org
changes. This should allow someone an easy way to query specific bugs and may
not require the creation of a separate list.

I'm not opposed to the creation of a new list, of course -- just want to make
sure it's aligned with the improvements we are working to make available.

-K

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[James Bottomley]

On Mon, 2021-03-22 at 13:16 -0400, Konstantin Ryabitsev wrote:
> On Mon, Mar 22, 2021 at 04:18:14PM +0100, Thorsten Leemhuis wrote:
> > Note, there is a second reason why ksummit-discuss is CCed: another
> > reason why I want to create this new list is making it easier to
> > find and track regressions reported to our various mailing lists
> > (often without LKML in CC, as for some subsystems it's seems to be
> > custom to not CC it). 
> 
> FYI, there will soon be a unified "search all of lore.kernel.org
> regardless of the list/feed source" capability that may make it
> unnecessary to create a separate list for this purpose. There's
> active ongoing work in the public-inbox project to provide parallel
> ways to follow aggregate topics, including query-based subscriptions
> (i.e. "put a thread into my inbox whenever someone mentions my
> favourite file/function/device name"). This work is not complete yet,
> but I have great hopes that it will become available in the next
> little while.
> 
> Once we have this ability, we should be able to plug in multiple
> sources beyond just mailing lists, including a feed of all
> bugzilla.kernel.org changes. This should allow someone an easy way to
> query specific bugs and may not require the creation of a separate
> list.
> 
> I'm not opposed to the creation of a new list, of course -- just want
> to make sure it's aligned with the improvements we are working to
> make available.

I suspect the problem is that there's no known useful search string to
find a bug report even given a searchable set of lists, so the main
purpose of this list would be "if it's on here, it's a bug report" and
the triage team can cc additional lists as appropriate.  Then we simply
tell everyone to send kernel bugs to this list and ask maintainers to
cc it if a bug report shows up on their list?

James

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Linus Torvalds]

On Mon, Mar 22, 2021 at 8:18 AM Thorsten Leemhuis <linux@leemhuis.info> wrote:
>
>     I even requested a
> "linux-regressions@vger.kernel.org" a while later, but didn't hear
> anything back; and, sadly, about the same time I started having trouble
> finding spare time for working on regression tracking. :-/

Honestly, I'd much prefer the name 'linux-regressions' as being much
more targeted than 'linux-issues'. Make it clear that the list is only
for regressions that people can describe some way, rather than some
general "I have issues with xyz".

The more clear-cut the list is, the better, I think.

            Linus

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Eric Wong]

James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> On Mon, 2021-03-22 at 13:16 -0400, Konstantin Ryabitsev wrote:
> > On Mon, Mar 22, 2021 at 04:18:14PM +0100, Thorsten Leemhuis wrote:
> > > Note, there is a second reason why ksummit-discuss is CCed: another
> > > reason why I want to create this new list is making it easier to
> > > find and track regressions reported to our various mailing lists
> > > (often without LKML in CC, as for some subsystems it's seems to be
> > > custom to not CC it). 
> > 
> > FYI, there will soon be a unified "search all of lore.kernel.org
> > regardless of the list/feed source" capability that may make it
> > unnecessary to create a separate list for this purpose. There's
> > active ongoing work in the public-inbox project to provide parallel
> > ways to follow aggregate topics, including query-based subscriptions
> > (i.e. "put a thread into my inbox whenever someone mentions my
> > favourite file/function/device name"). This work is not complete yet,
> > but I have great hopes that it will become available in the next
> > little while.

Yes, making progress and learning new tricks to make the WWW UI faster :>

> > Once we have this ability, we should be able to plug in multiple
> > sources beyond just mailing lists, including a feed of all
> > bugzilla.kernel.org changes. This should allow someone an easy way to
> > query specific bugs and may not require the creation of a separate
> > list.
> > 
> > I'm not opposed to the creation of a new list, of course -- just want
> > to make sure it's aligned with the improvements we are working to
> > make available.
> 
> I suspect the problem is that there's no known useful search string to
> find a bug report even given a searchable set of lists, so the main
> purpose of this list would be "if it's on here, it's a bug report" and
> the triage team can cc additional lists as appropriate.  Then we simply
> tell everyone to send kernel bugs to this list and ask maintainers to
> cc it if a bug report shows up on their list?

It seems having "bug" or "regression" in the subject could be sufficient?

"s:Regression" or "s:Bug" can be used to query messages reasonably
quickly:

https://80x24.org/lore/all/?q=s:Bug || https://yhbt.net/lore/all/?q=s:Bug
http://rskvuqcfnfizkjg6h5jvovwb3wkikzcwskf54lfpymus6mxrzw67b5ad.onion/all/?q=s:Bug

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Thorsten Leemhuis]

On 22.03.21 19:34, Eric Wong wrote:
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
>> On Mon, 2021-03-22 at 13:16 -0400, Konstantin Ryabitsev wrote:
>>> On Mon, Mar 22, 2021 at 04:18:14PM +0100, Thorsten Leemhuis wrote:
>>>> Note, there is a second reason why ksummit-discuss is CCed: another
>>>> reason why I want to create this new list is making it easier to
>>>> find and track regressions reported to our various mailing lists
>>>> (often without LKML in CC, as for some subsystems it's seems to be
>>>> custom to not CC it). 
>>>
>>> FYI, there will soon be a unified "search all of lore.kernel.org
>>> regardless of the list/feed source" capability

Ahh, nice, thx to everyone working on that!

> [...]
>>> Once we have this ability, we should be able to plug in multiple
>>> sources beyond just mailing lists, including a feed of all
>>> bugzilla.kernel.org changes.

Out of curiosity: will that work for other bug trackers as well? Like
the gitlab instance used by the drm developers? It's not really
important and I guess the answer will be "no", but the question came up
while at it...

>>> This should allow someone an easy way to
>>> query specific bugs and may not require the creation of a separate
>>> list.
>>>
>>> I'm not opposed to the creation of a new list, of course -- just want
>>> to make sure it's aligned with the improvements we are working to
>>> make available.
>>
>> I suspect the problem is that there's no known useful search string to
>> find a bug report even given a searchable set of lists,

Exactly. Due to my work on reporting-issues.rst I try to look at it from
the users point of view. And they currently have no easy way to search
for existing reports without getting lots of other stuff mixed into the
results they are not interested in. That makes it hard. :-/

>> so the main
>> purpose of this list would be "if it's on here, it's a bug report" and
>> the triage team

If one exists ;-)

>> can cc additional lists as appropriate.  Then we simply
>> tell everyone to send kernel bugs to this list and ask maintainers to
>> cc it if a bug report shows up on their list?
> 
> It seems having "bug" or "regression" in the subject could be sufficient?
>
> "s:Regression" or "s:Bug" can be used to query messages reasonably
> quickly:

Could, but I fear it might fail, as modifying the subject is a little
unusual to the normal working style; but "adding people and appropriate
mailing lists to the CC" OTOH is something that people do every day.

And that's why I still think having a separate list is the best idea.
But using tags is totally fine for me, if that the general consensus.

Ciao, Thorsten

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Konstantin Ryabitsev]

On Mon, Mar 22, 2021 at 07:55:29PM +0100, Thorsten Leemhuis wrote:
> Out of curiosity: will that work for other bug trackers as well? Like
> the gitlab instance used by the drm developers? It's not really
> important and I guess the answer will be "no", but the question came up
> while at it...

If there's an API to poll changes, then it can be easily turned into a
public-inbox "feed" -- same as I intend to do for bugzilla. Each entry simply
gets converted into an RFC2822 message and stored in the repository that can
be indexed and queried by public-inbox alongside any other source.

-K

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Thorsten Leemhuis]

On 22.03.21 19:32, Linus Torvalds wrote:
> On Mon, Mar 22, 2021 at 8:18 AM Thorsten Leemhuis <linux@leemhuis.info> wrote:
>>
>>     I even requested a
>> "linux-regressions@vger.kernel.org" a while later, but didn't hear
>> anything back; and, sadly, about the same time I started having trouble
>> finding spare time for working on regression tracking. :-/
>
> Honestly, I'd much prefer the name 'linux-regressions' as being much
> more targeted than 'linux-issues'.

That only solves one of the two problem I'm trying to solve (albeit the
one that is more important to me). That way users still have no easy way
to query for reports about issues that are no regressions – say
something is broken and they have no idea if it once worked or never
worked at all.

> Make it clear that the list is only
> for regressions that people can describe some way, rather than some
> general "I have issues with xyz".
> 
> The more clear-cut the list is, the better, I think.

I agree to the last point and yeah, maybe regressions are the more
important problem we should work on – at least from the perspective of
kernel development.  But from the users perspective (and
reporting-issues.rst is written for that perspective) it feel a bit
unsatisfying to not have a solution to query for existing report,
regressions or not. Hmmmm...

Ciao, Thorsten

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Thorsten Leemhuis]

On 22.03.21 17:55, Lukas Bulwahn wrote:
> On Mon, Mar 22, 2021 at 4:38 PM Thorsten Leemhuis <linux@leemhuis.info> wrote:
>> Lo! I want to provide users with an easier way to search our multitude
>> of mailing lists for reports about issues (aka bugs), as reporting the
>> same kernel problem multiple times has known downsides for everyone
>> involved. That's why I propose to create this new mailing list:
>>
>> linux-issues@lists.linux.dev
>>
>> Developers and users reporting or handling issues then can CC it or
>> search it via lore. But this will only fly if the idea has buy-in from
>> at least the core kernel maintainers, to make sure they and the
>> developers actually use it. That's why I'm looking for feedback with
>> this mail and also CCed ksummit-discuss, as that's the easiest way to
>> make sure maintainers get aware of this idea and can raise their voice.
>>
>>
>> Note, there is a second reason why ksummit-discuss is CCed: another
>> reason why I want to create this new list is making it easier to find
>> and track regressions reported to our various mailing lists (often
>> without LKML in CC, as for some subsystems it's seems to be custom to
>> not CC it). Back on the maintainers summit in 2017 it was agreed to
>> create a dedicated list for this purpose
>> (https://lwn.net/Articles/738216/). I even requested a
>> "linux-regressions@vger.kernel.org" a while later, but didn't hear
>> anything back; and, sadly, about the same time I started having trouble
>> finding spare time for working on regression tracking. :-/
>>
>> But I soon will get back into that area:
>> https://linux-regtracking.leemhuis.info/post/hello-world/ Hence it's a
>> good time to prepare some groundwork for that. But these days I think
>> having something like linux-regressions@lists.linux.dev might be over
>> engineered, at least for now: a linux-issues@lists.linux.dev with a
>> simple "[regressions]" in the subject will suffice, as that tag is
>> something a lot of people are used to already. And if we think we need
>> that list we can still create it in the future. Or what do you folks
>> think about it?
> 
> Thorsten, I generally support this initiative. I am just wondering:
> 
> What is the definition of an issue for you?

Good question, but that is up to us. But FWIW, here is my stance:

> Just four examples that come to my mind:
> 
> - all the warnings that Stephen Rothwell reports on linux-next, such
> as https://lore.kernel.org/linux-next/20210322170452.726525e8@canb.auug.org.au/
> or https://lore.kernel.org/linux-next/20210322075108.3967d5d1@canb.auug.org.au/?

I'd have no problem with those, but those are more devel internal issues
and nothing users would face, so I'm don't think it will buy us anything.

> - all the issues that the kernel test robot reports?

I'd say so, at least those that are regressions, as I want to track them ;-)

> - all the errors and warnings that kernel ci reports? Basically, each
> "issue" that is already aggregated in this email,
> https://lore.kernel.org/linux-next/6058787c.1c69fb81.d3f10.8c6d@mx.google.com/?
> - all the syzbot reports?
>
> Are you including all those automatic testing and checking efforts as
> reporting valid "issues"?

Well, the purpose of that list would be "make it easier to find existing
reports". Too much noise works against that, so we should try to limit
it, so maybe those are better left out, unless it's something we known
users might face.

> Or would you like to keep this list only for
> reports from single individual human users that need to detect the
> "issue" without using one of the tools above?

Guess something like that might be best strategy.

Ciao, Thorsten

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Theodore Ts'o]

On Mon, Mar 22, 2021 at 08:25:15PM +0100, Thorsten Leemhuis wrote:
> I agree to the last point and yeah, maybe regressions are the more
> important problem we should work on – at least from the perspective of
> kernel development.  But from the users perspective (and
> reporting-issues.rst is written for that perspective) it feel a bit
> unsatisfying to not have a solution to query for existing report,
> regressions or not. Hmmmm...

First of all, thanks for working on reporting-issues.rst.  If we can
actually get users to *read* it, I think it's going to save kernel
developers a huge amount of time and frustration.

I wonder if it might be useful to have a form which users could be
encouraged to fill out so that (a) the information is available in a
structured format so it's easier for developers to find the relevant
information, (b) so it is easier for programs to parse, for easier
reporting or indexing, and (c) as a nudge so that users remember to
report critical bits of information such as the hardware
configuration, the exact kernel version, which distribution userspace
was in use, etc.

There could also be something in the text form which would make it
easier for lore.kernel.org searches to identify bug reports.  (e.g.,
"LINUX KERNEL BUG REPORTER TEMPLATE")

					- Ted

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Thorsten Leemhuis]

On 22.03.21 22:56, Theodore Ts'o wrote:
> On Mon, Mar 22, 2021 at 08:25:15PM +0100, Thorsten Leemhuis wrote:
>> I agree to the last point and yeah, maybe regressions are the more
>> important problem we should work on – at least from the perspective of
>> kernel development.  But from the users perspective (and
>> reporting-issues.rst is written for that perspective) it feel a bit
>> unsatisfying to not have a solution to query for existing report,
>> regressions or not. Hmmmm...
> First of all, thanks for working on reporting-issues.rst.

Thx, very glad to hear that. I didn't get much feedback on it, which
made me wonder if anybody besides docs folks actually looked at it...

>  If we can
> actually get users to *read* it, I think it's going to save kernel
> developers a huge amount of time and frustration.

And users hopefully as well. But yes, making them read it is the
problem. :-/

> I wonder if it might be useful to have a form which users could be
> encouraged to fill out so that (a) the information is available in a
> structured format so it's easier for developers to find the relevant
> information, (b) so it is easier for programs to parse, for easier
> reporting or indexing, and (c) as a nudge so that users remember to
> report critical bits of information such as the hardware
> configuration, the exact kernel version, which distribution userspace
> was in use, etc.
> 
> There could also be something in the text form which would make it
> easier for lore.kernel.org searches to identify bug reports.  (e.g.,
> "LINUX KERNEL BUG REPORTER TEMPLATE")

Hmmm, yeah, I like that idea. I'll keep it in mind for later: I would
prefer to get reporting-issues.rst officially blessed and
reporting-bugs.rst gone before working on further enhancements.

Maybe the best idea would be to have a script and/or webpage that helps
people creating the proper text form (which they then could simply
copy-and-paste into their mailer). I had such a script/webpage in mind
already to help with one of the IMHO biggest pain points when it comes
to reporting issues: finding where the report needs to go, as decoding
MAINTAINERS requires that you have a at least a vague idea which
component might be causing the issue – which afaics is hard even for
people that known a thing or two about the kernel. :-/

Ciao, Thorsten

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Luis Chamberlain]

On Mon, Mar 22, 2021 at 1:25 PM Thorsten Leemhuis <linux@leemhuis.info> wrote:
>
>
>
> On 22.03.21 19:32, Linus Torvalds wrote:
> > On Mon, Mar 22, 2021 at 8:18 AM Thorsten Leemhuis <linux@leemhuis.info> wrote:
> >>
> >>     I even requested a
> >> "linux-regressions@vger.kernel.org" a while later, but didn't hear
> >> anything back; and, sadly, about the same time I started having trouble
> >> finding spare time for working on regression tracking. :-/
> >
> > Honestly, I'd much prefer the name 'linux-regressions' as being much
> > more targeted than 'linux-issues'.
>
> That only solves one of the two problem I'm trying to solve (albeit the
> one that is more important to me). That way users still have no easy way
> to query for reports about issues that are no regressions – say
> something is broken and they have no idea if it once worked or never
> worked at all.

Without a known baseline of what works OK an issue cannot easily be
categorized as a regression. This "problem" I think deserves its own
considerations.

There are some kernel-ci solutions out there which report "issues"
which help develop such baselines, however what we need is a community
visible list of the sum, a list of *known issues upstream* represents
a baseline. The easiest way to develop such baselines are with
respective tests. We however obviously need to also accept new user
reported issues as possible issues which can be candidate baseline
issues, for which perhaps there are no known tests yet available to
reproduce.

Then there are the considerations also that some distribution issues,
which can be part of a distribution baseline, might fit into the
circle of upstream known issues, or baseline as well. But not all
issues part of a distribution baseline are part of the upstream
baseline, an example is a botched backport. Most distribution
baselines however tend to be private, however I'd like to see that
changed using OpenSUSE/SLE as an example in order to help with the
upstream baseline effort. I'd like to encourage other distributions to
follow suit.

Test frameworks help develop a baseline and so working on them helps
reduce the scope of this problem. We have many test frameworks. What I
have not seen is a public generic baseline "list" for each of these. I
have spent a bit of time on this problem and have come up with a
generic format for issues on test frameworks as part of kdevops [0] in
the hopes that it could be used to easily grep for known issues
against upstream kernels / distribution releases. The format is
simple:

mcgrof@bicho ~/kdevops (git::master)$ cat
workflows/blktests/expunges/5.12.0-rc1-next-20210304/failures.txt
block/009 # korg#212305 failure rate 1/669
block/011
block/012

The korg#212305 refers to bugzilla.kernel.org bug ID #212305 [0].

Distribution issues:

mcgrof@bicho ~/kdevops (git::master)$ cat
workflows/blktests/expunges/debian/testing/failures.txt
block/011
block/012
meta/005
meta/006
meta/009
nbd/002
nbd/003 # causes a hang after running a few times
scsi/004

I have support for blktests and fstests, will add selftests soon. I
tend to work on debian baseline as a public demo for work. The
OpenSUSE Leap 15.3 baseline will be reflective of the real SLE15.3
baseline.

The nice thing about having a public baseline is we can then really be
confident into labelling a new issue that comes up as a possible
regression. However, confidence is subjective, and so one must also
define confidence clearly. You associate confidence to a baseline by
the number of full tests you have run against a baseline for a
respective test framework. Borrowing IO stabilizing terms, I'm using a
test "steady state goal" for this, it means how many times have you
run all possible tests against a known baseline without failure. So a
steady state of 100 for blktests means your confidence in the baseline
you have developed is of 100 full tests. A higher steady state goal
however means more time is required to test, and so sometimes you
might be confined to only use a low steady state goal, but then use
side workers to run random tests with a higher test count. So for
instance, the failure rate of the issue reported on korg#212305 is
defined by the average number of times one must run a test in order
for it to fail. If your baseline steady state goal was just 100,
chances are low you may have run into that issue.

Are there other known collections of public baselines easily grep'able
for different test frameworks? Where can we contribute and collaborate
to such a thing?

PS. My current goal for steady state goal for upstream is 1000 for
blktests, 100 for fstests per filesystem.

[0] https://github.com/mcgrof/kdevops
[1] https://bugzilla.kernel.org/show_bug.cgi?id=212305

  Luis

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Konstantin Ryabitsev]

On Tue, 23 Mar 2021 at 04:58, Thorsten Leemhuis <linux@leemhuis.info> wrote:
> >  If we can
> > actually get users to *read* it, I think it's going to save kernel
> > developers a huge amount of time and frustration.
>
> And users hopefully as well. But yes, making them read it is the
> problem. :-/

I've added a very visible admonition on the front of
bugzilla.kernel.org. Hopefully, it will help direct some users to
their distro bug trackers first.

> > I wonder if it might be useful to have a form which users could be
> > encouraged to fill out so that (a) the information is available in a
> > structured format so it's easier for developers to find the relevant
> > information, (b) so it is easier for programs to parse, for easier
> > reporting or indexing, and (c) as a nudge so that users remember to
> > report critical bits of information such as the hardware
> > configuration, the exact kernel version, which distribution userspace
> > was in use, etc.
> >
> > There could also be something in the text form which would make it
> > easier for lore.kernel.org searches to identify bug reports.  (e.g.,
> > "LINUX KERNEL BUG REPORTER TEMPLATE")
>
> Hmmm, yeah, I like that idea. I'll keep it in mind for later: I would
> prefer to get reporting-issues.rst officially blessed and
> reporting-bugs.rst gone before working on further enhancements.

To my knowledge, git project uses a tool for that:
https://git-scm.com/docs/git-bugreport

Theoretically, a similar tool could exist for the kernel.

-K

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Steven Rostedt]

On Mon, 22 Mar 2021 20:25:15 +0100
Thorsten Leemhuis <linux@leemhuis.info> wrote:

> I agree to the last point and yeah, maybe regressions are the more
> important problem we should work on – at least from the perspective of
> kernel development.  But from the users perspective (and
> reporting-issues.rst is written for that perspective) it feel a bit
> unsatisfying to not have a solution to query for existing report,
> regressions or not. Hmmmm...

I think the bulk of user issues are going to be regressions. Although you
may be in a better position to know for sure, but at least for me, wearing
my "user" hat, the thing that gets me the most is upgrading to a new kernel
and suddenly something that use to work no longer does. And that is the
definition of a regression. My test boxes still run old distros (one is
running fedora 13). These are the boxes that catch the most issues, and if
they do, they are pretty much guaranteed to be a regression.

I like the "linux-regressions" mailing list idea.

-- Steve

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[James Bottomley]

On Tue, 2021-03-23 at 12:20 -0400, Steven Rostedt wrote:
> On Mon, 22 Mar 2021 20:25:15 +0100
> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
> > I agree to the last point and yeah, maybe regressions are the more
> > important problem we should work on – at least from the perspective
> > of kernel development.  But from the users perspective (and
> > reporting-issues.rst is written for that perspective) it feel a bit
> > unsatisfying to not have a solution to query for existing report,
> > regressions or not. Hmmmm...
> 
> I think the bulk of user issues are going to be regressions. Although
> you may be in a better position to know for sure, but at least for
> me, wearing my "user" hat, the thing that gets me the most is
> upgrading to a new kernel and suddenly something that use to work no
> longer does. And that is the definition of a regression. My test
> boxes still run old distros (one is running fedora 13). These are the
> boxes that catch the most issues, and if they do, they are pretty
> much guaranteed to be a regression.
> 
> I like the "linux-regressions" mailing list idea.

Can't we use the fancy features of public inbox to get the best of both
worlds?  Have the bug list (or even a collection of lists) but make the
linux-regressions one a virtual list keying off an imap flag which a
group of people control.  That way anything that is flagged as a
regression appears in that public inbox.  I assume the search can be
quite wide so we could flag a regression on any list indexed by lore?

James

Re: RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Theodore Ts'o]

On Tue, Mar 23, 2021 at 12:20:25PM -0400, Steven Rostedt wrote:
> On Mon, 22 Mar 2021 20:25:15 +0100
> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
> > I agree to the last point and yeah, maybe regressions are the more
> > important problem we should work on – at least from the perspective of
> > kernel development.  But from the users perspective (and
> > reporting-issues.rst is written for that perspective) it feel a bit
> > unsatisfying to not have a solution to query for existing report,
> > regressions or not. Hmmmm...
> 
> I think the bulk of user issues are going to be regressions. Although you
> may be in a better position to know for sure, but at least for me, wearing
> my "user" hat, the thing that gets me the most is upgrading to a new kernel
> and suddenly something that use to work no longer does. And that is the
> definition of a regression. My test boxes still run old distros (one is
> running fedora 13). These are the boxes that catch the most issues, and if
> they do, they are pretty much guaranteed to be a regression.

I think it depends on the user and the subsystem.  You're a
sophisticated user, but I've fielded a goodly number of ext4 "bug
reports" which were coming from a Ubuntu 16.04 kernel, or a user who
is seeing a block device issue (either a driver bug or a hardware
failure), or in some cases both.

A lot of these "bug reports" would be headed off at the pass if we
advertised:

https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html

much more heavily; assuming we can get the users to actually read it,
first.

						- Ted

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Theodore Ts'o]

On Tue, Mar 23, 2021 at 09:57:57AM +0100, Thorsten Leemhuis wrote:
> On 22.03.21 22:56, Theodore Ts'o wrote:
> > On Mon, Mar 22, 2021 at 08:25:15PM +0100, Thorsten Leemhuis wrote:
> >> I agree to the last point and yeah, maybe regressions are the more
> >> important problem we should work on – at least from the perspective of
> >> kernel development.  But from the users perspective (and
> >> reporting-issues.rst is written for that perspective) it feel a bit
> >> unsatisfying to not have a solution to query for existing report,
> >> regressions or not. Hmmmm...
> > First of all, thanks for working on reporting-issues.rst.
> 
> Thx, very glad to hear that. I didn't get much feedback on it, which
> made me wonder if anybody besides docs folks actually looked at it...

I'll admit that I had missed your initial submission, but having
looked at it, while I could imagine some nits where it could be
improved, in my opinion, it's strictly better than the older
reporting-bugs doc.

> Hmmm, yeah, I like that idea. I'll keep it in mind for later: I would
> prefer to get reporting-issues.rst officially blessed and
> reporting-bugs.rst gone before working on further enhancements.

Is there anyone following this thread who believes that there is
anything we should change *before* oficially blessing
reporting-issues.rst?  Given that Konstantin has already linked to
reporting-issues from the front page of kernel.bugzilla.org, I think
we we should just go ahead and officially bless it and be done with
it.   :-)

Once it is blessed, I'd also suggest putting a link to
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
as an "other resources" at https://www.kernel.org.

					- Ted

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Thorsten Leemhuis]

On 23.03.21 19:11, Theodore Ts'o wrote:
> On Tue, Mar 23, 2021 at 09:57:57AM +0100, Thorsten Leemhuis wrote:
>> On 22.03.21 22:56, Theodore Ts'o wrote:
>>> On Mon, Mar 22, 2021 at 08:25:15PM +0100, Thorsten Leemhuis wrote:
>>>> I agree to the last point and yeah, maybe regressions are the more
>>>> important problem we should work on – at least from the perspective of
>>>> kernel development.  But from the users perspective (and
>>>> reporting-issues.rst is written for that perspective) it feel a bit
>>>> unsatisfying to not have a solution to query for existing report,
>>>> regressions or not. Hmmmm...
>>> First of all, thanks for working on reporting-issues.rst.
>> Thx, very glad to hear that. I didn't get much feedback on it, which
>> made me wonder if anybody besides docs folks actually looked at it...
> I'll admit that I had missed your initial submission,

No wonder with all the lists and mails. :-D That's actually why I wanted
to point to the text on ksummit-list once in the near future after two
remaining issues with the text were solved (see below), but before
removing the "WIP" box at the top and deleting reporting-bugs.rst.

> but having
> looked at it, while I could imagine some nits where it could be
> improved,

Yeah, for sure, with such a text that will always be the case. And I
really would like if a few more people take a closer look and provide
feedback, that really helps to get such a text in shape. I have stared
so much at the text in recent months, that makes it quite easy to miss
typos and errors in the logical flow that a fresh pair of eyes will
immediately spot...

> in my opinion, it's strictly better than the older
> reporting-bugs doc.

Great, thx.

>> Hmmm, yeah, I like that idea. I'll keep it in mind for later: I would
>> prefer to get reporting-issues.rst officially blessed and
>> reporting-bugs.rst gone before working on further enhancements.
> Is there anyone following this thread who believes that there is
> anything we should change *before* oficially blessing
> reporting-issues.rst?  Given that Konstantin has already linked to
> reporting-issues from the front page of kernel.bugzilla.org, I think
> we we should just go ahead and officially bless it and be done with
> it.   :-)

FWIW, here is my current todo list from the top of my head:

* get this patchset reviewed and applied:
https://lore.kernel.org/linux-doc/cover.1616181657.git.linux@leemhuis.info/

* *afterwards* make sure Greg and/or Sasha (now CCed) check the text
from their point of view (above patchset changes quite a few things in
that area, that's why it needs to be applied first)

* get feedback reg. the two FIXME boxes remaining afterwards. One is
about bugzilla (```The old text took a totally different approach to
bugzilla.kernel.org...```), the other about CCing LKML  (```Above
section tells users to always CC LKML […] Should we create mailing list
like linux-issues@vger.kernel.org```). But I guess the approach taken
should be fine for most people, so we could simply remove them. We can
still change things later anyway, I just put those boxes there to
highlight these differences to the old approach.

* remove the note at the top (```This document is being prepared to
replace 'Reporting bugs...``` and delete reporting-bugs.rst

> Once it is blessed, I'd also suggest putting a link to
> https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html
> as an "other resources" at https://www.kernel.org.

+1

Ciao, Thorsten

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Thorsten Leemhuis]

On 23.03.21 16:01, Konstantin Ryabitsev wrote:
> On Tue, 23 Mar 2021 at 04:58, Thorsten Leemhuis <linux@leemhuis.info> wrote:
>>>  If we can
>>> actually get users to *read* it, I think it's going to save kernel
>>> developers a huge amount of time and frustration.
>> And users hopefully as well. But yes, making them read it is the
>> problem. :-/
> I've added a very visible admonition on the front of
> bugzilla.kernel.org. Hopefully, it will help direct some users to
> their distro bug trackers first.

Ahh, great, thx!

>>> I wonder if it might be useful to have a form which users could be
>>> encouraged to fill out so that (a) the information is available in a
>>> structured format so it's easier for developers to find the relevant
>>> information, (b) so it is easier for programs to parse, for easier
>>> reporting or indexing, and (c) as a nudge so that users remember to
>>> report critical bits of information such as the hardware
>>> configuration, the exact kernel version, which distribution userspace
>>> was in use, etc.
>>>
>>> There could also be something in the text form which would make it
>>> easier for lore.kernel.org searches to identify bug reports.  (e.g.,
>>> "LINUX KERNEL BUG REPORTER TEMPLATE")
>>
>> Hmmm, yeah, I like that idea. I'll keep it in mind for later: I would
>> prefer to get reporting-issues.rst officially blessed and
>> reporting-bugs.rst gone before working on further enhancements.
> 
> To my knowledge, git project uses a tool for that:
> https://git-scm.com/docs/git-bugreport
> 
> Theoretically, a similar tool could exist for the kernel.

Wasn't aware of that tool, thx for pointing it out, will take a closer look.

Ciao, Thorsten

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Konstantin Ryabitsev]

On Tue, Mar 23, 2021 at 09:30:33AM -0700, James Bottomley wrote:
> > I think the bulk of user issues are going to be regressions. Although
> > you may be in a better position to know for sure, but at least for
> > me, wearing my "user" hat, the thing that gets me the most is
> > upgrading to a new kernel and suddenly something that use to work no
> > longer does. And that is the definition of a regression. My test
> > boxes still run old distros (one is running fedora 13). These are the
> > boxes that catch the most issues, and if they do, they are pretty
> > much guaranteed to be a regression.
> > 
> > I like the "linux-regressions" mailing list idea.
> 
> Can't we use the fancy features of public inbox to get the best of both
> worlds?  Have the bug list (or even a collection of lists) but make the
> linux-regressions one a virtual list keying off an imap flag which a
> group of people control.  That way anything that is flagged as a
> regression appears in that public inbox.  I assume the search can be
> quite wide so we could flag a regression on any list indexed by lore?

There's a number of ways we can accomplish this, sure.

However, this functionality is not in production yet, and I'm not sure which
upcoming public-inbox features we'll be implementing as a public
lore.kernel.org service, which ones we'll only offer to kernel.org account
holders, and which ones should really be running locally by developers
themselves.

So, I don't want to say either yes or no to this one for the fear of
over-promising. I guess this is why I'm not in sales. :)

-K

Re: [Ksummit-discuss] RFC: create mailing list "linux-issues" focussed on issues/bugs and regressions

[Eric Wong]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Tue, Mar 23, 2021 at 09:30:33AM -0700, James Bottomley wrote:
> > > I think the bulk of user issues are going to be regressions. Although
> > > you may be in a better position to know for sure, but at least for
> > > me, wearing my "user" hat, the thing that gets me the most is
> > > upgrading to a new kernel and suddenly something that use to work no
> > > longer does. And that is the definition of a regression. My test
> > > boxes still run old distros (one is running fedora 13). These are the
> > > boxes that catch the most issues, and if they do, they are pretty
> > > much guaranteed to be a regression.
> > > 
> > > I like the "linux-regressions" mailing list idea.
> > 
> > Can't we use the fancy features of public inbox to get the best of both
> > worlds?  Have the bug list (or even a collection of lists) but make the
> > linux-regressions one a virtual list keying off an imap flag which a
> > group of people control.  That way anything that is flagged as a
> > regression appears in that public inbox.  I assume the search can be
> > quite wide so we could flag a regression on any list indexed by lore?

The lei (local email interface) data model will have "labels"(*)
and developers will be able to publish mail with it via static
HTML/Atom/JSON feed via cronjob and whatnot.

> There's a number of ways we can accomplish this, sure.
> 
> However, this functionality is not in production yet, and I'm not sure which
> upcoming public-inbox features we'll be implementing as a public
> lore.kernel.org service,

> which ones we'll only offer to kernel.org account holders,

lei could offer read-write JMAP support; either as a CGI tied
to Unix user accounts or some virtual user system.  Some fixes
I'm currently making to speed up the test suite will also make
it more suitable for a largish virtual user system.

> and which ones should really be running locally by developers
> themselves.

lei will be MY dream mail/git tool that fills in the gaps
left by other tools; I hope it can make others happy, too :)

> So, I don't want to say either yes or no to this one for the fear of
> over-promising. I guess this is why I'm not in sales. :)

Heh, same here.  Once I start using lei to handle all of my mail
and there's a data-loss bug, I could conceivably never know
about it because the bug reports would be lost... :x


[1] "labels" are "mailboxes" in JMAP-speak; and lei's per-user data
    model will be tied to JMAP.  "keywords" are Maildir/IMAP-system
    flags (seen/flagged/answered/...).  JMAP doesn't allow arbitrary
    keywords, but does allow arbitrary labels.