Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

Hi!

TLDR: Core Linux kernel developers are unhappy with the state of
bugzilla.kernel.org; to improve things I plan to change a few important
aspects of its configuration, unless somebody comes up with better ideas
to tackle current problems: (1) Create a catch-all product making it
totally obvious to submitters that likely nobody will look into the
ticket. (2) Remove or hide all products & components where the subsystem
didn't fully commit to look into newly submitted reports. (3) Change the
text on the front page to make it clear that most kernel bug reports
need to be sent by mail.

I recently brought the state of bugzilla.kernel.org up for discussion on
the kernel summit and the kernel maintainers summit in sessions about my
regression tracking efforts. Long story short and rough: in both
sessions attendees were quite unhappy about the current state and wanted
things to change for the better. As I brought that topic up, I guess I
have to get things rolling now.

But before getting into the details, a quick & rough reminder about the
current state of bugzilla.kernel.org:

 * The server and the software running on it are well maintained by the
the infrastructure team (Konstantin et al.); many thx for this!

 * Products, components, default assignees, et al. OTOH are heavily
outdated, incomplete, or wrong: maintaining this is not the job of the
infrastructure team and nobody else has stepped up to take care of this
(for a few more details see:
https://lore.kernel.org/lkml/20220420163223.kz32qomzj3y4hjj5@nitro.local/).

 * To the best of my knowledge bugzilla.kernel.org was never really
sanctioned as the official place to report all sorts of kernel bugs:
only 20 (most of them from the area of ACPI/PM and PCI) out of ~2500
entries in MAINTAINERS currently tell users to report issues there; most
other subsystems just mention email contacts, a few (like the DRM
developers) point reporters to external bugtrackers.

 * Developers of subsystems committed to the bug-tracker afaics usually
react to reports submitted in bugzilla.kernel.org. A few other
developers & subsystems keep an eye on reports, too; some do this
directly, others rely on bugzilla forwarding reports for certain
products/components by mail to the subsystem's mailing list. Quite some
or a lot of tickets are not forwarded to any developer or mailing list
at all.

 * In the end lots of bug and regression reports (even good ones!) never
get a reply from a developer, as a brief analysis of mine showed
(https://lore.kernel.org/lkml/6808cd17-b48c-657d-de60-ef9d8bfa151e@leemhuis.info/
). I at least currently try to work a bit against this by briefly
looking at each new report and forwarding any by mail that looks like a
regression worth forwarding (I ignore everything else). Artem S.
Tashkinov also looks into some (all?) reports and tries to help reporters.

The sessions on kernel summit and the kernel maintainers summit
discussed the current state only for a few minutes. It's hard to
summarize these discussions, but let me try to mention the aspects that
are important for now:

 * In both sessions members of the audience seemed pretty unhappy to me
about the current state of things.

 * In the kernel summit sessions (recording:
https://youtu.be/e2SZoUPhDRg?t=5370 ) Len Brown stated that he and
fellow ACPI/PM developers rely on bugzilla.kernel.org and would need
some replacement if it's decommissioned.

 * On the maintainers summit (see the last section of
https://lwn.net/Articles/908324/ for a brief write-up that coined the
term "Bugzilla blues") someone brought up the upstream development of
bugzilla the software seems to be dead; there was not even one strong
advocate for bugzilla.kernel.org and the general vibe tented into the
direction of "let's get rid of it". But it was also mentioned that
bugzilla.kernel.org does something useful which will need a replacement:
a place where reporters can upload big files needed for debugging problems.

In the end that made me settle on this plan of action:

 1. Finding a replacement for bugzilla will take a while, so for now
let's try to reduce some of its aspects that are bothering people:

  1a. Create a new product/component that can act as a catch-all bug,
but makes it pretty clear that nobody might see the report because it's
not forwarded to anyone. People can use it to upload files for debugging
and link to them in mailed reports. People unable or unwilling to report
issues my mail (see 1c) could use it to submit issues, too. The outcome
then is the same as before, but at least people were told upfront about
the likely outcome; it also gives users a chance to help each other or
to coordinate before properly reporting an issue.

  1b. Go through the list of products and components and hide or remove
*all* where the subsystem didn't fully commit to look into newly
submitted reports. Minimum requirements to remain listed will be along
these lines: subsystem mentions bugzilla.kernel.org in MAINTAINERS or a
developer listed in MAINTAINERS is one of the default assignees in
bugzilla. Subsystems where bugzilla forwards mails to a mailing list can
remain listed as well, if the recent history shows the developers look
into newly filed bugs. I'll use my best judgment in the transition
process and will file "anyone listening?" bugs if in a doubt.

  1c. Make it obvious on the front-page of bugzilla.kernel.org that most
kernel developers want bug reports to be submitted by mail; mention the
subsystems that accept reports there and point to the catch-all bug (see
1a) as a last straw.

 2. See if everybody is happy with the new state for the time being; if
not further fine-tune things or speed up step (3).

 3. Work out what we want as replacement.

Anyone any comments on this or helpful ideas how to make things even
better? Otherwise, I'll in a week or two get down and start working on
realizing the points listed under step (1).

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

[resent with the right ksummit list in CC]

On 29.09.22 13:19, Thorsten Leemhuis wrote:
> Hi!
> 
> TLDR: Core Linux kernel developers are unhappy with the state of
> bugzilla.kernel.org; to improve things I plan to change a few important
> aspects of its configuration, unless somebody comes up with better ideas
> to tackle current problems: (1) Create a catch-all product making it
> totally obvious to submitters that likely nobody will look into the
> ticket. (2) Remove or hide all products & components where the subsystem
> didn't fully commit to look into newly submitted reports. (3) Change the
> text on the front page to make it clear that most kernel bug reports
> need to be sent by mail.
> 
> I recently brought the state of bugzilla.kernel.org up for discussion on
> the kernel summit and the kernel maintainers summit in sessions about my
> regression tracking efforts. Long story short and rough: in both
> sessions attendees were quite unhappy about the current state and wanted
> things to change for the better. As I brought that topic up, I guess I
> have to get things rolling now.
> 
> But before getting into the details, a quick & rough reminder about the
> current state of bugzilla.kernel.org:
> 
>  * The server and the software running on it are well maintained by the
> the infrastructure team (Konstantin et al.); many thx for this!
> 
>  * Products, components, default assignees, et al. OTOH are heavily
> outdated, incomplete, or wrong: maintaining this is not the job of the
> infrastructure team and nobody else has stepped up to take care of this
> (for a few more details see:
> https://lore.kernel.org/lkml/20220420163223.kz32qomzj3y4hjj5@nitro.local/).
> 
>  * To the best of my knowledge bugzilla.kernel.org was never really
> sanctioned as the official place to report all sorts of kernel bugs:
> only 20 (most of them from the area of ACPI/PM and PCI) out of ~2500
> entries in MAINTAINERS currently tell users to report issues there; most
> other subsystems just mention email contacts, a few (like the DRM
> developers) point reporters to external bugtrackers.
> 
>  * Developers of subsystems committed to the bug-tracker afaics usually
> react to reports submitted in bugzilla.kernel.org. A few other
> developers & subsystems keep an eye on reports, too; some do this
> directly, others rely on bugzilla forwarding reports for certain
> products/components by mail to the subsystem's mailing list. Quite some
> or a lot of tickets are not forwarded to any developer or mailing list
> at all.
> 
>  * In the end lots of bug and regression reports (even good ones!) never
> get a reply from a developer, as a brief analysis of mine showed
> (https://lore.kernel.org/lkml/6808cd17-b48c-657d-de60-ef9d8bfa151e@leemhuis.info/
> ). I at least currently try to work a bit against this by briefly
> looking at each new report and forwarding any by mail that looks like a
> regression worth forwarding (I ignore everything else). Artem S.
> Tashkinov also looks into some (all?) reports and tries to help reporters.
> 
> The sessions on kernel summit and the kernel maintainers summit
> discussed the current state only for a few minutes. It's hard to
> summarize these discussions, but let me try to mention the aspects that
> are important for now:
> 
>  * In both sessions members of the audience seemed pretty unhappy to me
> about the current state of things.
> 
>  * In the kernel summit sessions (recording:
> https://youtu.be/e2SZoUPhDRg?t=5370 ) Len Brown stated that he and
> fellow ACPI/PM developers rely on bugzilla.kernel.org and would need
> some replacement if it's decommissioned.
> 
>  * On the maintainers summit (see the last section of
> https://lwn.net/Articles/908324/ for a brief write-up that coined the
> term "Bugzilla blues") someone brought up the upstream development of
> bugzilla the software seems to be dead; there was not even one strong
> advocate for bugzilla.kernel.org and the general vibe tented into the
> direction of "let's get rid of it". But it was also mentioned that
> bugzilla.kernel.org does something useful which will need a replacement:
> a place where reporters can upload big files needed for debugging problems.
> 
> In the end that made me settle on this plan of action:
> 
>  1. Finding a replacement for bugzilla will take a while, so for now
> let's try to reduce some of its aspects that are bothering people:
> 
>   1a. Create a new product/component that can act as a catch-all bug,
> but makes it pretty clear that nobody might see the report because it's
> not forwarded to anyone. People can use it to upload files for debugging
> and link to them in mailed reports. People unable or unwilling to report
> issues my mail (see 1c) could use it to submit issues, too. The outcome
> then is the same as before, but at least people were told upfront about
> the likely outcome; it also gives users a chance to help each other or
> to coordinate before properly reporting an issue.
> 
>   1b. Go through the list of products and components and hide or remove
> *all* where the subsystem didn't fully commit to look into newly
> submitted reports. Minimum requirements to remain listed will be along
> these lines: subsystem mentions bugzilla.kernel.org in MAINTAINERS or a
> developer listed in MAINTAINERS is one of the default assignees in
> bugzilla. Subsystems where bugzilla forwards mails to a mailing list can
> remain listed as well, if the recent history shows the developers look
> into newly filed bugs. I'll use my best judgment in the transition
> process and will file "anyone listening?" bugs if in a doubt.
> 
>   1c. Make it obvious on the front-page of bugzilla.kernel.org that most
> kernel developers want bug reports to be submitted by mail; mention the
> subsystems that accept reports there and point to the catch-all bug (see
> 1a) as a last straw.
> 
>  2. See if everybody is happy with the new state for the time being; if
> not further fine-tune things or speed up step (3).
> 
>  3. Work out what we want as replacement.
> 
> Anyone any comments on this or helpful ideas how to make things even
> better? Otherwise, I'll in a week or two get down and start working on
> realizing the points listed under step (1).
> 
> Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

Hello everyone,

I'm glad the issue has been brought up again, I did it earlier but the
discussion never gained any traction:

https://lkml.org/lkml/2021/11/5/425
https://lore.kernel.org/lkml/6abc7248-efda-b569-9030-5384e5ce1f29@gmx.com/

Let me be brutally honest here, if you're working on the kernel,
specially for a large company such as e.g. Intel, you're _expected_ to
address the issues which are related to the kernel component[s] you're
maintaining/developing otherwise it's not "development" it's "I'm
dumping my code because my employer pays me to do that". That also means
you're expected to address bug reports.

It's correct I've tried to help people with bug reports posted on
bugzilla.kernel.org but it's a tough task considering that absolute most
kernel developers are not signed up, thus most bug reports are never
seen by respective developers.

How I'd go about the whole situation:

* Bugzilla must be there whether people like it or not. I've dealt with
LKML and other subsystems' mailing lists and the situation is even
_worse_: absolute most emails are simply completely ignored and _never_
replied to. The related bug reports, of course, are rarely if ever
addressed. It's so easy to say "sorry, yesterday I received 200 new
emails and simply didn't notice a new issue". That's ugly.

* All the components in the kernel bugzilla must be synchronized with
Kconfig's, in a perfect world automatically.

* Some kernel components, e.g. amdgpu, i915 and others have their own
bug trackers. Here's my proposal. People don't need to dig deep and
understand the intricacies of kernel development, all the components
must be there.

   Whenever a person tries to file a bug report for e.g. Drivers ->
Video Intel (not currently there),

   * * they either must be redirected to the appropriate bug tracker (
https://gitlab.freedesktop.org/drm/intel ) automatically, or

   * * a copy of the bug report in the appropriate bug tracker must be
created, or

   * * an email must be sent to the appropriate mailing list.

   Not sure if Bugzilla supports any of that but it's hugely important.

* Subsystem _maintainers_ must be present in the bugzilla by definition.
You're a maintainer after all. You're expected to be responsible (that
excludes the previous point if it's addressed).

* Kernel bugzilla must be opt-out, not opt-in. To be honest I'd
automatally add everyone who's commited to the kernel in the past 6
months and of course if new developers commit to the kernel, I'll add
them as well. Only if they _hate_ getting bugzilla emails, they are free
to unsubscribe.

* Speaking of a catch-all component. Mozilla does exactly that: bug
reports are filed under such a component however then AI/agorithm or a
person assigns them to a proper component. As a person someone could
certainly do that but I've not seen any open positions/vacancies for that.

TLDR: it's so easy to hate/dismiss bugzilla and say "use our mailing
list instead". Practice shows that "your mailing lists" are too often
completely disfunctional and allow [bug] reports to linger and get never
addressed which is not good for the kernel. I strongly oppose the idea
of kernel bugzilla deprecation.

AFAIK, the kernel bugzilla is a Linux Foundation project and the
organization receives funding from its very rich members including
Google, Meta, Intel, and even Microsoft. The fact that no one is
seriously working on it looks shameful and sad. We are not talking about
a minor odd library with a dozen users we are talking about the kernel.

Sorry about the tone of the message, I'm just too invested. It pains to
see how the kernel issues in regard to its use on the desktop receive
very little attention and how things which are important for major
companies (server use and Android) are all the rage and there are
specific people addressing them.

Best regards,
Artem

On 9/29/22 11:33, Thorsten Leemhuis wrote:
> [resent with the right ksummit list in CC]
>
> On 29.09.22 13:19, Thorsten Leemhuis wrote:
>> Hi!
>>
>> TLDR: Core Linux kernel developers are unhappy with the state of
>> bugzilla.kernel.org; to improve things I plan to change a few important
>> aspects of its configuration, unless somebody comes up with better ideas
>> to tackle current problems: (1) Create a catch-all product making it
>> totally obvious to submitters that likely nobody will look into the
>> ticket. (2) Remove or hide all products & components where the subsystem
>> didn't fully commit to look into newly submitted reports. (3) Change the
>> text on the front page to make it clear that most kernel bug reports
>> need to be sent by mail.
>>
>> I recently brought the state of bugzilla.kernel.org up for discussion on
>> the kernel summit and the kernel maintainers summit in sessions about my
>> regression tracking efforts. Long story short and rough: in both
>> sessions attendees were quite unhappy about the current state and wanted
>> things to change for the better. As I brought that topic up, I guess I
>> have to get things rolling now.
>>
>> But before getting into the details, a quick & rough reminder about the
>> current state of bugzilla.kernel.org:
>>
>>   * The server and the software running on it are well maintained by the
>> the infrastructure team (Konstantin et al.); many thx for this!
>>
>>   * Products, components, default assignees, et al. OTOH are heavily
>> outdated, incomplete, or wrong: maintaining this is not the job of the
>> infrastructure team and nobody else has stepped up to take care of this
>> (for a few more details see:
>> https://lore.kernel.org/lkml/20220420163223.kz32qomzj3y4hjj5@nitro.local/).
>>
>>   * To the best of my knowledge bugzilla.kernel.org was never really
>> sanctioned as the official place to report all sorts of kernel bugs:
>> only 20 (most of them from the area of ACPI/PM and PCI) out of ~2500
>> entries in MAINTAINERS currently tell users to report issues there; most
>> other subsystems just mention email contacts, a few (like the DRM
>> developers) point reporters to external bugtrackers.
>>
>>   * Developers of subsystems committed to the bug-tracker afaics usually
>> react to reports submitted in bugzilla.kernel.org. A few other
>> developers & subsystems keep an eye on reports, too; some do this
>> directly, others rely on bugzilla forwarding reports for certain
>> products/components by mail to the subsystem's mailing list. Quite some
>> or a lot of tickets are not forwarded to any developer or mailing list
>> at all.
>>
>>   * In the end lots of bug and regression reports (even good ones!) never
>> get a reply from a developer, as a brief analysis of mine showed
>> (https://lore.kernel.org/lkml/6808cd17-b48c-657d-de60-ef9d8bfa151e@leemhuis.info/
>> ). I at least currently try to work a bit against this by briefly
>> looking at each new report and forwarding any by mail that looks like a
>> regression worth forwarding (I ignore everything else). Artem S.
>> Tashkinov also looks into some (all?) reports and tries to help reporters.
>>
>> The sessions on kernel summit and the kernel maintainers summit
>> discussed the current state only for a few minutes. It's hard to
>> summarize these discussions, but let me try to mention the aspects that
>> are important for now:
>>
>>   * In both sessions members of the audience seemed pretty unhappy to me
>> about the current state of things.
>>
>>   * In the kernel summit sessions (recording:
>> https://youtu.be/e2SZoUPhDRg?t=5370 ) Len Brown stated that he and
>> fellow ACPI/PM developers rely on bugzilla.kernel.org and would need
>> some replacement if it's decommissioned.
>>
>>   * On the maintainers summit (see the last section of
>> https://lwn.net/Articles/908324/ for a brief write-up that coined the
>> term "Bugzilla blues") someone brought up the upstream development of
>> bugzilla the software seems to be dead; there was not even one strong
>> advocate for bugzilla.kernel.org and the general vibe tented into the
>> direction of "let's get rid of it". But it was also mentioned that
>> bugzilla.kernel.org does something useful which will need a replacement:
>> a place where reporters can upload big files needed for debugging problems.
>>
>> In the end that made me settle on this plan of action:
>>
>>   1. Finding a replacement for bugzilla will take a while, so for now
>> let's try to reduce some of its aspects that are bothering people:
>>
>>    1a. Create a new product/component that can act as a catch-all bug,
>> but makes it pretty clear that nobody might see the report because it's
>> not forwarded to anyone. People can use it to upload files for debugging
>> and link to them in mailed reports. People unable or unwilling to report
>> issues my mail (see 1c) could use it to submit issues, too. The outcome
>> then is the same as before, but at least people were told upfront about
>> the likely outcome; it also gives users a chance to help each other or
>> to coordinate before properly reporting an issue.
>>
>>    1b. Go through the list of products and components and hide or remove
>> *all* where the subsystem didn't fully commit to look into newly
>> submitted reports. Minimum requirements to remain listed will be along
>> these lines: subsystem mentions bugzilla.kernel.org in MAINTAINERS or a
>> developer listed in MAINTAINERS is one of the default assignees in
>> bugzilla. Subsystems where bugzilla forwards mails to a mailing list can
>> remain listed as well, if the recent history shows the developers look
>> into newly filed bugs. I'll use my best judgment in the transition
>> process and will file "anyone listening?" bugs if in a doubt.
>>
>>    1c. Make it obvious on the front-page of bugzilla.kernel.org that most
>> kernel developers want bug reports to be submitted by mail; mention the
>> subsystems that accept reports there and point to the catch-all bug (see
>> 1a) as a last straw.
>>
>>   2. See if everybody is happy with the new state for the time being; if
>> not further fine-tune things or speed up step (3).
>>
>>   3. Work out what we want as replacement.
>>
>> Anyone any comments on this or helpful ideas how to make things even
>> better? Otherwise, I'll in a week or two get down and start working on
>> realizing the points listed under step (1).
>>
>> Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Greg KH]

On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
> Let me be brutally honest here, if you're working on the kernel,
> specially for a large company such as e.g. Intel, you're _expected_ to
> address the issues which are related to the kernel component[s] you're
> maintaining/developing otherwise it's not "development" it's "I'm
> dumping my code because my employer pays me to do that". That also means
> you're expected to address bug reports.

I wish that were the case, unfortunately it is quite rare that
maintainers of subsystems of the kernel are allowed to work on upstream
issues like this all the time.  Heck, part of the time would be
wonderful too, but that is also quite rare.  So while maintainers would
love to be able to work like this, getting their management to agree to
this is not very common, sadly.

> AFAIK, the kernel bugzilla is a Linux Foundation project and the
> organization receives funding from its very rich members including
> Google, Meta, Intel, and even Microsoft. The fact that no one is
> seriously working on it looks shameful and sad. We are not talking about
> a minor odd library with a dozen users we are talking about the kernel.

bugzilla.kernel.org is _hosted_ by the LF, and does a great job of
keeping it running and alive.  The LF has nothing to do with the content
of the bugs in it, the reporting, the response of people to reported
bugs, assigning bugs to anyone, getting them fixed, or anything else
related to the content in the database at all.  Please don't get
confused with the resources provided to host the system vs. the people
who actually do the kernel development itself.

Note, the LF does sponsor a few kernel developers to do work on the
kernel, including myself, but we are a tiny drop in the bucket compared
to the 4000+ developers who contribute to the kernel every year.

thanks,

greg k-h

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
> AFAIK, the kernel bugzilla is a Linux Foundation project and the
> organization receives funding from its very rich members including
> Google, Meta, Intel, and even Microsoft. The fact that no one is
> seriously working on it looks shameful and sad. We are not talking about
> a minor odd library with a dozen users we are talking about the kernel.

The bugzilla as a software platform is a Mozilla product, not Linux
Foundation. Unfortunately, it's pretty much dead:

1. all development has stopped years ago
2. it doesn't even work with recent MySQL servers
3. it is written in perl5 and can only pretty much run with mod_perl

We're committed to running it as far as we can, but we all must also admit
that the platform is near-death and probably will become an ever-increasing
burden to keep it operating. Heck, one of our IT staff is currently trying to
convert bugzilla.kernel.org to use Postgres just so we can keep operating it
past the end of 2022.

The Linux Foundation IT is in charge of running infrastructure -- we're not a
development shop. All of our software projects are pretty much "skunkworks"
efforts (and yes, this includes b4).

We do have ability to fund development efforts -- LF has been the primary
sponsor behind public-inbox.org over the past 3 years. However, there must be
a clear, strong, and well-articulated mandate from the community. From what I
heard, the vast majority of maintainers simply want a web form that would
allow someone to:

1. clearly state what kernel version they are using
2. clearly describe what they were trying to do
3. explain what they expected vs. what they got
4. attach any files
5. give this bug report a unique identifier

Then a designated person would look through the bug report and either:

a. quick-close it (with the usual "talk to your distro" or "don't use a
   tainted kernel" etc)
b. identify the responsible maintainers and notify them

The hard part is not technical -- the hard part is that "designated person."
Being a bugmaster is a thankless job that leads to burnout, regardless of how
well you are paid. Everyone is constantly irate at you from both ends -- the
users are annoyed because their stuff doesn't work, and the maintainers are
annoyed because you keep yanking them to work on dull problems that require a
ton of back-and-forth with people who aren't capable of applying patches and
booting custom kernels.

Before we try to fix/replace bugzilla, we really need to figure out the entire
process and pinpoint who is going to be the one in charge of bug reports. If
you think that LF should establish a fund for a position like that, then you
should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk
to LF management. The IT team will be happy to support you with the tooling,
but tooling should come second to that -- otherwise we'll just be replacing an
old and rusty dumpster on fire with a new and shiny dumpster on fire.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 9/29/22 12:52, Greg KH wrote:
> On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
>> Let me be brutally honest here, if you're working on the kernel,
>> specially for a large company such as e.g. Intel, you're _expected_ to
>> address the issues which are related to the kernel component[s] you're
>> maintaining/developing otherwise it's not "development" it's "I'm
>> dumping my code because my employer pays me to do that". That also means
>> you're expected to address bug reports.
>
> I wish that were the case, unfortunately it is quite rare that
> maintainers of subsystems of the kernel are allowed to work on upstream
> issues like this all the time.  Heck, part of the time would be
> wonderful too, but that is also quite rare.  So while maintainers would
> love to be able to work like this, getting their management to agree to
> this is not very common, sadly.

Understood but it's illogical and I cannot/will not accept it. For
instance, here's a very common scenario: you work for company X. The
company tells you to fix a bug/add a new feature/etc. You write the
code, submit it and it results in a regression for other use cases. Are
you saying it's alright and shouldn't be addressed? That's _exactly_ how
many if not _most_ regressions in the kernel are introduced.

>
>> AFAIK, the kernel bugzilla is a Linux Foundation project and the
>> organization receives funding from its very rich members including
>> Google, Meta, Intel, and even Microsoft. The fact that no one is
>> seriously working on it looks shameful and sad. We are not talking about
>> a minor odd library with a dozen users we are talking about the kernel.
>
> bugzilla.kernel.org is _hosted_ by the LF, and does a great job of
> keeping it running and alive.  The LF has nothing to do with the content
> of the bugs in it, the reporting, the response of people to reported
> bugs, assigning bugs to anyone, getting them fixed, or anything else
> related to the content in the database at all.  Please don't get
> confused with the resources provided to host the system vs. the people
> who actually do the kernel development itself.
>
> Note, the LF does sponsor a few kernel developers to do work on the
> kernel, including myself, but we are a tiny drop in the bucket compared
> to the 4000+ developers who contribute to the kernel every year.
 >
 > thanks,
 >
 > greg k-h

Keeping the website up and running requires next to zero human
time/resources, that's not proper maintenance. The
components/subsystems/developers haven't been updated in over a decade
which results in a bug tracker which is nearly useless. People often
file bug reports under "Other/Other" and no one is notified of anything.
I don't even want to touch on the fact that the  Bugzilla version the
website is running is terribly outdated.

That's the issue I thought we're trying to resolve - making bugzilla
useful. Under no circumstances I want to engage kernel developers or
blame them for anything. I'm grateful you exist and do your work :-)

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Greg KH]

On Thu, Sep 29, 2022 at 01:09:35PM +0000, Artem S. Tashkinov wrote:
> 
> 
> On 9/29/22 12:52, Greg KH wrote:
> > On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
> > > Let me be brutally honest here, if you're working on the kernel,
> > > specially for a large company such as e.g. Intel, you're _expected_ to
> > > address the issues which are related to the kernel component[s] you're
> > > maintaining/developing otherwise it's not "development" it's "I'm
> > > dumping my code because my employer pays me to do that". That also means
> > > you're expected to address bug reports.
> > 
> > I wish that were the case, unfortunately it is quite rare that
> > maintainers of subsystems of the kernel are allowed to work on upstream
> > issues like this all the time.  Heck, part of the time would be
> > wonderful too, but that is also quite rare.  So while maintainers would
> > love to be able to work like this, getting their management to agree to
> > this is not very common, sadly.
> 
> Understood but it's illogical and I cannot/will not accept it.

Wonderful, please work with the companies involved to get them to change
their internal structures to support this.  Have fun! :)

Seriously, many of us have been working with many companies to try to
change this, but it's slow going and requires constant retraining of new
managers when they get moved around.  It's a thankless job, please reach
out to any contacts that you have to help out.

> For
> instance, here's a very common scenario: you work for company X. The
> company tells you to fix a bug/add a new feature/etc. You write the
> code, submit it and it results in a regression for other use cases. Are
> you saying it's alright and shouldn't be addressed? That's _exactly_ how
> many if not _most_ regressions in the kernel are introduced.

True, but the time of a report of a regression is usually much delayed
from the original development, and sometimes the original developers
have moved on.

But the topic here was the maintainers of the subsystems, not the people
who did the original development effort.  That's two totally different
groups of people, with different managers, working for different
companies.  Please don't get them confused.

> > > AFAIK, the kernel bugzilla is a Linux Foundation project and the
> > > organization receives funding from its very rich members including
> > > Google, Meta, Intel, and even Microsoft. The fact that no one is
> > > seriously working on it looks shameful and sad. We are not talking about
> > > a minor odd library with a dozen users we are talking about the kernel.
> > 
> > bugzilla.kernel.org is _hosted_ by the LF, and does a great job of
> > keeping it running and alive.  The LF has nothing to do with the content
> > of the bugs in it, the reporting, the response of people to reported
> > bugs, assigning bugs to anyone, getting them fixed, or anything else
> > related to the content in the database at all.  Please don't get
> > confused with the resources provided to host the system vs. the people
> > who actually do the kernel development itself.
> > 
> > Note, the LF does sponsor a few kernel developers to do work on the
> > kernel, including myself, but we are a tiny drop in the bucket compared
> > to the 4000+ developers who contribute to the kernel every year.
> >
> > thanks,
> >
> > greg k-h
> 
> Keeping the website up and running requires next to zero human
> time/resources, that's not proper maintenance.

Um, no, not at all.  See Konstantin's response for just a small snippet
of what is involved here.

> The
> components/subsystems/developers haven't been updated in over a decade
> which results in a bug tracker which is nearly useless. People often
> file bug reports under "Other/Other" and no one is notified of anything.
> I don't even want to touch on the fact that the  Bugzilla version the
> website is running is terribly outdated.
> 
> That's the issue I thought we're trying to resolve - making bugzilla
> useful. Under no circumstances I want to engage kernel developers or
> blame them for anything. I'm grateful you exist and do your work :-)

I agree, making bugzilla, or something else, useful would be wonderful,
but so far no one has stepped up to do that work except for Thorsten
here.  And it will involve getting 700+ different maintainers to agree
what to do, that's going to be "fun" :)

thanks,

greg k-h

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Thu, Sep 29, 2022 at 01:09:35PM +0000, Artem S. Tashkinov wrote:
> Keeping the website up and running requires next to zero human
> time/resources, that's not proper maintenance.

I'm just going to let this one slide -- but so you know, just keeping spam
hidden/deleted on bugzilla.kernel.org takes 5-10 minutes out of my day. Heck,
I even wrote a script for this:
https://git.kernel.org/pub/scm/linux/kernel/git/mricon/korg-helpers.git/tree/bugzilla-junker.py

> The components/subsystems/developers haven't been updated in over a decade
> which results in a bug tracker which is nearly useless.

Nobody's asked, and it's not the job of IT folks to figure out what these
should be.

> People often file bug reports under "Other/Other" and no one is notified of
> anything.  I don't even want to touch on the fact that the  Bugzilla version
> the website is running is terribly outdated.

It's not. We're running 5.1.1. The latest is 5.1.2 (and it breaks in subtle
ways, which is why we're not running it).

Again. This situation is not a tooling problem. It's a "nobody wants to do
this because it's boring and unglamorous" problem.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 9/29/22 13:04, Konstantin Ryabitsev wrote:
> On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
>> AFAIK, the kernel bugzilla is a Linux Foundation project and the
>> organization receives funding from its very rich members including
>> Google, Meta, Intel, and even Microsoft. The fact that no one is
>> seriously working on it looks shameful and sad. We are not talking about
>> a minor odd library with a dozen users we are talking about the kernel.
>
> The bugzilla as a software platform is a Mozilla product, not Linux
> Foundation. Unfortunately, it's pretty much dead:
>
> 1. all development has stopped years ago
> 2. it doesn't even work with recent MySQL servers
> 3. it is written in perl5 and can only pretty much run with mod_perl
>
> We're committed to running it as far as we can, but we all must also admit
> that the platform is near-death and probably will become an ever-increasing
> burden to keep it operating. Heck, one of our IT staff is currently trying to
> convert bugzilla.kernel.org to use Postgres just so we can keep operating it
> past the end of 2022.
>
> The Linux Foundation IT is in charge of running infrastructure -- we're not a
> development shop. All of our software projects are pretty much "skunkworks"
> efforts (and yes, this includes b4).
>
> We do have ability to fund development efforts -- LF has been the primary
> sponsor behind public-inbox.org over the past 3 years. However, there must be
> a clear, strong, and well-articulated mandate from the community. From what I
> heard, the vast majority of maintainers simply want a web form that would
> allow someone to:
>
> 1. clearly state what kernel version they are using
> 2. clearly describe what they were trying to do
> 3. explain what they expected vs. what they got
> 4. attach any files
> 5. give this bug report a unique identifier
>
> Then a designated person would look through the bug report and either:
>
> a. quick-close it (with the usual "talk to your distro" or "don't use a
>     tainted kernel" etc)
> b. identify the responsible maintainers and notify them
>
> The hard part is not technical -- the hard part is that "designated person."
> Being a bugmaster is a thankless job that leads to burnout, regardless of how
> well you are paid. Everyone is constantly irate at you from both ends -- the
> users are annoyed because their stuff doesn't work, and the maintainers are
> annoyed because you keep yanking them to work on dull problems that require a
> ton of back-and-forth with people who aren't capable of applying patches and
> booting custom kernels.
>
> Before we try to fix/replace bugzilla, we really need to figure out the entire
> process and pinpoint who is going to be the one in charge of bug reports. If
> you think that LF should establish a fund for a position like that, then you
> should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk
> to LF management. The IT team will be happy to support you with the tooling,
> but tooling should come second to that -- otherwise we'll just be replacing an
> old and rusty dumpster on fire with a new and shiny dumpster on fire.
>
> -K

To me it sounds like the best way to keep moving forward is simply
convert git.kernel.org + patchwork.kernel.org + bugzilla to
gitlab.kernel.org and that will solve all the issues immediately. That
will require of course a ton of work but:

1) All the commiters will be automatically present and you can easily CC
them

2) All the kernel directories could be split into components with the
respective developers being subscribed to them automatically. There's an
issue though: sometimes directories/components are rearranged. Gitlab
however is quite powerful, so issues can be easily moved between components.

3) It's gonna be a ton easier to keep track of commits and
discuss/review them. AFAIK it's now done using LKML +
patchwork.kernel.org and then commits are merged by maintainers. So many
places to keep track of.

4) Gitlab probably can be integrated with other gitlabs (at least AMD,
Intel and Nouveau drivers are developed on gitlab.freedesktop.org).

Gitlab simplifies all of that tremendously. Github will work as well but
I know many people don't like it.

Linus, as a commander, may continue having his local git repo or using
its own git website and get merge requests from gitlab.kernel.org. For
him barely anything will change (aside from URLs to fetch from).

Gitlab works as a docker container and requires only a Postgres backend
which simplifies updates and backups.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Greg KH]

On Thu, Sep 29, 2022 at 01:31:49PM +0000, Artem S. Tashkinov wrote:
> 
> 
> On 9/29/22 13:04, Konstantin Ryabitsev wrote:
> > On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
> > > AFAIK, the kernel bugzilla is a Linux Foundation project and the
> > > organization receives funding from its very rich members including
> > > Google, Meta, Intel, and even Microsoft. The fact that no one is
> > > seriously working on it looks shameful and sad. We are not talking about
> > > a minor odd library with a dozen users we are talking about the kernel.
> > 
> > The bugzilla as a software platform is a Mozilla product, not Linux
> > Foundation. Unfortunately, it's pretty much dead:
> > 
> > 1. all development has stopped years ago
> > 2. it doesn't even work with recent MySQL servers
> > 3. it is written in perl5 and can only pretty much run with mod_perl
> > 
> > We're committed to running it as far as we can, but we all must also admit
> > that the platform is near-death and probably will become an ever-increasing
> > burden to keep it operating. Heck, one of our IT staff is currently trying to
> > convert bugzilla.kernel.org to use Postgres just so we can keep operating it
> > past the end of 2022.
> > 
> > The Linux Foundation IT is in charge of running infrastructure -- we're not a
> > development shop. All of our software projects are pretty much "skunkworks"
> > efforts (and yes, this includes b4).
> > 
> > We do have ability to fund development efforts -- LF has been the primary
> > sponsor behind public-inbox.org over the past 3 years. However, there must be
> > a clear, strong, and well-articulated mandate from the community. From what I
> > heard, the vast majority of maintainers simply want a web form that would
> > allow someone to:
> > 
> > 1. clearly state what kernel version they are using
> > 2. clearly describe what they were trying to do
> > 3. explain what they expected vs. what they got
> > 4. attach any files
> > 5. give this bug report a unique identifier
> > 
> > Then a designated person would look through the bug report and either:
> > 
> > a. quick-close it (with the usual "talk to your distro" or "don't use a
> >     tainted kernel" etc)
> > b. identify the responsible maintainers and notify them
> > 
> > The hard part is not technical -- the hard part is that "designated person."
> > Being a bugmaster is a thankless job that leads to burnout, regardless of how
> > well you are paid. Everyone is constantly irate at you from both ends -- the
> > users are annoyed because their stuff doesn't work, and the maintainers are
> > annoyed because you keep yanking them to work on dull problems that require a
> > ton of back-and-forth with people who aren't capable of applying patches and
> > booting custom kernels.
> > 
> > Before we try to fix/replace bugzilla, we really need to figure out the entire
> > process and pinpoint who is going to be the one in charge of bug reports. If
> > you think that LF should establish a fund for a position like that, then you
> > should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk
> > to LF management. The IT team will be happy to support you with the tooling,
> > but tooling should come second to that -- otherwise we'll just be replacing an
> > old and rusty dumpster on fire with a new and shiny dumpster on fire.
> > 
> > -K
> 
> To me it sounds like the best way to keep moving forward is simply
> convert git.kernel.org + patchwork.kernel.org + bugzilla to
> gitlab.kernel.org and that will solve all the issues immediately. That
> will require of course a ton of work but:

For loads of reasons that have been stated before, we aren't going to
move everything to gitlab, sorry.  That's a non-starter for a wide range
of reasons, not the least being you are trying to solve a "we have no
one who wants to wrangle bugs in bugzilla" problem with "move all of our
code hosting infrastructure to a totally different thing that can't even
provide the basic things that we have today".

Sorry, not going to happen, gitlab is not the solution here.

greg k-h

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Thu, 29 Sep 2022 13:33:53 +0200
Thorsten Leemhuis <linux@leemhuis.info> wrote:

Thanks Thorsten for doing this.

> >  * In the kernel summit sessions (recording:
> > https://youtu.be/e2SZoUPhDRg?t=5370 ) Len Brown stated that he and
> > fellow ACPI/PM developers rely on bugzilla.kernel.org and would need
> > some replacement if it's decommissioned.
> > 

I also use bugzilla.kernel.org with trace-cmd/kernelshark and the
libraries, although I don't really use it for the Linux tracing subsystem
(but I probably should :-/).

That is, the tools portion of bugzilla is not part of the MAINTAINERS file
(that I know of), so probably shouldn't be affected by this.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Thu, Sep 29, 2022 at 01:31:49PM +0000, Artem S. Tashkinov wrote:
> To me it sounds like the best way to keep moving forward is simply
> convert git.kernel.org + patchwork.kernel.org + bugzilla to
> gitlab.kernel.org and that will solve all the issues immediately.

No, you will still have all the exact same problems as long as nobody is in
charge of handling incoming bugs. There are plenty of active github/gitlab
projects with thousands of open issues that nobody is working on for the exact
same reason nobody is working on bugs filed via bugzilla -- the right people
didn't see it (or are actively ignoring it, because they are working on
something else).

> That will require of course a ton of work but:
> 
> 1) All the commiters will be automatically present and you can easily CC
>    them

Just like with bugzilla.

> 2) All the kernel directories could be split into components with the
>    respective developers being subscribed to them automatically. There's an
>    issue though: sometimes directories/components are rearranged. Gitlab
>    however is quite powerful, so issues can be easily moved between
>    components.

Just like bugzilla.

> 3) It's gonna be a ton easier to keep track of commits and discuss/review
>    them. AFAIK it's now done using LKML + patchwork.kernel.org and then
>    commits are merged by maintainers. So many places to keep track of.

Now there will be a single place someone can knock out to stop all kernel
development and coordination, subject to countrywide IP blocks, political
influence, etc.

Besides, maintainers already have a single place to keep track of things --
their inbox. If they didn't see something in their inbox, how is it going to
be different when it's a web interface?

> 4) Gitlab probably can be integrated with other gitlabs (at least AMD, Intel
>    and Nouveau drivers are developed on gitlab.freedesktop.org).
> 
> Gitlab simplifies all of that tremendously. Github will work as well but I
> know many people don't like it.

Gitlab is also a commercial open-core project. It is permanently in danger of
being swallowed by some $ENTITY_NOBODY_LIKES, who will for sure look to
prioritize what kinds of things go in to the "open core" part and what kinds
of things are only available with subscription, in order to improve profit
margins.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 29.09.22 15:47, Steven Rostedt wrote:
> On Thu, 29 Sep 2022 13:33:53 +0200
> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
> Thanks Thorsten for doing this.

Thx for saying this, as I fear it in the end will again be more work
then anticipated -- but well, that's how life often is :-D

>>>  * In the kernel summit sessions (recording:
>>> https://youtu.be/e2SZoUPhDRg?t=5370 ) Len Brown stated that he and
>>> fellow ACPI/PM developers rely on bugzilla.kernel.org and would need
>>> some replacement if it's decommissioned.
> 
> I also use bugzilla.kernel.org with trace-cmd/kernelshark and the
> libraries, although I don't really use it for the Linux tracing subsystem
> (but I probably should :-/).
> 
> That is, the tools portion of bugzilla is not part of the MAINTAINERS file
> (that I know of), so probably shouldn't be affected by this.

Ohh, yeah, sorry, should have mentioned this. Don't worry, I'm aware of
this particular and a few similar products/components in bugzilla. I
don't plan changing any of them, unless something unforeseen or a very
good reason comes up (for example if they're obviously unused for years
or something like that).

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Thu, 29 Sep 2022 16:02:56 +0200
Thorsten Leemhuis <linux@leemhuis.info> wrote:

> Ohh, yeah, sorry, should have mentioned this. Don't worry, I'm aware of
> this particular and a few similar products/components in bugzilla. I
> don't plan changing any of them, unless something unforeseen or a very
> good reason comes up (for example if they're obviously unused for years
> or something like that).

Well, years of not being used may not be a good indicator. I use bugzilla
as a dumping ground of my "todo lists". And I find sometimes it may take
years before I get to them ;-)

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 9/29/22 13:53, Konstantin Ryabitsev wrote:
> On Thu, Sep 29, 2022 at 01:31:49PM +0000, Artem S. Tashkinov wrote:
>> To me it sounds like the best way to keep moving forward is simply
>> convert git.kernel.org + patchwork.kernel.org + bugzilla to
>> gitlab.kernel.org and that will solve all the issues immediately.
>
> No, you will still have all the exact same problems as long as nobody is in
> charge of handling incoming bugs. There are plenty of active github/gitlab
> projects with thousands of open issues that nobody is working on for the exact
> same reason nobody is working on bugs filed via bugzilla -- the right people
> didn't see it (or are actively ignoring it, because they are working on
> something else).
>
>> That will require of course a ton of work but:
>>
>> 1) All the commiters will be automatically present and you can easily CC
>>     them
>
> Just like with bugzilla.
>
>> 2) All the kernel directories could be split into components with the
>>     respective developers being subscribed to them automatically. There's an
>>     issue though: sometimes directories/components are rearranged. Gitlab
>>     however is quite powerful, so issues can be easily moved between
>>     components.
>
> Just like bugzilla.
>
>> 3) It's gonna be a ton easier to keep track of commits and discuss/review
>>     them. AFAIK it's now done using LKML + patchwork.kernel.org and then
>>     commits are merged by maintainers. So many places to keep track of.
>
> Now there will be a single place someone can knock out to stop all kernel
> development and coordination, subject to countrywide IP blocks, political
> influence, etc.
>
> Besides, maintainers already have a single place to keep track of things --
> their inbox. If they didn't see something in their inbox, how is it going to
> be different when it's a web interface?
>
>> 4) Gitlab probably can be integrated with other gitlabs (at least AMD, Intel
>>     and Nouveau drivers are developed on gitlab.freedesktop.org).
>>
>> Gitlab simplifies all of that tremendously. Github will work as well but I
>> know many people don't like it.
>
> Gitlab is also a commercial open-core project. It is permanently in danger of
> being swallowed by some $ENTITY_NOBODY_LIKES, who will for sure look to
> prioritize what kinds of things go in to the "open core" part and what kinds
> of things are only available with subscription, in order to improve profit
> margins.
>
> -K

Not going to argue about that.

That leaves us with Bugzilla that no one wants to touch and some people
actively want to delete altogether. In other words, no central place to
report bugs or keep track of them.

I've mentioned several times already that mailing lists are _even worse_
in terms of reporting issues. Developers get emails and simply ignore
them (for a multitude of reasons).

And finding and engaging with existing issues becomes near impossible.
With Bugzilla you can easily leave a comment, attach a patch, attach
debug files, etc. For many mailing lists following up on an old message
is not possible unless you know the old message ID (the nature of mail
clients). With bugzilla you can see a list of reported/open bugs. With
bugzilla you can easily engage with other bug reporters.

To me it all sounds like kernel developers simply do not want any
responsibility or any extra emails whatsoever and instead would love
everyone to spam LKML: when you feel like it, you can check your inbox,
maybe leave a message, maybe not. Who cares? It's LKML.

Getting back to my first message in this discussion,

* Let's refresh all the components in Bugzilla
* Components may not have any people responsible for them at all. Bug
reporters will have to CC the people they are interested in.
* Let's subscribe the past six months of developers (using git commit logs)
* Whoever wants to unsubscribe is free to do so.

Alternatively:

* Delete all the components.
* Leave a catch-all one.
* Let bug reports rot because no one will ever see them. Almost just
like now. Don't remind me of mailing lists.

If not for bugzilla, let's use something more modern. I don't know any
comparable projects however. Trac is truly horrible. You cannot even
unsubscribe from bug reports. Maybe I've missed something. Discourse?
Not a bug tracker per se but can certainly work this way.

To me it all sounds like the sentiment is to absolve from any and all
responsibility and shut your ears and eyes to any reports of your code
malfunctioning/breaking. Fine, let's do it.

Sarcasm and pain aside, Linus Torvalds himself _via Bugzilla_ has helped
me resolve critical issues on several occasions while my messages to
LKML were simply _ignored_. Think about that.

Mailing lists will not work for such a huge project. Period. In the
early 90s they worked, but we are 25 years later with millions more
users. With a ton more of a ton more complicated hardware.

Maybe let's try Discourse with just a few "forums":

* arch
* drivers
* fs
* block/init/ipc/kernel/mm/crypto/virt/scripts - a single core component
* net
* security
* sound
* tools

That's it. Just 8 forums, zero maintenance, no need to add/remove/manage
anything.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hey!

Jumping in here to offer my input...

> On Sep 29, 2022, at 10:22 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> 
> That leaves us with Bugzilla that no one wants to touch and some people
> actively want to delete altogether. In other words, no central place to
> report bugs or keep track of them.

This is the current problem that seems to be appearing here. I get why no one wants to touch it, but it doesn’t solve the problem. 

As you said:

> I've mentioned several times already that mailing lists are _even worse_
> in terms of reporting issues. Developers get emails and simply ignore
> them (for a multitude of reasons).

It’s 100% true that emails get _buried_ as waves of them come in (LKML itself gets hundreds upon hundreds a day, as I’m sure all of you know) and it just isn’t something I personally see as viable, especially for issues that may or may not be high priority.

> Getting back to my first message in this discussion,
> 
> * Let's refresh all the components in Bugzilla
> * Components may not have any people responsible for them at all. Bug
> reporters will have to CC the people they are interested in.
> * Let's subscribe the past six months of developers (using git commit logs)
> * Whoever wants to unsubscribe is free to do so.

Not a terrible idea to me, though obviously, that’s up for debate.

> 
> If not for bugzilla, let's use something more modern. I don't know any
> comparable projects however. Trac is truly horrible. You cannot even
> unsubscribe from bug reports. Maybe I've missed something. Discourse?
> Not a bug tracker per se but can certainly work this way.

Discourse probably isn’t the best fit here, in my opinion. Jira and YouTrack are the only ones I personally know of that are similar to Bugzilla, although as far as I know, they aren’t open source...

Best,
-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[James Bottomley]

On Thu, 2022-09-29 at 12:22 +0000, Artem S. Tashkinov wrote:
> Let me be brutally honest here, if you're working on the kernel,
> specially for a large company such as e.g. Intel, you're _expected_
> to address the issues which are related to the kernel component[s]
> you're maintaining/developing otherwise it's not "development" it's
> "I'm dumping my code because my employer pays me to do that". That
> also means you're expected to address bug reports.
> 
> It's correct I've tried to help people with bug reports posted on
> bugzilla.kernel.org but it's a tough task considering that absolute
> most kernel developers are not signed up, thus most bug reports are
> never seen by respective developers.

The never seen/never responded to metric is rather bogus.  The sad fact
is that a lot of bug reports aren't actionable, meaning the reporter
can't give a reproducer and also can't easily test patches  sometimes
by luck the maintainers can work out what the issue is but a lot of the
time they have no idea.  Then there are ton's of bug reports with
responses like "I think xxx commit fixes your problem, can you test it"
for which the conversation dies there.  There's also the thundering
herd problem: some bugs get reported by many different people (as
different bug reports) but usually the subsystem only engages with one
to fix the issue.  In theory bugzilla can mark the latter as dups, but
that requires someone to spend an enormous amount of time on evaluation
and admin.

That's not to say we can't improve our process, it's just to set
expectations that we're never going to approach anywhere near a perfect
bug process.  Most of the improvements that worked so far involve
having someone coach bug reporters through the process of either
testing patches or reproducing the problem in a more generic
environment ... which I think most people would agree can't really fall
wholly on maintainers.

Regards,

James

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Thu, Sep 29, 2022 at 02:22:10PM +0000, Artem S. Tashkinov wrote:
> * Delete all the components.
> * Leave a catch-all one.
> * Let bug reports rot because no one will ever see them. Almost just
> like now. Don't remind me of mailing lists.

This is my proposal, except also:

1. post all new bugs and comments to a public-inbox feed that people can query
   via lore.kernel.org and tooling like lei.

> Sarcasm and pain aside, Linus Torvalds himself _via Bugzilla_ has helped
> me resolve critical issues on several occasions while my messages to
> LKML were simply _ignored_. Think about that.

In fact, he probably did this by replying to emails, not via the web
interface.

> Mailing lists will not work for such a huge project. Period. In the
> early 90s they worked, but we are 25 years later with millions more
> users. With a ton more of a ton more complicated hardware.

We've recognized this a while ago, which is why our efforts have been targeted
at query-based message feeds. Hence, tools like lore.kernel.org and lei. It's
a work in progress, for sure, but it doesn't require any "everyone must switch
workflows today" kind of coordination, and avoids introducing single points of
failure by making it easy to replicate everything to mirrored systems.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hey there,

> On Sep 29, 2022, at 11:31 AM, Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> 
> On Thu, Sep 29, 2022 at 02:22:10PM +0000, Artem S. Tashkinov wrote:
>> * Delete all the components.
>> * Leave a catch-all one.
>> * Let bug reports rot because no one will ever see them. Almost just
>> like now. Don't remind me of mailing lists.
> 
> This is my proposal, except also:
> 
> 1. post all new bugs and comments to a public-inbox feed that people can query
>   via lore.kernel.org and tooling like lei.

Honestly, giving it a lot more thought, this is a brilliant idea.

> 
>> Mailing lists will not work for such a huge project. Period. In the
>> early 90s they worked, but we are 25 years later with millions more
>> users. With a ton more of a ton more complicated hardware.
> 
> We've recognized this a while ago, which is why our efforts have been targeted
> at query-based message feeds. Hence, tools like lore.kernel.org and lei. It's
> a work in progress, for sure, but it doesn't require any "everyone must switch
> workflows today" kind of coordination, and avoids introducing single points of
> failure by making it easy to replicate everything to mirrored systems.

My parents taught me growing up that you can only ever _improve_: you can never be perfect. 

Needless to say, it’s worth a shot.


Best,

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 9/29/22 15:31, Konstantin Ryabitsev wrote:
> On Thu, Sep 29, 2022 at 02:22:10PM +0000, Artem S. Tashkinov wrote:
>> * Delete all the components.
>> * Leave a catch-all one.
>> * Let bug reports rot because no one will ever see them. Almost just
>> like now. Don't remind me of mailing lists.
>
> This is my proposal, except also:
>
> 1. post all new bugs and comments to a public-inbox feed that people can query
>     via lore.kernel.org and tooling like lei.
>
>> Sarcasm and pain aside, Linus Torvalds himself _via Bugzilla_ has helped
>> me resolve critical issues on several occasions while my messages to
>> LKML were simply _ignored_. Think about that.
>
> In fact, he probably did this by replying to emails, not via the web
> interface.

Nope, I CC'ed him.

>
>> Mailing lists will not work for such a huge project. Period. In the
>> early 90s they worked, but we are 25 years later with millions more
>> users. With a ton more of a ton more complicated hardware.
>
> We've recognized this a while ago, which is why our efforts have been targeted
> at query-based message feeds. Hence, tools like lore.kernel.org and lei. It's
> a work in progress, for sure, but it doesn't require any "everyone must switch
> workflows today" kind of coordination, and avoids introducing single points of
> failure by making it easy to replicate everything to mirrored systems.
>
> -K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Thu, Sep 29, 2022 at 10:54:17AM -0400, Slade Watkins wrote:
> Hey!
> 
> Jumping in here to offer my input...
> 
> > On Sep 29, 2022, at 10:22 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> > 
> > That leaves us with Bugzilla that no one wants to touch and some people
> > actively want to delete altogether. In other words, no central place to
> > report bugs or keep track of them.
> 
> This is the current problem that seems to be appearing here. I get why
> no one wants to touch it, but it doesn’t solve the problem. 
> 
> As you said:
> 
> > I've mentioned several times already that mailing lists are _even worse_
> > in terms of reporting issues. Developers get emails and simply ignore
> > them (for a multitude of reasons).
> 
> It’s 100% true that emails get _buried_ as waves of them come in (LKML
> itself gets hundreds upon hundreds a day, as I’m sure all of you know)
> and it just isn’t something I personally see as viable, especially for
> issues that may or may not be high priority.

E-mails are not that bad to report issues, but they can't provide the
core feature that any bug tracker oughts to have: tracking. There's no
way, with the tools we have at the moment (including public-inbox, b4
and lei), to track the status of bug reports and fixes. Even for patches
we need to rely on patchwork, and that's far from perfect.

When things fall through the cracks (and at the moment it's more of a
sieve with very large holes, if not a bottom-less pot), we mostly assume
that, if the problem is important enough, the submitter will ping time
after time until a fix is produced and merged. There is no way to
produce a list of open issues.

I agree with the comment that was repeated multiple times: it's quite
pointless to improve the tooling if we don't first improve the process,
and find a way to allocate people and time to handling bug reports. Even
if bugzilla has reached EOL upstream, and even if it isn't perfect, the
instance runs today, and gives us a tracker that could be used to design
a proper process and implement it, should we desire to do so. There's no
chicken-and-egg issue to be solved where lack of tooling would prevent
the implementation of a bug tracking process. I'm quite confident that,
if we manage to implement a bug tracking process, we will find a way to
get the tooling we need, be it through bugzilla or something else.

> > Getting back to my first message in this discussion,
> > 
> > * Let's refresh all the components in Bugzilla
> > * Components may not have any people responsible for them at all. Bug
> > reporters will have to CC the people they are interested in.
> > * Let's subscribe the past six months of developers (using git commit logs)
> > * Whoever wants to unsubscribe is free to do so.
> 
> Not a terrible idea to me, though obviously, that’s up for debate.
> 
> > If not for bugzilla, let's use something more modern. I don't know any
> > comparable projects however. Trac is truly horrible. You cannot even
> > unsubscribe from bug reports. Maybe I've missed something. Discourse?
> > Not a bug tracker per se but can certainly work this way.
> 
> Discourse probably isn’t the best fit here, in my opinion. Jira and
> YouTrack are the only ones I personally know of that are similar to
> Bugzilla, although as far as I know, they aren’t open source...

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hey there,

> On Sep 29, 2022, at 12:42 PM, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> 
> E-mails are not that bad to report issues, but they can't provide the
> core feature that any bug tracker oughts to have: tracking. There's no
> way, with the tools we have at the moment (including public-inbox, b4
> and lei), to track the status of bug reports and fixes. Even for patches
> we need to rely on patchwork, and that's far from perfect.

Yeah, I (sort of) changed my mind on this [1], but you’re right: it lacks tracking issues from reported-to-fixed, and unless something was engineered specifically for it, we’re out of luck.

> 
> I agree with the comment that was repeated multiple times: it's quite
> pointless to improve the tooling if we don't first improve the process,
> and find a way to allocate people and time to handling bug reports. Even
> if bugzilla has reached EOL upstream, and even if it isn't perfect, the
> instance runs today, and gives us a tracker that could be used to design
> a proper process and implement it, should we desire to do so. There's no
> chicken-and-egg issue to be solved where lack of tooling would prevent
> the implementation of a bug tracking process. I'm quite confident that,
> if we manage to implement a bug tracking process, we will find a way to
> get the tooling we need, be it through bugzilla or something else.

Right. 

To be honest, I think the best move from here is improve the process first, then worry about everything else when the time comes. Can’t really go any further without first addressing the process in my opinion. The important thing to remember is that nothing implemented will ever be "perfect.” That’s just not possible. However, we _can_ make something that’ll work and continue refining it over time. 

[1] https://lore.kernel.org/lkml/C4935ACC-65C8-4705-B9FF-A1CA0A648B9D@sladewatkins.net/

Cheers,
-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hi,

> On Sep 29, 2022, at 12:06 PM, Artem S. Tashkinov <aros@gmx.com> wrote:
> 
> On 9/29/22 15:31, Konstantin Ryabitsev wrote:
>> 
>> 
>> In fact, he probably did this by replying to emails, not via the web
>> interface.
> 
> Nope, I CC'ed him.

I think you can still reply via email if you’re a Cc. Been a while though. 

Regardless — not the point of the thread so it’s not worth arguing about.

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 29.09.22 15:04, Konstantin Ryabitsev wrote:
> On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
> [...]
> We do have ability to fund development efforts -- LF has been the primary
> sponsor behind public-inbox.org over the past 3 years. However, there must be
> a clear, strong, and well-articulated mandate from the community. From what I
> heard, the vast majority of maintainers simply want a web form that would
> allow someone to:
> 
> 1. clearly state what kernel version they are using
> 2. clearly describe what they were trying to do
> 3. explain what they expected vs. what they got
> 4. attach any files
> 5. give this bug report a unique identifier

Sometimes there are days where I think "let's go down the 'do everything
by mail' rabbit hole some more and couple a pastebin and a somewhat
improved regzbot with an app (usable both locally and on the web) that
helps users preparing a report they can then send with their usual
mailer". And then there are days "ohh, no, that might be a totally
stupid thing to do". :-/

> Then a designated person would look through the bug report and either:
> 
> a. quick-close it (with the usual "talk to your distro" or "don't use a
>    tainted kernel" etc)

I think having some app would be good here, as it could help gathering
everything and catch problems early, to prevent users from spending a
lot of time on preparing a report that will be ignored.

> b. identify the responsible maintainers and notify them
> 
> The hard part is not technical -- the hard part is that "designated person."

+1

> Being a bugmaster is a thankless job that leads to burnout, regardless of how
> well you are paid. Everyone is constantly irate at you from both ends [...]

Tell me about it. Nevertheless I sometimes wonder if I should give it a
try once I got all this regression tracking thing established somewhat
more, as in the end there I'm kind of a bugmaster for regressions already...

> Before we try to fix/replace bugzilla,

Just to be sure: I assume you meant "replacing bugzilla or fixing it for
real" here, and not my band-aid efforts outlined at the start of this
thread? Or do you have a problem with what I proposed to at least make
things less bad for now?

> we really need to figure out the entire
> process and pinpoint who is going to be the one in charge of bug reports. If
> you think that LF should establish a fund for a position like that, then you
> should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk
> to LF management. The IT team will be happy to support you with the tooling,
> but tooling should come second to that -- otherwise we'll just be replacing an
> old and rusty dumpster on fire with a new and shiny dumpster on fire.

+1

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 9/29/22 15:26, James Bottomley wrote:
> On Thu, 2022-09-29 at 12:22 +0000, Artem S. Tashkinov wrote:
>> Let me be brutally honest here, if you're working on the kernel,
>> specially for a large company such as e.g. Intel, you're _expected_
>> to address the issues which are related to the kernel component[s]
>> you're maintaining/developing otherwise it's not "development" it's
>> "I'm dumping my code because my employer pays me to do that". That
>> also means you're expected to address bug reports.
>>
>> It's correct I've tried to help people with bug reports posted on
>> bugzilla.kernel.org but it's a tough task considering that absolute
>> most kernel developers are not signed up, thus most bug reports are
>> never seen by respective developers.
>
> The never seen/never responded to metric is rather bogus.  The sad fact
> is that a lot of bug reports aren't actionable, meaning the reporter
> can't give a reproducer and also can't easily test patches  sometimes
> by luck the maintainers can work out what the issue is but a lot of the
> time they have no idea.  Then there are ton's of bug reports with
> responses like "I think xxx commit fixes your problem, can you test it"
> for which the conversation dies there.  There's also the thundering
> herd problem: some bugs get reported by many different people (as
> different bug reports) but usually the subsystem only engages with one
> to fix the issue.  In theory bugzilla can mark the latter as dups, but
> that requires someone to spend an enormous amount of time on evaluation
> and admin.

   Not only that, many bug reporters simply report something only not to
ever follow up - you ask them for additional information and it looks
like as if they don't receive emails from bugzilla or don't understand
English despite their report being in English.

>
> That's not to say we can't improve our process, it's just to set
> expectations that we're never going to approach anywhere near a perfect
> bug process.  Most of the improvements that worked so far involve
> having someone coach bug reporters through the process of either
> testing patches or reproducing the problem in a more generic
> environment ... which I think most people would agree can't really fall
> wholly on maintainers.


   Bug reporting is an intricate process which requires certain
experience and skills and it's far outside the scope of this
conversation. I still absolutely prefer Bugzilla or a similar bug
tracker to stay. There has to be a place where all the bug reports are
congregated together in an easy to search for form. Someone has proposed
alternatives but I know nothing about them. What I'm looking forward  to
from a new bug tracker:

* An ability to CC anyone and everyone
* Preferably an email interface since some developers just love replying
to emails instead of opening a website
* Categories representing major kernel components

To be honest it feels like refreshing Bugzilla is a lot more easier than
migrating to something new. If I'm given access to it, I could certainly
try to do that.

It's been mentioned that the product is "dead" and "unmaintained" but
that's not what I see on bugzilla.mozilla.org - it has become extremely
powerful. Maybe it's not even Bugzilla but something totally new which
looks like bugzilla.

Other major projects continue to use Bugzilla seemingly without big
problems:

* KDE ( https://bugs.kde.org/ )
* GCC ( https://gcc.gnu.org/bugzilla/ )
* Wine ( https://bugs.winehq.org/ )

It feels to me we just need a dedicated Bugzilla maintainer. That's it.
I could probably do it.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 9/30/22 08:47, Thorsten Leemhuis wrote:
> On 29.09.22 15:04, Konstantin Ryabitsev wrote:
>> On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
>> [...]
>> We do have ability to fund development efforts -- LF has been the primary
>> sponsor behind public-inbox.org over the past 3 years. However, there must be
>> a clear, strong, and well-articulated mandate from the community. From what I
>> heard, the vast majority of maintainers simply want a web form that would
>> allow someone to:
>>
>> 1. clearly state what kernel version they are using
>> 2. clearly describe what they were trying to do
>> 3. explain what they expected vs. what they got
>> 4. attach any files
>> 5. give this bug report a unique identifier
>
> Sometimes there are days where I think "let's go down the 'do everything
> by mail' rabbit hole some more and couple a pastebin and a somewhat
> improved regzbot with an app (usable both locally and on the web) that
> helps users preparing a report they can then send with their usual
> mailer". And then there are days "ohh, no, that might be a totally
> stupid thing to do". :-/

Emails are absolutely horrible in terms of keeping track of the state of
the issue. Who has replied? Who has provided the necessary data? Where
can this data be found? What if a person has forgotten to "Reply All"
and instead clicked "Reply"? Hell, no. Then people get swamped with
their own emails, the previous email from this discussion went straight
to SPAM for my email provider. It's too easy to lose track of everything.

The kernel bugzilla has helped resolve critical issues and add
impressive features with dozens of people collaborating. This is nearly
impossible to carry out using email except for dedicated developers
working on something.

In the LKML and other Open Source mailing lists I've seen a ton of RFC
patches with no follow up. Even core developers themselves aren't
particularly enjoying the format. And those patches often perish and
work goes to waste.

>
>> Then a designated person would look through the bug report and either:
>>
>> a. quick-close it (with the usual "talk to your distro" or "don't use a
>>     tainted kernel" etc)
>
> I think having some app would be good here, as it could help gathering
> everything and catch problems early, to prevent users from spending a
> lot of time on preparing a report that will be ignored.
>
>> b. identify the responsible maintainers and notify them
>>
>> The hard part is not technical -- the hard part is that "designated person."
>
> +1
>
>> Being a bugmaster is a thankless job that leads to burnout, regardless of how
>> well you are paid. Everyone is constantly irate at you from both ends [...]
>
> Tell me about it. Nevertheless I sometimes wonder if I should give it a
> try once I got all this regression tracking thing established somewhat
> more, as in the end there I'm kind of a bugmaster for regressions already...
>
>> Before we try to fix/replace bugzilla,
>
> Just to be sure: I assume you meant "replacing bugzilla or fixing it for
> real" here, and not my band-aid efforts outlined at the start of this
> thread? Or do you have a problem with what I proposed to at least make
> things less bad for now?
>
>> we really need to figure out the entire
>> process and pinpoint who is going to be the one in charge of bug reports. If
>> you think that LF should establish a fund for a position like that, then you
>> should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk
>> to LF management. The IT team will be happy to support you with the tooling,
>> but tooling should come second to that -- otherwise we'll just be replacing an
>> old and rusty dumpster on fire with a new and shiny dumpster on fire.

Bugzilla with all its issues is still super convenient.

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 29.09.22 18:42, Laurent Pinchart wrote:
> On Thu, Sep 29, 2022 at 10:54:17AM -0400, Slade Watkins wrote:
>>> On Sep 29, 2022, at 10:22 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
>>>
>>> I've mentioned several times already that mailing lists are _even worse_
>>> in terms of reporting issues. Developers get emails and simply ignore
>>> them (for a multitude of reasons).
>>
>> It’s 100% true that emails get _buried_ as waves of them come in (LKML
>> itself gets hundreds upon hundreds a day, as I’m sure all of you know)
>> and it just isn’t something I personally see as viable, especially for
>> issues that may or may not be high priority.
> 
> E-mails are not that bad to report issues, but they can't provide the
> core feature that any bug tracker oughts to have: tracking. There's no
> way, with the tools we have at the moment (including public-inbox, b4
> and lei), to track the status of bug reports and fixes.

Well, I'd disagree partially with that, as my regression tracking bot
"regzbot"
(https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
; https://linux-regtracking.leemhuis.info/regzbot/mainline/) does
exactly does that: tracking, by connect the dots (e.g. monitoring
replies to a report as well recording when patches are posted or
committed that link to the report using Link: tags), while making sure
nothing important is forgotten. But sure, it's still very rough and
definitely not a full bug-tracker (my goal is/was to not create yet
another one) and needs quite a bit of hand holding from my side. And I
only use it for regressions and not for bugs (on purpose).

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hi,

> On Sep 30, 2022, at 5:03 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> On 9/30/22 08:47, Thorsten Leemhuis wrote:
>> On 29.09.22 15:04, Konstantin Ryabitsev wrote:
	[trimmed]
>> Sometimes there are days where I think "let's go down the 'do everything
>> by mail' rabbit hole some more and couple a pastebin and a somewhat
>> improved regzbot with an app (usable both locally and on the web) that
>> helps users preparing a report they can then send with their usual
>> mailer". And then there are days "ohh, no, that might be a totally
>> stupid thing to do". :-/
> 
> Emails are absolutely horrible in terms of keeping track of the state of
> the issue. Who has replied? Who has provided the necessary data? Where
> can this data be found? What if a person has forgotten to "Reply All"
> and instead clicked "Reply"? Hell, no. Then people get swamped with
> their own emails, the previous email from this discussion went straight
> to SPAM for my email provider. It's too easy to lose track of everything.

Email deliverability and spam filters are certainly something to consider. (Thanks email providers.)

> 
> The kernel bugzilla has helped resolve critical issues and add
> impressive features with dozens of people collaborating. This is nearly
> impossible to carry out using email except for dedicated developers
> working on something.

Exactly.

>> 
>> we really need to figure out the entire
>> process and pinpoint who is going to be the one in charge of bug reports. If
>> you think that LF should establish a fund for a position like that, then you
>> should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk
>> to LF management. The IT team will be happy to support you with the tooling,
>> but tooling should come second to that -- otherwise we'll just be replacing an
>> old and rusty dumpster on fire with a new and shiny dumpster on fire.
> 
> Bugzilla with all its issues is still super convenient.


+1, I don’t think the solution longterm is to _not_ have a system like Bugzilla for this reason. Emails can certainly be sent from the system but it should continue existing.

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

Hi Thorsten,

On Fri, Sep 30, 2022 at 11:35:16AM +0200, Thorsten Leemhuis wrote:
> On 29.09.22 18:42, Laurent Pinchart wrote:
> > On Thu, Sep 29, 2022 at 10:54:17AM -0400, Slade Watkins wrote:
> >>> On Sep 29, 2022, at 10:22 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> >>>
> >>> I've mentioned several times already that mailing lists are _even worse_
> >>> in terms of reporting issues. Developers get emails and simply ignore
> >>> them (for a multitude of reasons).
> >>
> >> It’s 100% true that emails get _buried_ as waves of them come in (LKML
> >> itself gets hundreds upon hundreds a day, as I’m sure all of you know)
> >> and it just isn’t something I personally see as viable, especially for
> >> issues that may or may not be high priority.
> > 
> > E-mails are not that bad to report issues, but they can't provide the
> > core feature that any bug tracker oughts to have: tracking. There's no
> > way, with the tools we have at the moment (including public-inbox, b4
> > and lei), to track the status of bug reports and fixes.
> 
> Well, I'd disagree partially with that, as my regression tracking bot
> "regzbot"
> (https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
> ; https://linux-regtracking.leemhuis.info/regzbot/mainline/) does
> exactly does that: tracking, by connect the dots (e.g. monitoring
> replies to a report as well recording when patches are posted or
> committed that link to the report using Link: tags), while making sure
> nothing important is forgotten. But sure, it's still very rough and
> definitely not a full bug-tracker (my goal is/was to not create yet
> another one) and needs quite a bit of hand holding from my side. And I
> only use it for regressions and not for bugs (on purpose).

Patchwork does something similar for patches, and I agree that it would
be possible to use e-mail to manage and track bug reports with tools on
top (and don't worry, I'm not asking for regzbot to be turned into a bug
tracker :-)). It however has to rely on lots of heuristics at the
moment, as the data we exchange over e-mail is free-formed and lacks
structure. I've been dreaming of support for structured data in e-mails,
but that's a pipe dream really.

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Fri, Sep 30, 2022 at 09:03:39AM +0000, Artem S. Tashkinov wrote:
> On 9/30/22 08:47, Thorsten Leemhuis wrote:
> > On 29.09.22 15:04, Konstantin Ryabitsev wrote:
> >> On Thu, Sep 29, 2022 at 12:22:35PM +0000, Artem S. Tashkinov wrote:
> >> [...]
> >> We do have ability to fund development efforts -- LF has been the primary
> >> sponsor behind public-inbox.org over the past 3 years. However, there must be
> >> a clear, strong, and well-articulated mandate from the community. From what I
> >> heard, the vast majority of maintainers simply want a web form that would
> >> allow someone to:
> >>
> >> 1. clearly state what kernel version they are using
> >> 2. clearly describe what they were trying to do
> >> 3. explain what they expected vs. what they got
> >> 4. attach any files
> >> 5. give this bug report a unique identifier
> >
> > Sometimes there are days where I think "let's go down the 'do everything
> > by mail' rabbit hole some more and couple a pastebin and a somewhat
> > improved regzbot with an app (usable both locally and on the web) that
> > helps users preparing a report they can then send with their usual
> > mailer". And then there are days "ohh, no, that might be a totally
> > stupid thing to do". :-/
> 
> Emails are absolutely horrible in terms of keeping track of the state of
> the issue. Who has replied? Who has provided the necessary data? Where
> can this data be found? What if a person has forgotten to "Reply All"
> and instead clicked "Reply"?

E-mail *clients* are horrible to keep track of state. E-mail itself, as
in RFC822 (and newer), SMTP and other protocols, only handle transport
of data. As the data within the e-mail body is free-formed, and wasn't
meant to track items and their state, clients never evolved in that
direction. This could (possibly) be (partially) fixed, but likely at a
very high development cost, and getting users on board would be very
hard too. I do agree with Thorsten though, I'm often tempted to go
through the "let's do everything by e-mail" path. More than 10 years
ago, I worked for a large OEM that had an e-mail frontend for
integration and testing. You would send a specially-crafted e-mail to a
bot, with a base image version, plus a list of repositories and
branches, and the bot would build a new image for you, run all the
automated integration tests, and if you requested it (and had permission
to do so), would push the image down a manual testing queue. It was just
magic.

> Hell, no. Then people get swamped with their own emails,

Bugzilla won't solve this. The huge elephant in the room is that most
maintainers are overworked. Whether a bug report arrives in my mailbox
as an e-mail straight from the reporter or from a bug tracker will make
very little difference if I don't have time to look into it (I would
even argue that bug trackers are even worse there: if I'm really short
of time, I'm more likely to prioritize replying to e-mails instead of
having to open a link in a web browser).

As long as we don't address the maintainer bottleneck in the kernel, bug
tracking will suffer.

> the previous email from this discussion went straight
> to SPAM for my email provider. It's too easy to lose track of everything.
> 
> The kernel bugzilla has helped resolve critical issues and add
> impressive features with dozens of people collaborating. This is nearly
> impossible to carry out using email except for dedicated developers
> working on something.
> 
> In the LKML and other Open Source mailing lists I've seen a ton of RFC
> patches with no follow up. Even core developers themselves aren't
> particularly enjoying the format. And those patches often perish and
> work goes to waste.
> 
> >> Then a designated person would look through the bug report and either:
> >>
> >> a. quick-close it (with the usual "talk to your distro" or "don't use a
> >>     tainted kernel" etc)
> >
> > I think having some app would be good here, as it could help gathering
> > everything and catch problems early, to prevent users from spending a
> > lot of time on preparing a report that will be ignored.
> >
> >> b. identify the responsible maintainers and notify them
> >>
> >> The hard part is not technical -- the hard part is that "designated person."
> >
> > +1
> >
> >> Being a bugmaster is a thankless job that leads to burnout, regardless of how
> >> well you are paid. Everyone is constantly irate at you from both ends [...]
> >
> > Tell me about it. Nevertheless I sometimes wonder if I should give it a
> > try once I got all this regression tracking thing established somewhat
> > more, as in the end there I'm kind of a bugmaster for regressions already...
> >
> >> Before we try to fix/replace bugzilla,
> >
> > Just to be sure: I assume you meant "replacing bugzilla or fixing it for
> > real" here, and not my band-aid efforts outlined at the start of this
> > thread? Or do you have a problem with what I proposed to at least make
> > things less bad for now?
> >
> >> we really need to figure out the entire
> >> process and pinpoint who is going to be the one in charge of bug reports. If
> >> you think that LF should establish a fund for a position like that, then you
> >> should probably approach LF fellows (Greg KH, Shuah Khan), who can then talk
> >> to LF management. The IT team will be happy to support you with the tooling,
> >> but tooling should come second to that -- otherwise we'll just be replacing an
> >> old and rusty dumpster on fire with a new and shiny dumpster on fire.
> 
> Bugzilla with all its issues is still super convenient.

-- 
Regards,

Laurent Pinchart

RE: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Bird, Tim]

> -----Original Message-----
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Hi Thorsten,
> 
> On Fri, Sep 30, 2022 at 11:35:16AM +0200, Thorsten Leemhuis wrote:
> > On 29.09.22 18:42, Laurent Pinchart wrote:
> > > On Thu, Sep 29, 2022 at 10:54:17AM -0400, Slade Watkins wrote:
> > >>> On Sep 29, 2022, at 10:22 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> > >>>
> > >>> I've mentioned several times already that mailing lists are _even worse_
> > >>> in terms of reporting issues. Developers get emails and simply ignore
> > >>> them (for a multitude of reasons).
> > >>
> > >> It’s 100% true that emails get _buried_ as waves of them come in (LKML
> > >> itself gets hundreds upon hundreds a day, as I’m sure all of you know)
> > >> and it just isn’t something I personally see as viable, especially for
> > >> issues that may or may not be high priority.
> > >
> > > E-mails are not that bad to report issues, but they can't provide the
> > > core feature that any bug tracker oughts to have: tracking. There's no
> > > way, with the tools we have at the moment (including public-inbox, b4
> > > and lei), to track the status of bug reports and fixes.
> >
> > Well, I'd disagree partially with that, as my regression tracking bot
> > "regzbot"
> > (https://urldefense.com/v3/__https://gitlab.com/knurd42/regzbot/-
> /blob/main/docs/getting_started.md__;!!JmoZiZGBv3RvKRSx!7f8O2QaGyWgxASwg1_bxsV53uWPINzzBa_MLMZMooa6qL6jdk8ZBVYrB_
> mypjw0H3yv5IPdNJ2qQThzMLKbrOUQMFMO1x2V2$
> > ; https://urldefense.com/v3/__https://linux-
> regtracking.leemhuis.info/regzbot/mainline/__;!!JmoZiZGBv3RvKRSx!7f8O2QaGyWgxASwg1_bxsV53uWPINzzBa_MLMZMooa6qL6jdk8Z
> BVYrB_mypjw0H3yv5IPdNJ2qQThzMLKbrOUQMFKROzSJY$  ) does
> > exactly does that: tracking, by connect the dots (e.g. monitoring
> > replies to a report as well recording when patches are posted or
> > committed that link to the report using Link: tags), while making sure
> > nothing important is forgotten. But sure, it's still very rough and
> > definitely not a full bug-tracker (my goal is/was to not create yet
> > another one) and needs quite a bit of hand holding from my side. And I
> > only use it for regressions and not for bugs (on purpose).
> 
> Patchwork does something similar for patches, and I agree that it would
> be possible to use e-mail to manage and track bug reports with tools on
> top (and don't worry, I'm not asking for regzbot to be turned into a bug
> tracker :-)). It however has to rely on lots of heuristics at the
> moment, as the data we exchange over e-mail is free-formed and lacks
> structure. I've been dreaming of support for structured data in e-mails,
> but that's a pipe dream really.

E-mails sent from a web interface could have as much structure as you'd like.
So one avenue would be to set up a nice interface for bug reporting, that just
delivered the form data in e-mail format to the proposed bug-receiving mail list.

Also, if an e-mail receiver (something automated) gave a quick response on missing fields, I think
you could quickly train users (even first-time bug submitters) to provide required
data, even if they're sending from a free-form e-mail client.

Just my 2 cents.

 -- Tim

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 9/30/22 16:19, Bird, Tim wrote:
> E-mails sent from a web interface could have as much structure as you'd like.
> So one avenue would be to set up a nice interface for bug reporting, that just
> delivered the form data in e-mail format to the proposed bug-receiving mail list.
>
> Also, if an e-mail receiver (something automated) gave a quick response on missing fields, I think
> you could quickly train users (even first-time bug submitters) to provide required
> data, even if they're sending from a free-form e-mail client.
>
> Just my 2 cents.
>
>   -- Tim

Debian uses an email based bug tracker and you know what? Most people
avoid it like a plague. It's a hell on earth to use. Ubunutu's Launchpad
which looks and feels like Bugzilla is a hundred times more popular.

Sometimes programmers have to realize that most people around are not as
smart as they are.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Fri, Sep 30, 2022 at 04:19:56PM +0000, Bird, Tim wrote:
> > -----Original Message-----
> > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > 
> > Hi Thorsten,
> > 
> > On Fri, Sep 30, 2022 at 11:35:16AM +0200, Thorsten Leemhuis wrote:
> > > On 29.09.22 18:42, Laurent Pinchart wrote:
> > > > On Thu, Sep 29, 2022 at 10:54:17AM -0400, Slade Watkins wrote:
> > > >>> On Sep 29, 2022, at 10:22 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> > > >>>
> > > >>> I've mentioned several times already that mailing lists are _even worse_
> > > >>> in terms of reporting issues. Developers get emails and simply ignore
> > > >>> them (for a multitude of reasons).
> > > >>
> > > >> It’s 100% true that emails get _buried_ as waves of them come in (LKML
> > > >> itself gets hundreds upon hundreds a day, as I’m sure all of you know)
> > > >> and it just isn’t something I personally see as viable, especially for
> > > >> issues that may or may not be high priority.
> > > >
> > > > E-mails are not that bad to report issues, but they can't provide the
> > > > core feature that any bug tracker oughts to have: tracking. There's no
> > > > way, with the tools we have at the moment (including public-inbox, b4
> > > > and lei), to track the status of bug reports and fixes.
> > >
> > > Well, I'd disagree partially with that, as my regression tracking bot
> > > "regzbot"
> > > (https://urldefense.com/v3/__https://gitlab.com/knurd42/regzbot/-
> > /blob/main/docs/getting_started.md__;!!JmoZiZGBv3RvKRSx!7f8O2QaGyWgxASwg1_bxsV53uWPINzzBa_MLMZMooa6qL6jdk8ZBVYrB_
> > mypjw0H3yv5IPdNJ2qQThzMLKbrOUQMFMO1x2V2$
> > > ; https://urldefense.com/v3/__https://linux-
> > regtracking.leemhuis.info/regzbot/mainline/__;!!JmoZiZGBv3RvKRSx!7f8O2QaGyWgxASwg1_bxsV53uWPINzzBa_MLMZMooa6qL6jdk8Z
> > BVYrB_mypjw0H3yv5IPdNJ2qQThzMLKbrOUQMFKROzSJY$  ) does
> > > exactly does that: tracking, by connect the dots (e.g. monitoring
> > > replies to a report as well recording when patches are posted or
> > > committed that link to the report using Link: tags), while making sure
> > > nothing important is forgotten. But sure, it's still very rough and
> > > definitely not a full bug-tracker (my goal is/was to not create yet
> > > another one) and needs quite a bit of hand holding from my side. And I
> > > only use it for regressions and not for bugs (on purpose).
> > 
> > Patchwork does something similar for patches, and I agree that it would
> > be possible to use e-mail to manage and track bug reports with tools on
> > top (and don't worry, I'm not asking for regzbot to be turned into a bug
> > tracker :-)). It however has to rely on lots of heuristics at the
> > moment, as the data we exchange over e-mail is free-formed and lacks
> > structure. I've been dreaming of support for structured data in e-mails,
> > but that's a pipe dream really.
> 
> E-mails sent from a web interface could have as much structure as you'd like.
> So one avenue would be to set up a nice interface for bug reporting, that just
> delivered the form data in e-mail format to the proposed bug-receiving mail list.
> 
> Also, if an e-mail receiver (something automated) gave a quick response on missing fields, I think
> you could quickly train users (even first-time bug submitters) to provide required
> data, even if they're sending from a free-form e-mail client.

In my dream, we could even teach some mail clients to do so. There's a
bit of chicken and egg issue of course, but if the form data is in a
human-writable form, it may be possible to start on the server side
first, and then address clients.

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Fri, Sep 30, 2022 at 04:34:08PM +0000, Artem S. Tashkinov wrote:
> On 9/30/22 16:19, Bird, Tim wrote:
> > E-mails sent from a web interface could have as much structure as you'd like.
> > So one avenue would be to set up a nice interface for bug reporting, that just
> > delivered the form data in e-mail format to the proposed bug-receiving mail list.
> >
> > Also, if an e-mail receiver (something automated) gave a quick response on missing fields, I think
> > you could quickly train users (even first-time bug submitters) to provide required
> > data, even if they're sending from a free-form e-mail client.
> >
> > Just my 2 cents.
> >
> >   -- Tim
> 
> Debian uses an email based bug tracker and you know what? Most people
> avoid it like a plague. It's a hell on earth to use. Ubunutu's Launchpad
> which looks and feels like Bugzilla is a hundred times more popular.

It would be pretty sad if the only options we could come up with for bug
tracking would be either popular with reporters and ignored by
maintainers, or the other way around. Ideally we wouldn't have to decide
which of those two classes of users to prioritize, but I fear that,
given resource starvation, we'll have to make a decision there that will
be unpopular with one of the two sides.

> Sometimes programmers have to realize that most people around are not as
> smart as they are.

I wouldn't equate familiarity with classes of tools (and related usage
habbits) and intelligence. Some tools may be easier to learn and use,
but it doesn't mean they're good for the problem at hand. I used to joke
several years ago that the younger generation will force older
maintainers to switch to doing code review on facebook (nowadays I would
probably say tik-tok). And then
https://github.blog/2021-05-13-video-uploads-available-github/ happened.
And that https://www.videocode.review/ (meanwhile, git..b still don't
support commenting on a commit message in a review).

-- 
Regards,

Laurent Pinchart

RE: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Luck, Tony]

> E-mails sent from a web interface could have as much structure as you'd like.
> So one avenue would be to set up a nice interface for bug reporting, that just
> delivered the form data in e-mail format to the proposed bug-receiving mail list.

Web interfaces have the advantage that they can be full of boxes which indicate
useful details to supply. Like what kernel version? Did this work on an older version,
is so, which one? Which CPU vendor/model are you using? Is there an error message?
Are there warnings in the console log before the error? Can you upload a full console log?
Does this happen repeatably? What are the steps to reproduce?

Etc.etc.

Sometimes it takes a few round trips by e-mail to establish the baseline facts.

-Tony

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Artem, all,

> On Sep 30, 2022, at 12:34 PM, Artem S. Tashkinov <aros@gmx.com> wrote:
> Debian uses an email based bug tracker and you know what? Most people
> avoid it like a plague. It's a hell on earth to use. Ubunutu's Launchpad
> which looks and feels like Bugzilla is a hundred times more popular.

Yeah, Ubuntu’s Launchpad instance is definitely easier to navigate than Bugzilla and has more info at a glance (when looking at individual bug reports.) Do I necessarily think they look and feel the same? Nah. But, hey, it’s all subjective so it’s cool!

Ultimately I’m conflicted here (even my own opinions have already changed twice since I jumped in on the discussion.) Some say email is terrible, others say it’s the way they want to do it. Because that’s all subjective: that was bound to happen, of course. My take is that if the goal is to please *everyone*, we’re not going to get anywhere. 

Email — imo — is good for discussions, but not for reporting bugs. Web has upsides of being easier to navigate (sometimes faster) with just a few clicks/keyboard shortcuts and some words to describe an issue, steps to reproduce, kernel versions it affects, etc.

But no matter what system (email, web, etc.) you use — there will always be things that gets lost, whether that’s because someone didn’t see something and/or it got buried by something else more urgent. Sadly, that's just going to happen, and the best thing to do is improve it so that it’s _less likely_ to do so.

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hey:

> On Sep 30, 2022, at 1:28 PM, Luck, Tony <tony.luck@intel.com> wrote:
> 
>> E-mails sent from a web interface could have as much structure as you'd like.
>> So one avenue would be to set up a nice interface for bug reporting, that just
>> delivered the form data in e-mail format to the proposed bug-receiving mail list.
> 
> Web interfaces have the advantage that they can be full of boxes which indicate
> useful details to supply. Like what kernel version? Did this work on an older version,
> is so, which one? Which CPU vendor/model are you using? Is there an error message?
> Are there warnings in the console log before the error? Can you upload a full console log?
> Does this happen repeatably? What are the steps to reproduce?

Not to mention, they have the advantage of being faster, in a lot of cases. (Sometimes, just a few keystrokes and an issue is filed. Saves everyone time.)

> Sometimes it takes a few round trips by e-mail to establish the baseline facts.

+1 to this, it’s much easier to have that information immediately at hand. It helps no one when you have to wait for it all to trickle via email taking days if not weeks. Waste of everyone’s time, in my mind.

As I’ve said—my opinion is that email is good for discussions, but  is not great for bug reports (at least, the initial ones that have all the base information.) 

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Randy Dunlap]

On 9/30/22 10:28, Luck, Tony wrote:
>> E-mails sent from a web interface could have as much structure as you'd like.
>> So one avenue would be to set up a nice interface for bug reporting, that just
>> delivered the form data in e-mail format to the proposed bug-receiving mail list.
> 
> Web interfaces have the advantage that they can be full of boxes which indicate
> useful details to supply. Like what kernel version? Did this work on an older version,
> is so, which one? Which CPU vendor/model are you using? Is there an error message?
> Are there warnings in the console log before the error? Can you upload a full console log?
> Does this happen repeatably? What are the steps to reproduce?
> 
> Etc.etc.

We have Documentation for all of that, but (a) people don't read documentation
and/or (b) it's too longwinded (not brief).

> Sometimes it takes a few round trips by e-mail to establish the baseline facts.

Ack.

-- 
~Randy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 30.09.22 22:04, Randy Dunlap wrote:
> 
> 
> On 9/30/22 10:28, Luck, Tony wrote:
>>> E-mails sent from a web interface could have as much structure as you'd like.
>>> So one avenue would be to set up a nice interface for bug reporting, that just
>>> delivered the form data in e-mail format to the proposed bug-receiving mail list.
>>
>> Web interfaces have the advantage that they can be full of boxes which indicate
>> useful details to supply. Like what kernel version? Did this work on an older version,
>> is so, which one? Which CPU vendor/model are you using? Is there an error message?
>> Are there warnings in the console log before the error? Can you upload a full console log?
>> Does this happen repeatably? What are the steps to reproduce?
>>
>> Etc.etc.
> 
> We have Documentation for all of that, but (a) people don't read documentation
> and/or (b) it's too longwinded (not brief).

Yup. But as the one that is partly (mainly?) responsible for "(b)",
please allow me to quote the last sentence of reporting-issues.rst here:

"""The main author of this text hopes documenting the state of the art
will lay some groundwork to improve the situation over time."""

IOW: I really hope we over time can shorten that text somewhat (or even
a lot?) by...

* making some things a lot easier that currently are unnecessarily hard

* hiding some things in a reporting app or something like that (ideally
usable on the web and locally) that only bothers reporters with tainted
status, bisection, decoding of stack-traces, and things like that if
they are relevant in the particular case

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Theodore Ts'o]

On Fri, Sep 30, 2022 at 07:47:26PM +0300, Laurent Pinchart wrote:
> > Debian uses an email based bug tracker and you know what? Most people
> > avoid it like a plague. It's a hell on earth to use. Ubunutu's Launchpad
> > which looks and feels like Bugzilla is a hundred times more popular.
> 
> It would be pretty sad if the only options we could come up with for bug
> tracking would be either popular with reporters and ignored by
> maintainers, or the other way around. Ideally we wouldn't have to decide
> which of those two classes of users to prioritize, but I fear that,
> given resource starvation, we'll have to make a decision there that will
> be unpopular with one of the two sides.

Funny thing.  I've largely given up on getting any kind of useful bug
report from Launchpad, so I've largely ignored it.  In contast, the
bug reports I get for e2fsprogs from Debian are generally far more
actionable, with bug reports that have all of the data so I can
actually root cause the problem, and help the user.

So Launchpad may be pretty, but perhaps because of selection bias, the
bug reports I've seen there are generally a waste of my time, and if
I'm going to choose which users I'm going to help for ***free***, it's
going to be the one which is far less frustrating to me as the
volunteer.

"100 times more popular" is not necessarily a feature if what we get
is 1000 times the noise.

> > Sometimes programmers have to realize that most people around are not as
> > smart as they are.

Given my personal experience having seen bug repors from Launchpad,
I'm sure that's true.  The question is whether I want to engage with
people who can't me decent bug reports...  (or who are just asking for
free consulting help; there's a lot of that too).

     	    	       	     	       - Ted

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hey there:

> On Sep 30, 2022, at 9:18 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> 
> On Fri, Sep 30, 2022 at 07:47:26PM +0300, Laurent Pinchart wrote:
>>> Debian uses an email based bug tracker and you know what? Most people
>>> avoid it like a plague. It's a hell on earth to use. Ubunutu's Launchpad
>>> which looks and feels like Bugzilla is a hundred times more popular.
>> 
>> It would be pretty sad if the only options we could come up with for bug
>> tracking would be either popular with reporters and ignored by
>> maintainers, or the other way around. Ideally we wouldn't have to decide
>> which of those two classes of users to prioritize, but I fear that,
>> given resource starvation, we'll have to make a decision there that will
>> be unpopular with one of the two sides.
> 
> Funny thing.  I've largely given up on getting any kind of useful bug
> report from Launchpad, so I've largely ignored it.  In contast, the
> bug reports I get for e2fsprogs from Debian are generally far more
> actionable, with bug reports that have all of the data so I can
> actually root cause the problem, and help the user.

Yeah, this all comes down to personal preference and experience. For instance, that hasn’t been my experience and I happen to like Launchpad’s layout for reports, but I can see why you’d feel that way. (Different roles, different experiences, so-to-speak.)

> So Launchpad may be pretty, but perhaps because of selection bias, the
> bug reports I've seen there are generally a waste of my time, and if
> I'm going to choose which users I'm going to help for ***free***, it's
> going to be the one which is far less frustrating to me as the
> volunteer.

Yep, valid point there. 

> "100 times more popular" is not necessarily a feature if what we get
> is 1000 times the noise.

I mean, I get it. And, obviously, something like Launchpad isn’t going to scale the way that it needs to in order for it to work for this.  

Not everyone is going to like whatever solution is put in place, if there’s even any solution put in place: there will always be complaints from folks.

*Sigh.*

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

On Fri, Sep 30, 2022 at 09:18:13PM -0400, Theodore Ts'o wrote:
> On Fri, Sep 30, 2022 at 07:47:26PM +0300, Laurent Pinchart wrote:
> > > Debian uses an email based bug tracker and you know what? Most people
> > > avoid it like a plague. It's a hell on earth to use. Ubunutu's Launchpad
> > > which looks and feels like Bugzilla is a hundred times more popular.
> > 
> > It would be pretty sad if the only options we could come up with for bug
> > tracking would be either popular with reporters and ignored by
> > maintainers, or the other way around. Ideally we wouldn't have to decide
> > which of those two classes of users to prioritize, but I fear that,
> > given resource starvation, we'll have to make a decision there that will
> > be unpopular with one of the two sides.
> 
> Funny thing.  I've largely given up on getting any kind of useful bug
> report from Launchpad, so I've largely ignored it.  In contast, the
> bug reports I get for e2fsprogs from Debian are generally far more
> actionable, with bug reports that have all of the data so I can
> actually root cause the problem, and help the user.
> 

So no matter how the bug tracker interface is, the etiquette is:
Whenever something buggy happens, try to gather all information related
to that event (reproduction steps and reproducer, logs, crash dumps,
etc), then file the polished report. From your experience, it seems like
Debian people knows it.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

Here are two other issues which absolutely suck in terms of dealing with
the kernel.

- 1 -

I have a 20+ years experience in IT and some kernel issues are just
baffling in terms of trying to understand what to do about them.

Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274

What should I do about that? Who's responsible for this? Who should I CC?

And this is an issue which is easy to describe and identify.

- 2 -

Here's another one which is outright puzzling:

You run: dmesg -t --level=emerg,crit,err

And you see some non-descript errors of some kernel subsystems seemingly
failing or being unhappy about your hardware. Errors are as cryptic as
humanly possible, you don't even know what part of kernel has produced them.

OK, as a "power" user I download the kernel source, run `grep -R message
/tmp/linux-5.19` and there are _multiple_ different modules and places
which contain this message.

I'm lost. Send this to LKML? Did that in the long past, no one cared, I
stopped.

Here's what I'm getting with Linux 5.19.12:

platform wdat_wdt: failed to claim resource 5: [mem
0x00000000-0xffffffff7fffffff]
ACPI: watchdog: Device creation failed: -16
ACPI BIOS Error (bug): Could not resolve symbol
[\_SB.PCI0.XHC.RHUB.TPLD], AE_NOT_FOUND (20220331/psargs-330)
ACPI Error: Aborting method \_SB.UBTC.CR01._PLD due to previous error
(AE_NOT_FOUND) (20220331/psparse-529)
platform MSFT0101:00: failed to claim resource 1: [mem
0xfed40000-0xfed40fff]
acpi MSFT0101:00: platform device creation failed: -16
lis3lv02d: unknown sensor type 0x0

Are they serious? Should they be reported or not? Is my laptop properly
working? I have no clue at all.

---

While we've been talking about bugzilla I thought it would be pertinent
to bring this up.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Greg KH]

On Sat, Oct 01, 2022 at 10:30:22AM +0000, Artem S. Tashkinov wrote:
> Here are two other issues which absolutely suck in terms of dealing with
> the kernel.
> 
> - 1 -
> 
> I have a 20+ years experience in IT and some kernel issues are just
> baffling in terms of trying to understand what to do about them.
> 
> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
> 
> What should I do about that? Who's responsible for this? Who should I CC?

Input subsystem.

> Here's what I'm getting with Linux 5.19.12:
> 
> platform wdat_wdt: failed to claim resource 5: [mem
> 0x00000000-0xffffffff7fffffff]

$ find . | grep wdat_wdt
./drivers/watchdog/wdat_wdt.c
$ ./scripts/get_maintainer.pl --file ./drivers/watchdog/wdat_wdt.c
Wim Van Sebroeck <wim@linux-watchdog.org> (maintainer:WATCHDOG DEVICE DRIVERS)
Guenter Roeck <linux@roeck-us.net> (maintainer:WATCHDOG DEVICE DRIVERS)
linux-watchdog@vger.kernel.org (open list:WATCHDOG DEVICE DRIVERS)
linux-kernel@vger.kernel.org (open list)

> ACPI: watchdog: Device creation failed: -16
> ACPI BIOS Error (bug): Could not resolve symbol
> [\_SB.PCI0.XHC.RHUB.TPLD], AE_NOT_FOUND (20220331/psargs-330)
> ACPI Error: Aborting method \_SB.UBTC.CR01._PLD due to previous error
> (AE_NOT_FOUND) (20220331/psparse-529)

Send to ACPI list as described in the MAINTAINERS file.

thanks,

greg k-h

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/1/22 10:39, Greg KH wrote:
> On Sat, Oct 01, 2022 at 10:30:22AM +0000, Artem S. Tashkinov wrote:
>> Here are two other issues which absolutely suck in terms of dealing with
>> the kernel.
>>
>> - 1 -
>>
>> I have a 20+ years experience in IT and some kernel issues are just
>> baffling in terms of trying to understand what to do about them.
>>
>> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
>>
>> What should I do about that? Who's responsible for this? Who should I CC?
>
> Input subsystem.

It's great you've replied immediately, what about hundreds or even
thousands of other bug reports where people have no clue who has to be
CC'ed?

>
>> Here's what I'm getting with Linux 5.19.12:
>>
>> platform wdat_wdt: failed to claim resource 5: [mem
>> 0x00000000-0xffffffff7fffffff]
>
> $ find . | grep wdat_wdt
> ./drivers/watchdog/wdat_wdt.c
> $ ./scripts/get_maintainer.pl --file ./drivers/watchdog/wdat_wdt.c
> Wim Van Sebroeck <wim@linux-watchdog.org> (maintainer:WATCHDOG DEVICE DRIVERS)
> Guenter Roeck <linux@roeck-us.net> (maintainer:WATCHDOG DEVICE DRIVERS)
> linux-watchdog@vger.kernel.org (open list:WATCHDOG DEVICE DRIVERS)
> linux-kernel@vger.kernel.org (open list)
>
>> ACPI: watchdog: Device creation failed: -16
>> ACPI BIOS Error (bug): Could not resolve symbol
>> [\_SB.PCI0.XHC.RHUB.TPLD], AE_NOT_FOUND (20220331/psargs-330)
>> ACPI Error: Aborting method \_SB.UBTC.CR01._PLD due to previous error
>> (AE_NOT_FOUND) (20220331/psparse-529)
>
> Send to ACPI list as described in the MAINTAINERS file.

The more important question is whether I should even bother anyone in
the first place.

And it's not just ACPI, what about "platform wdat_wdt: failed to claim
resource 5", "platform MSFT0101:00: failed to claim resource 1",
"lis3lv02d: unknown sensor type 0x0"?

I know you can probably identify the related components in a few
seconds, but I'm quite sure you won't be doing that for every bug report.

Should this be addressed?

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 01.10.22 12:47, Artem S. Tashkinov wrote:
> On 10/1/22 10:39, Greg KH wrote:
>> On Sat, Oct 01, 2022 at 10:30:22AM +0000, Artem S. Tashkinov wrote:

>>> I have a 20+ years experience in IT and some kernel issues are just
>>> baffling in terms of trying to understand what to do about them.
>>>
>>> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
>>>
>>> What should I do about that? Who's responsible for this? Who should I
>>> CC?
>>
>> Input subsystem.
> 
> It's great you've replied immediately, what about hundreds or even
> thousands of other bug reports where people have no clue who has to be
> CC'ed?

Quoting from https://docs.kernel.org/admin-guide/reporting-issues.html:

"[...] try your best guess which kernel part might be causing the issue.
Check the MAINTAINERS file [...] In case tricks like these don’t bring
you any further, try to search the internet on how to narrow down the
driver or subsystem in question. And if you are unsure which it is: just
try your best guess, somebody will help you if you guessed poorly. [...]"

HTH, Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/1/22 10:57, Thorsten Leemhuis wrote:
> On 01.10.22 12:47, Artem S. Tashkinov wrote:
>> On 10/1/22 10:39, Greg KH wrote:
>>> On Sat, Oct 01, 2022 at 10:30:22AM +0000, Artem S. Tashkinov wrote:
>
>>>> I have a 20+ years experience in IT and some kernel issues are just
>>>> baffling in terms of trying to understand what to do about them.
>>>>
>>>> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
>>>>
>>>> What should I do about that? Who's responsible for this? Who should I
>>>> CC?
>>>
>>> Input subsystem.
>>
>> It's great you've replied immediately, what about hundreds or even
>> thousands of other bug reports where people have no clue who has to be
>> CC'ed?
>
> Quoting from https://docs.kernel.org/admin-guide/reporting-issues.html:
>
> "[...] try your best guess which kernel part might be causing the issue.
> Check the MAINTAINERS file [...] In case tricks like these don’t bring
> you any further, try to search the internet on how to narrow down the
> driver or subsystem in question. And if you are unsure which it is: just
> try your best guess, somebody will help you if you guessed poorly. [...]"
>
> HTH, Ciao, Thorsten

Absolute most people:

* Will never read this document
* Will not be able to "search the internet on how to narrow down the
driver or subsystem in question"

Lastly "unsure which it is: just try your best guess, somebody will help
you if you guessed poorly", so send a message to LKML and hope someone
will reply? This generally does not work. LKML is such a high volume
mailing list, individual messages get lost right away.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 01.10.22 13:21, Artem S. Tashkinov wrote:
>
> Lastly "unsure which it is: just try your best guess, somebody will help
> you if you guessed poorly", so send a message to LKML 

Read the quotes in context please, they don't tell people to just send a
report to some mailing list, as they tell people to chose a subsystem
and mail its maintainers (with lists in CC).

Is that perfect and will in work in 100% of the cases? No, definitely
not. Would it be good to have a a kind of first level support group that
can help in this case? Sure. But we don't have one right now. I sooner
or later hope to work towards forming such a group, but there are other
things that are higher on my todo list for now.

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Theodore Ts'o]

On Sat, Oct 01, 2022 at 02:57:27PM +0700, Bagas Sanjaya wrote:
> > Funny thing.  I've largely given up on getting any kind of useful bug
> > report from Launchpad, so I've largely ignored it.  In contast, the
> > bug reports I get for e2fsprogs from Debian are generally far more
> > actionable, with bug reports that have all of the data so I can
> > actually root cause the problem, and help the user.
> 
> So no matter how the bug tracker interface is, the etiquette is:
> Whenever something buggy happens, try to gather all information related
> to that event (reproduction steps and reproducer, logs, crash dumps,
> etc), then file the polished report. From your experience, it seems like
> Debian people knows it.

Another critical part of the bug tracker etiquette is when in doubt,
always file a separate bug report.  More than once, both with
Launchpad or Kernel Bugzilla, users will do a web search for "my file
system lost data" or "EXT4-fs error" and assume it's the same problem
as what they are seeing.

In some cases, for a bug report that is years old and already closed.
That's actually less damaging, because it's obviously noise, and it
can be ignored.  The more annoying one is when the bug is actively
being worked, and people dog-pile onto that bug, and the bugs might be
caused by hardware issues (more often than not, a "bug report" is
really due to someone with a failing hard drive, or a USB stick which
is sticking out of the laptop, and gets jostled).  Even it's a real
software bug, if there are two bugs whose bug reports are getting
jumbled together into a single bug tracker web page, it can get
horribly confusing for the poor maintainer being asked to work the
issue, and the two users who start aguing amongst themselves about
their pet theory.

(Another bug reporter etiquette: clearly differenciate between *facts*
that you are reporting, and your pet theories about what might be
going wrong.  If you're so smart that you think you know the problem,
express your theory in the form of a patch.  Otherwise, putting
theories into a bug report which is not backed up by facts is worse
than useless.)

Of course, all of this can happy with bug reports filed by e-mail, or
via the Debian BTS.  However, it seems that people who are smart
enough to figure out how to send e-mail to a vger.kernel.org mailing
list, or how to use Debian's command-line interface "reportbug"
script, generally have enough experience that they can file a decent
bug report.  Whereas people who can only fill in a web page.... tend
not to have that (minimal) filter applied.

					- Ted

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Theodore Ts'o]

On Sat, Oct 01, 2022 at 01:34:26PM +0200, Thorsten Leemhuis wrote:
> 
> Is that perfect and will in work in 100% of the cases? No, definitely
> not. Would it be good to have a a kind of first level support group that
> can help in this case? Sure. But we don't have one right now. I sooner
> or later hope to work towards forming such a group, but there are other
> things that are higher on my todo list for now.

I think the other thing which we really need to say is that if you
really want better support, there are plenty of places who will
happily accept your money and provide you that support.  

Artem, it seems to me that you are hoping that volunteers will provide
a commercial level of support --- and that's just never going to
happen.

The users vastly outnumber us developers by orders of magnitude, and
if someone needs a huge amount of hand-holding, maybe they should be
paying for a support contract with Red Hat, or Suse or Canonical, or
CIQ.

Can we do better?  Sure!  But I think we need to clearly set
expectations for what upstream developers will and will not provide
support for.  (Example: bug reports for LTS kernels are not
interesting to me, unless you can also reproduce them in the latest
upstream kernel --- and if you can't build your own kernel from
scratch --- boo, hoo, maybe you need to pay someone to help you out.)

I also think that we need to clearly express that any kind of support
is best efforts only, and if someone has anything business-, mission-,
or life-critical, they should darned well pay $$$ for a proper support
contract.

						- Ted

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/1/22 13:07, Theodore Ts'o wrote:
> On Sat, Oct 01, 2022 at 01:34:26PM +0200, Thorsten Leemhuis wrote:
>>
>> Is that perfect and will in work in 100% of the cases? No, definitely
>> not. Would it be good to have a a kind of first level support group that
>> can help in this case? Sure. But we don't have one right now. I sooner
>> or later hope to work towards forming such a group, but there are other
>> things that are higher on my todo list for now.
>
> I think the other thing which we really need to say is that if you
> really want better support, there are plenty of places who will
> happily accept your money and provide you that support.
>
> Artem, it seems to me that you are hoping that volunteers will provide
> a commercial level of support --- and that's just never going to
> happen.
>
> The users vastly outnumber us developers by orders of magnitude, and
> if someone needs a huge amount of hand-holding, maybe they should be
> paying for a support contract with Red Hat, or Suse or Canonical, or
> CIQ.
>
> Can we do better?  Sure!  But I think we need to clearly set
> expectations for what upstream developers will and will not provide
> support for.  (Example: bug reports for LTS kernels are not
> interesting to me, unless you can also reproduce them in the latest
> upstream kernel --- and if you can't build your own kernel from
> scratch --- boo, hoo, maybe you need to pay someone to help you out.)
>
> I also think that we need to clearly express that any kind of support
> is best efforts only, and if someone has anything business-, mission-,
> or life-critical, they should darned well pay $$$ for a proper support
> contract.

My expectations are actually quite low:

* A central place to collect bugs (yeah, bugzilla)
* Proper up to date components (they don't change too often, so there's
not a lot of work to be done - you can refresh them probably every 12-24
months and it's gonna be totally OK)
* An ability to CC the relevant people/mailing lists (this is the only
serious missing feature)

That's it. It's a billion times better than random emails sent to random
mailing lists. Signing up once is easier that to keep track of whom and
where you've emailed or not. And of course it's a ton lot easier to find
the existing bug reports.

Bugzilla as it is works nearly perfectly. We have a number of developers
who don't want to touch it or get emails from it - it's their right.
However it would be madness to take it from users. That will make filing
and following up on bug reports an absolutely poor experience for
absolute most users.

Here's a recent fresh example:

https://bugzilla.kernel.org/show_bug.cgi?id=204807

255 comments along with patches, ideas, contributions, etc. etc. over
the span of _two years_. Email will not work for such collabs, period.
No email client can even show you more than a dozen emails in such a way
you can easily follow the topic. Email is good maybe for slow-paced
interchanges of a small circle of people working on a particular
well-known issue. Even the issue we've been discussing here has become
nearly impossible to reach consensus on or remember who said who.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Takashi Iwai]

On Sat, 01 Oct 2022 12:30:22 +0200,
Artem S. Tashkinov wrote:
> 
> Here are two other issues which absolutely suck in terms of dealing with
> the kernel.
> 
> - 1 -
> 
> I have a 20+ years experience in IT and some kernel issues are just
> baffling in terms of trying to understand what to do about them.
> 
> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
> 
> What should I do about that? Who's responsible for this? Who should I CC?
> 
> And this is an issue which is easy to describe and identify.

IMO, this indicates one of the big problems of bugzilla -- or a bug
tracker in general -- with the complete lack of screening.

An initial bug report is sent only to the bug assignees of the given
component, and those are mostly destined to persons (usually
maintainers), not to a public ML or group.  That doesn't work nor
scale for lots of bug reports.  We need screening at the first place,
before maintainers try to take a deeper look.

One may change the default target of the bugzilla assignee to a ML,
too.  However, this leads to sending lots of noises from unqualified
bug reports straightly to ML, which shall upset developers, so it's no
better choice.

And, screening is a tiresome task; you'd have to deal sometimes with
people have no clue and no etiquette.  I understand many companies
trying to deploy AI for that place...


> - 2 -
> 
> Here's another one which is outright puzzling:
> 
> You run: dmesg -t --level=emerg,crit,err
> 
> And you see some non-descript errors of some kernel subsystems seemingly
> failing or being unhappy about your hardware. Errors are as cryptic as
> humanly possible, you don't even know what part of kernel has produced them.
> 
> OK, as a "power" user I download the kernel source, run `grep -R message
> /tmp/linux-5.19` and there are _multiple_ different modules and places
> which contain this message.
> 
> I'm lost. Send this to LKML? Did that in the long past, no one cared, I
> stopped.
> 
> Here's what I'm getting with Linux 5.19.12:
> 
> platform wdat_wdt: failed to claim resource 5: [mem
> 0x00000000-0xffffffff7fffffff]
> ACPI: watchdog: Device creation failed: -16
> ACPI BIOS Error (bug): Could not resolve symbol
> [\_SB.PCI0.XHC.RHUB.TPLD], AE_NOT_FOUND (20220331/psargs-330)
> ACPI Error: Aborting method \_SB.UBTC.CR01._PLD due to previous error
> (AE_NOT_FOUND) (20220331/psparse-529)
> platform MSFT0101:00: failed to claim resource 1: [mem
> 0xfed40000-0xfed40fff]
> acpi MSFT0101:00: platform device creation failed: -16
> lis3lv02d: unknown sensor type 0x0
> 
> Are they serious? Should they be reported or not? Is my laptop properly
> working? I have no clue at all.

That's a dilemma.  The kernel can't know whether it's "properly"
working, either -- that is, whether the lack of some functions matters
for you or not.  In your case above, it's about a watchdog, something
related with USB, TPM, and acceleration sensor, all of which likely
come from a buggy BIOS.  Would you mind if those features are missing?
Or even whether your device has a correct hardware implementation?
Kernel doesn't know, hence it complains as an error.

In many drivers, there are mechanisms to shut off superfluous error
messages for known devices.  So it's case-by-case solutions.

Or you can completely hide those errors at boot by a boot option
(e.g. loglevel=2).


Takashi

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 07:37, Takashi Iwai wrote:
> On Sat, 01 Oct 2022 12:30:22 +0200,
> Artem S. Tashkinov wrote:
>> - 2 -
>>
>> Here's another one which is outright puzzling:
>>
>> You run: dmesg -t --level=emerg,crit,err
>>
>> And you see some non-descript errors of some kernel subsystems seemingly
>> failing or being unhappy about your hardware. Errors are as cryptic as
>> humanly possible, you don't even know what part of kernel has produced them.
>>
>> OK, as a "power" user I download the kernel source, run `grep -R message
>> /tmp/linux-5.19` and there are _multiple_ different modules and places
>> which contain this message.
>>
>> I'm lost. Send this to LKML? Did that in the long past, no one cared, I
>> stopped.
>>
>> Here's what I'm getting with Linux 5.19.12:
>>
>> platform wdat_wdt: failed to claim resource 5: [mem
>> 0x00000000-0xffffffff7fffffff]
>> ACPI: watchdog: Device creation failed: -16
>> ACPI BIOS Error (bug): Could not resolve symbol
>> [\_SB.PCI0.XHC.RHUB.TPLD], AE_NOT_FOUND (20220331/psargs-330)
>> ACPI Error: Aborting method \_SB.UBTC.CR01._PLD due to previous error
>> (AE_NOT_FOUND) (20220331/psparse-529)
>> platform MSFT0101:00: failed to claim resource 1: [mem
>> 0xfed40000-0xfed40fff]
>> acpi MSFT0101:00: platform device creation failed: -16
>> lis3lv02d: unknown sensor type 0x0
>>
>> Are they serious? Should they be reported or not? Is my laptop properly
>> working? I have no clue at all.
>
> That's a dilemma.  The kernel can't know whether it's "properly"
> working, either -- that is, whether the lack of some functions matters
> for you or not.  In your case above, it's about a watchdog, something
> related with USB, TPM, and acceleration sensor, all of which likely
> come from a buggy BIOS.  Would you mind if those features are missing?
> Or even whether your device has a correct hardware implementation?
> Kernel doesn't know, hence it complains as an error.
>
> In many drivers, there are mechanisms to shut off superfluous error
> messages for known devices.  So it's case-by-case solutions.
>
> Or you can completely hide those errors at boot by a boot option
> (e.g. loglevel=2).

The problem is some of such messages are indeed indicative of certain
real issues which result in HW not working properly, including:

1) missing/incorrect firmware
2) most importantly: not enabled power saving modes
3) not enabled high performance modes
4) not enabled devices
5) not enabled devices' functions
6) drivers conflicts (i.e. the wrong module gets loaded for the device)
7) physically failing hardware

I'm quite sure you don't really know what half of those messages
actually mean.

Speaking of 7. Various kernel subsystems/drivers deal with e.g. mass
storage which is known to fail quite often. There's not a single driver
in the kernel which is actually brave enough to spew something like this:

"/dev/xxxx might be failing, please RMA or seek help online"

instead you get a dmesg choke full of "unable to read sector XXX" or
something like that.

To return to the previous errors: it's impossible for the user to assess
their severity and that sucks. What is "platform device creation
failed"? What is "unknown sensor type"? What am I missing? Who's
responsible? The kernel? My HW vendor? Are those errors actionable? In
my understanding a properly working computer must not produce
"emerg,crit,err" errors. I'm not even talking about "warn,info" and such.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Sun, Oct 2, 2022 at 10:23 AM Artem S. Tashkinov <aros@gmx.com> wrote:
> On 10/2/22 07:37, Takashi Iwai wrote:
> > On Sat, 01 Oct 2022 12:30:22 +0200,
> > Artem S. Tashkinov wrote:
> >> Here's another one which is outright puzzling:
> >>
> >> You run: dmesg -t --level=emerg,crit,err
> >>
> >> And you see some non-descript errors of some kernel subsystems seemingly
> >> failing or being unhappy about your hardware. Errors are as cryptic as
> >> humanly possible, you don't even know what part of kernel has produced them.
> >>
> >> OK, as a "power" user I download the kernel source, run `grep -R message
> >> /tmp/linux-5.19` and there are _multiple_ different modules and places
> >> which contain this message.
> >>
> >> I'm lost. Send this to LKML? Did that in the long past, no one cared, I
> >> stopped.
> >>
> >> Here's what I'm getting with Linux 5.19.12:
> >>
> >> platform wdat_wdt: failed to claim resource 5: [mem
> >> 0x00000000-0xffffffff7fffffff]
> >> ACPI: watchdog: Device creation failed: -16
> >> ACPI BIOS Error (bug): Could not resolve symbol
> >> [\_SB.PCI0.XHC.RHUB.TPLD], AE_NOT_FOUND (20220331/psargs-330)
> >> ACPI Error: Aborting method \_SB.UBTC.CR01._PLD due to previous error
> >> (AE_NOT_FOUND) (20220331/psparse-529)
> >> platform MSFT0101:00: failed to claim resource 1: [mem
> >> 0xfed40000-0xfed40fff]
> >> acpi MSFT0101:00: platform device creation failed: -16
> >> lis3lv02d: unknown sensor type 0x0
> >>
> >> Are they serious? Should they be reported or not? Is my laptop properly
> >> working? I have no clue at all.
> >
> > That's a dilemma.  The kernel can't know whether it's "properly"
> > working, either -- that is, whether the lack of some functions matters
> > for you or not.  In your case above, it's about a watchdog, something
> > related with USB, TPM, and acceleration sensor, all of which likely
> > come from a buggy BIOS.  Would you mind if those features are missing?
> > Or even whether your device has a correct hardware implementation?
> > Kernel doesn't know, hence it complains as an error.
> >
> > In many drivers, there are mechanisms to shut off superfluous error
> > messages for known devices.  So it's case-by-case solutions.
> >
> > Or you can completely hide those errors at boot by a boot option
> > (e.g. loglevel=2).
>
> The problem is some of such messages are indeed indicative of certain
> real issues which result in HW not working properly, including:
>
> 1) missing/incorrect firmware
> 2) most importantly: not enabled power saving modes
> 3) not enabled high performance modes
> 4) not enabled devices
> 5) not enabled devices' functions
> 6) drivers conflicts (i.e. the wrong module gets loaded for the device)
> 7) physically failing hardware
>
> I'm quite sure you don't really know what half of those messages
> actually mean.
>
> Speaking of 7. Various kernel subsystems/drivers deal with e.g. mass
> storage which is known to fail quite often. There's not a single driver
> in the kernel which is actually brave enough to spew something like this:
>
> "/dev/xxxx might be failing, please RMA or seek help online"
>
> instead you get a dmesg choke full of "unable to read sector XXX" or
> something like that.
>
> To return to the previous errors: it's impossible for the user to assess
> their severity and that sucks. What is "platform device creation
> failed"? What is "unknown sensor type"? What am I missing? Who's
> responsible? The kernel? My HW vendor? Are those errors actionable? In
> my understanding a properly working computer must not produce
> "emerg,crit,err" errors. I'm not even talking about "warn,info" and such.

I am afraid that for most of the above, the kernel cannot know the
answer.  Hence more investigation/debugging is needed.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Sat, Oct 1, 2022 at 1:21 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> On 10/1/22 10:57, Thorsten Leemhuis wrote:
> > On 01.10.22 12:47, Artem S. Tashkinov wrote:
> >> On 10/1/22 10:39, Greg KH wrote:
> >>> On Sat, Oct 01, 2022 at 10:30:22AM +0000, Artem S. Tashkinov wrote:
> >
> >>>> I have a 20+ years experience in IT and some kernel issues are just
> >>>> baffling in terms of trying to understand what to do about them.
> >>>>
> >>>> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
> >>>>
> >>>> What should I do about that? Who's responsible for this? Who should I
> >>>> CC?
> >>>
> >>> Input subsystem.
> >>
> >> It's great you've replied immediately, what about hundreds or even
> >> thousands of other bug reports where people have no clue who has to be
> >> CC'ed?
> >
> > Quoting from https://docs.kernel.org/admin-guide/reporting-issues.html:
> >
> > "[...] try your best guess which kernel part might be causing the issue.
> > Check the MAINTAINERS file [...] In case tricks like these don’t bring
> > you any further, try to search the internet on how to narrow down the
> > driver or subsystem in question. And if you are unsure which it is: just
> > try your best guess, somebody will help you if you guessed poorly. [...]"
> >
> > HTH, Ciao, Thorsten
>
> Absolute most people:
>
> * Will never read this document
> * Will not be able to "search the internet on how to narrow down the
> driver or subsystem in question"

So how did these people arrive at "bugzilla" in the first place? ;-)

Or is this a case of "if all you have is a hammer...", so you
actively start looking for a bugzilla?
I.e. people who are used to bugzilla/discourse/slack/irc/trac/... will
look for how to use bugzilla/discourse/slack/irc/trac/... to interact
with the developer and/or maintainer.

The definitive guide is the MAINTAINERS file.  If there is a (rare)
corresponding "B" entry, you can use that.  Else fall back to the
"M" and "L" entries.  "C" might be good for an initial query, but not
for the actual reporting, as there's even less traceability than with
mailing lists (the latter are archived by lore).

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 09:03, Geert Uytterhoeven wrote:
> Hi Artem,
>
> On Sat, Oct 1, 2022 at 1:21 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>> On 10/1/22 10:57, Thorsten Leemhuis wrote:
>>> On 01.10.22 12:47, Artem S. Tashkinov wrote:
>>>> On 10/1/22 10:39, Greg KH wrote:
>>>>> On Sat, Oct 01, 2022 at 10:30:22AM +0000, Artem S. Tashkinov wrote:
>>>
>>>>>> I have a 20+ years experience in IT and some kernel issues are just
>>>>>> baffling in terms of trying to understand what to do about them.
>>>>>>
>>>>>> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
>>>>>>
>>>>>> What should I do about that? Who's responsible for this? Who should I
>>>>>> CC?
>>>>>
>>>>> Input subsystem.
>>>>
>>>> It's great you've replied immediately, what about hundreds or even
>>>> thousands of other bug reports where people have no clue who has to be
>>>> CC'ed?
>>>
>>> Quoting from https://docs.kernel.org/admin-guide/reporting-issues.html:
>>>
>>> "[...] try your best guess which kernel part might be causing the issue.
>>> Check the MAINTAINERS file [...] In case tricks like these don’t bring
>>> you any further, try to search the internet on how to narrow down the
>>> driver or subsystem in question. And if you are unsure which it is: just
>>> try your best guess, somebody will help you if you guessed poorly. [...]"
>>>
>>> HTH, Ciao, Thorsten
>>
>> Absolute most people:
>>
>> * Will never read this document
>> * Will not be able to "search the internet on how to narrow down the
>> driver or subsystem in question"
>
> So how did these people arrive at "bugzilla" in the first place? ;-)

Google kernel bug -> bugzilla.kernel.org

>
> Or is this a case of "if all you have is a hammer...", so you
> actively start looking for a bugzilla?
> I.e. people who are used to bugzilla/discourse/slack/irc/trac/... will
> look for how to use bugzilla/discourse/slack/irc/trac/... to interact
> with the developer and/or maintainer.
>
> The definitive guide is the MAINTAINERS file.  If there is a (rare)
> corresponding "B" entry, you can use that.  Else fall back to the
> "M" and "L" entries.  "C" might be good for an initial query, but not
> for the actual reporting, as there's even less traceability than with
> mailing lists (the latter are archived by lore).

Just like I said before email sucks terribly for bug reporting except
for rare cases when the developer notices your email right away and
promptly submits a patch. This happens once in a blue moon unfortunately.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Takashi Iwai]

On Sun, 02 Oct 2022 10:23:07 +0200,
Artem S. Tashkinov wrote:
> 
> 
> 
> On 10/2/22 07:37, Takashi Iwai wrote:
> > On Sat, 01 Oct 2022 12:30:22 +0200,
> > Artem S. Tashkinov wrote:
> >> - 2 -
> >> 
> >> Here's another one which is outright puzzling:
> >> 
> >> You run: dmesg -t --level=emerg,crit,err
> >> 
> >> And you see some non-descript errors of some kernel subsystems seemingly
> >> failing or being unhappy about your hardware. Errors are as cryptic as
> >> humanly possible, you don't even know what part of kernel has produced them.
> >> 
> >> OK, as a "power" user I download the kernel source, run `grep -R message
> >> /tmp/linux-5.19` and there are _multiple_ different modules and places
> >> which contain this message.
> >> 
> >> I'm lost. Send this to LKML? Did that in the long past, no one cared, I
> >> stopped.
> >> 
> >> Here's what I'm getting with Linux 5.19.12:
> >> 
> >> platform wdat_wdt: failed to claim resource 5: [mem
> >> 0x00000000-0xffffffff7fffffff]
> >> ACPI: watchdog: Device creation failed: -16
> >> ACPI BIOS Error (bug): Could not resolve symbol
> >> [\_SB.PCI0.XHC.RHUB.TPLD], AE_NOT_FOUND (20220331/psargs-330)
> >> ACPI Error: Aborting method \_SB.UBTC.CR01._PLD due to previous error
> >> (AE_NOT_FOUND) (20220331/psparse-529)
> >> platform MSFT0101:00: failed to claim resource 1: [mem
> >> 0xfed40000-0xfed40fff]
> >> acpi MSFT0101:00: platform device creation failed: -16
> >> lis3lv02d: unknown sensor type 0x0
> >> 
> >> Are they serious? Should they be reported or not? Is my laptop properly
> >> working? I have no clue at all.
> > 
> > That's a dilemma.  The kernel can't know whether it's "properly"
> > working, either -- that is, whether the lack of some functions matters
> > for you or not.  In your case above, it's about a watchdog, something
> > related with USB, TPM, and acceleration sensor, all of which likely
> > come from a buggy BIOS.  Would you mind if those features are missing?
> > Or even whether your device has a correct hardware implementation?
> > Kernel doesn't know, hence it complains as an error.
> > 
> > In many drivers, there are mechanisms to shut off superfluous error
> > messages for known devices.  So it's case-by-case solutions.
> > 
> > Or you can completely hide those errors at boot by a boot option
> > (e.g. loglevel=2).
> 
> The problem is some of such messages are indeed indicative of certain
> real issues which result in HW not working properly, including:
> 
> 1) missing/incorrect firmware
> 2) most importantly: not enabled power saving modes
> 3) not enabled high performance modes
> 4) not enabled devices
> 5) not enabled devices' functions
> 6) drivers conflicts (i.e. the wrong module gets loaded for the device)
> 7) physically failing hardware
> 
> I'm quite sure you don't really know what half of those messages
> actually mean.

Of course: not because those messages are hardly understandable but
because those messages indicate only the cause, and the exact end
result can't be always known from the kernel at that point.  A lack of
physical failing hardware?  Not enabled devices?  Who knows.  There
might be some alternative, even a user-space driver.

> Speaking of 7. Various kernel subsystems/drivers deal with e.g. mass
> storage which is known to fail quite often. There's not a single driver
> in the kernel which is actually brave enough to spew something like this:
> 
> "/dev/xxxx might be failing, please RMA or seek help online"
> 
> instead you get a dmesg choke full of "unable to read sector XXX" or
> something like that.

Oh you suggest that we should put "please RMA or seek help online" to
each printk of KERN_ERR level, if it saves the world? ;)

IMO, what matters for users is whether the system works or not.  It's
not how the kernel message appears.  A kernel message may help for
diagnose, but the message itself is no solution; that is, the most
importance of a kernel message is that it indicates a real error that
can be diagnosed by developers.

If the end effect is pretty sure, a message may be more chatty.  OTOH,
people are annoyed by such too verbosity, too.  So it needs a sensible
choice.

> To return to the previous errors: it's impossible for the user to assess
> their severity and that sucks.

Right, that's why I wrote it's a dilemma.

> What is "platform device creation
> failed"? What is "unknown sensor type"? What am I missing? Who's
> responsible? The kernel? My HW vendor? Are those errors actionable?

All those depend on the driver implementation and the hardware
implementation.  There is no general answer at all, unfortunately.

> In
> my understanding a properly working computer must not produce
> "emerg,crit,err" errors. I'm not even talking about "warn,info" and such.

Yes, some errors can be downgraded to warn or even to info.
I myself find ACPI is way too chatty, too.

So I believe something we can improve is to define some more clear
guideline for KERN_ERR level errors.


Takashi

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Sun, Oct 2, 2022 at 11:07 AM Artem S. Tashkinov <aros@gmx.com> wrote:
> On 10/2/22 09:03, Geert Uytterhoeven wrote:
> > On Sat, Oct 1, 2022 at 1:21 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> >> On 10/1/22 10:57, Thorsten Leemhuis wrote:
> >>> On 01.10.22 12:47, Artem S. Tashkinov wrote:
> >>>> On 10/1/22 10:39, Greg KH wrote:
> >>>>> On Sat, Oct 01, 2022 at 10:30:22AM +0000, Artem S. Tashkinov wrote:
> >>>
> >>>>>> I have a 20+ years experience in IT and some kernel issues are just
> >>>>>> baffling in terms of trying to understand what to do about them.
> >>>>>>
> >>>>>> Here's an example: https://bugzilla.kernel.org/show_bug.cgi?id=216274
> >>>>>>
> >>>>>> What should I do about that? Who's responsible for this? Who should I
> >>>>>> CC?
> >>>>>
> >>>>> Input subsystem.
> >>>>
> >>>> It's great you've replied immediately, what about hundreds or even
> >>>> thousands of other bug reports where people have no clue who has to be
> >>>> CC'ed?
> >>>
> >>> Quoting from https://docs.kernel.org/admin-guide/reporting-issues.html:
> >>>
> >>> "[...] try your best guess which kernel part might be causing the issue.
> >>> Check the MAINTAINERS file [...] In case tricks like these don’t bring
> >>> you any further, try to search the internet on how to narrow down the
> >>> driver or subsystem in question. And if you are unsure which it is: just
> >>> try your best guess, somebody will help you if you guessed poorly. [...]"
> >>>
> >>> HTH, Ciao, Thorsten
> >>
> >> Absolute most people:
> >>
> >> * Will never read this document
> >> * Will not be able to "search the internet on how to narrow down the
> >> driver or subsystem in question"
> >
> > So how did these people arrive at "bugzilla" in the first place? ;-)
>
> Google kernel bug -> bugzilla.kernel.org

First three hits:

  1. Google just tripled its bounty for Linux kernel bugs. Here's
whyhttps://www.zdnet.com › Innovation › Security
     02 Nov 2021 — Google has kicked off a special three-month bug
bounty targeting flaws in the Linux kernel with triple the rewards for
security researchers.

  2. Google's New Bug Bounties Include Their Custom Linux
...https://linux.slashdot.org › story › googles-new-bug-bo...
     13 Aug 2022 — All of GKE and its dependencies are in scope, but
every flag caught so far has been a container breakout through a Linux
kernel vulnerability.

  3. How to report Linux kernel bugs - google/syzkaller -
GitHubhttps://github.com › syzkaller › blob › master › docs
     Reporting Linux kernel bugs. Before reporting a bug make sure
nobody else already reported it. The easiest way to do this is to
search through the syzkaller ...

Bingo, that page tells you to use the MAINTAINERS file.
Nothing about bugzilla on the first page (Google is adapting quickly ;-)

> > Or is this a case of "if all you have is a hammer...", so you
> > actively start looking for a bugzilla?
> > I.e. people who are used to bugzilla/discourse/slack/irc/trac/... will
> > look for how to use bugzilla/discourse/slack/irc/trac/... to interact
> > with the developer and/or maintainer.
> >
> > The definitive guide is the MAINTAINERS file.  If there is a (rare)
> > corresponding "B" entry, you can use that.  Else fall back to the
> > "M" and "L" entries.  "C" might be good for an initial query, but not
> > for the actual reporting, as there's even less traceability than with
> > mailing lists (the latter are archived by lore).
>
> Just like I said before email sucks terribly for bug reporting except
> for rare cases when the developer notices your email right away and
> promptly submits a patch. This happens once in a blue moon unfortunately.

During the last decade, I received three emails from kernel bugzilla
(four including the "please change your password" email).

The first one was incorrectly assigned to me, but closest as fixed
later.
The second was fixed by the reporter after feedback from someone else.
The third was a complex configuration issue (think randconfig) that
was not trivial to fix, but doesn't matter in the real world.

Now, for bug reports by email that I did fix, I cannot give you the
details easily, as the list would be a few orders of magnitude larger...

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Theodore Ts'o]

On Sat, Oct 01, 2022 at 02:58:04PM +0000, Artem S. Tashkinov wrote:
> 
> My expectations are actually quite low:
> 
> * A central place to collect bugs (yeah, bugzilla)
> * Proper up to date components (they don't change too often, so there's
> not a lot of work to be done - you can refresh them probably every 12-24
> months and it's gonna be totally OK)
> * An ability to CC the relevant people/mailing lists (this is the only
> serious missing feature)
> 
> That's it. It's a billion times better than random emails sent to random
> mailing lists. Signing up once is easier that to keep track of whom and
> where you've emailed or not. And of course it's a ton lot easier to find
> the existing bug reports.

First of all, some of the components do CC the relevant mailing lists
automatically.  And this is the part of Bugzilla which is hand-hacked
and has no, zero, nada support upstream.  I'll defer to Konstantin
about how easy it is to keep that working.

Secondly, not everyone is happy with getting an e-mail message sent to
a mailing list that has a lot of bugzilla metadata associated with it,
and depending on how they respond, the response might not make it back
to bugzilla.

Finally, in the absense of someone to actually close bugzilla entries
and do other necessary grooming, the bugzilla database will rapidly
become useless --- in which case, you might as well have a web form
that just helps the user send e-mail to the mailing list, and hope it
doesn't become a SPAM magnet.

> Bugzilla as it is works nearly perfectly. We have a number of developers
> who don't want to touch it or get emails from it - it's their right.
> However it would be madness to take it from users. That will make filing
> and following up on bug reports an absolutely poor experience for
> absolute most users.

At the moment, developers aren't following up on the bug reports.
There is some debate as to why.  Is it because users who can't figure
out how to send e-mail, and who send web-form based e-mails send low
quality bug reports that can't be easily responded to unless someone
is paid $$$ and/or has the patience of a saint?  Is it because
components aren't being gatewayed to mailing lists?

And if we force developers to get Bugzilla spam whether they want it
not, and they said, "absolutely not", is it there right to have the
mailing list gateway disabled --- and if so, what does that do to the
user experience?  Thats basically the situation we have right now.

     		  		      - Ted

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 12:18, Theodore Ts'o wrote:
> On Sat, Oct 01, 2022 at 02:58:04PM +0000, Artem S. Tashkinov wrote:
>>
>> My expectations are actually quite low:
>>
>> * A central place to collect bugs (yeah, bugzilla)
>> * Proper up to date components (they don't change too often, so there's
>> not a lot of work to be done - you can refresh them probably every 12-24
>> months and it's gonna be totally OK)
>> * An ability to CC the relevant people/mailing lists (this is the only
>> serious missing feature)
>>
>> That's it. It's a billion times better than random emails sent to random
>> mailing lists. Signing up once is easier that to keep track of whom and
>> where you've emailed or not. And of course it's a ton lot easier to find
>> the existing bug reports.
>
> First of all, some of the components do CC the relevant mailing lists
> automatically.  And this is the part of Bugzilla which is hand-hacked
> and has no, zero, nada support upstream.  I'll defer to Konstantin
> about how easy it is to keep that working.
>
> Secondly, not everyone is happy with getting an e-mail message sent to
> a mailing list that has a lot of bugzilla metadata associated with it,
> and depending on how they respond, the response might not make it back
> to bugzilla.

I've mentioned it a dozen times already - you're unhappy with emails
from bugzilla? Go there and unsubscribe. It takes a minute and we're
talking as if it's the actual issue we are dealing with. It's not.
Bugzilla maintenance and its up-to-date status are the issues.

>
> Finally, in the absense of someone to actually close bugzilla entries
> and do other necessary grooming, the bugzilla database will rapidly
> become useless --- in which case, you might as well have a web form
> that just helps the user send e-mail to the mailing list, and hope it
> doesn't become a SPAM magnet.

The current ill-maintained semi-functional bugzilla has proven to be a
ton more useful than random mailing lists no sane person can keep track
of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
is the worst of them probably.

>
>> Bugzilla as it is works nearly perfectly. We have a number of developers
>> who don't want to touch it or get emails from it - it's their right.
>> However it would be madness to take it from users. That will make filing
>> and following up on bug reports an absolutely poor experience for
>> absolute most users.
>
> At the moment, developers aren't following up on the bug reports.
> There is some debate as to why.  Is it because users who can't figure
> out how to send e-mail, and who send web-form based e-mails send low
> quality bug reports that can't be easily responded to unless someone
> is paid $$$ and/or has the patience of a saint?  Is it because
> components aren't being gatewayed to mailing lists?

This is not always true, some of them do, some of them actually check
new bug reports and do a tremendous job at helping people, e.g. Mario
Limonciello who helps resolve bugs which are not even his direct
responsibility. BTW, I'll now CC him since he's so active over there.
Would be great if he chimed in.

>
> And if we force developers to get Bugzilla spam whether they want it
> not, and they said, "absolutely not", is it there right to have the
> mailing list gateway disabled --- and if so, what does that do to the
> user experience?  Thats basically the situation we have right now.

As I've said many times already: bugzilla must be an opt-out, not opt-in
experience/option.

Let's subscribe the past six months of developers using git commits and
if someone doesn't like getting emails they go to the website and
unsubscribe _once_ which takes a minute. This is a non-issue I've no
clue why we're dwelling on it.

Let's operate with some examples:

Bugzilla gets around two dozen bug reports weekly which encompass at
most thirty emails, which equals to four emails daily on average.

LKML alone sees up to a hundred emails _daily_.

Getting worked up about it? I'm dumbfounded to be honest.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hello,

> On Oct 2, 2022, at 8:18 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> 
> On Sat, Oct 01, 2022 at 02:58:04PM +0000, Artem S. Tashkinov wrote:
>> 
>> My expectations are actually quite low:
>> 
>> * A central place to collect bugs (yeah, bugzilla)
>> * Proper up to date components (they don't change too often, so there's
>> not a lot of work to be done - you can refresh them probably every 12-24
>> months and it's gonna be totally OK)
>> * An ability to CC the relevant people/mailing lists (this is the only
>> serious missing feature)
>> 
>> That's it. It's a billion times better than random emails sent to random
>> mailing lists. Signing up once is easier that to keep track of whom and
>> where you've emailed or not. And of course it's a ton lot easier to find
>> the existing bug reports.
> 
> First of all, some of the components do CC the relevant mailing lists
> automatically.  And this is the part of Bugzilla which is hand-hacked
> and has no, zero, nada support upstream.  I'll defer to Konstantin
> about how easy it is to keep that working.
> 
> Secondly, not everyone is happy with getting an e-mail message sent to
> a mailing list that has a lot of bugzilla metadata associated with it,
> and depending on how they respond, the response might not make it back
> to bugzilla.

+1. 

Personally, I prefer Bugzilla _over_ getting e-mail. But that’s just my opinion.

>> Bugzilla as it is works nearly perfectly. We have a number of developers
>> who don't want to touch it or get emails from it - it's their right.
>> However it would be madness to take it from users. That will make filing
>> and following up on bug reports an absolutely poor experience for
>> absolute most users.
> 
> At the moment, developers aren't following up on the bug reports.
> There is some debate as to why.  Is it because users who can't figure
> out how to send e-mail, and who send web-form based e-mails send low
> quality bug reports that can't be easily responded to unless someone
> is paid $$$ and/or has the patience of a saint?  Is it because
> components aren't being gatewayed to mailing lists?

My hope is that we find a solution that *encourages* developers to follow-up on bug reports. So far, we’ve just gone back and forth on this and gotten nowhere.

> 
> And if we force developers to get Bugzilla spam whether they want it
> not, and they said, "absolutely not", is it there right to have the
> mailing list gateway disabled --- and if so, what does that do to the
> user experience?  Thats basically the situation we have right now.

Yep, agreed.

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Greg KH]

On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
> > And if we force developers to get Bugzilla spam whether they want it
> > not, and they said, "absolutely not", is it there right to have the
> > mailing list gateway disabled --- and if so, what does that do to the
> > user experience?  Thats basically the situation we have right now.
> 
> As I've said many times already: bugzilla must be an opt-out, not opt-in
> experience/option.
> 
> Let's subscribe the past six months of developers using git commits and
> if someone doesn't like getting emails they go to the website and
> unsubscribe _once_ which takes a minute. This is a non-issue I've no
> clue why we're dwelling on it.

auto-subscribing people to anything is a sure way to get lots of people
instantly mad at you and have them add the address to their filters.

That's just not how to do things well, sorry.

If you wish to be the one triaging all bugzilla bugs, wonderful, please
start doing so.  But to volunteer others and insist that they do it is a
non-starter for obvious reasons.

greg k-h

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hi,

> On Oct 2, 2022, at 8:49 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> As I've said many times already: bugzilla must be an opt-out, not opt-in
> experience/option.
> 
> Let's subscribe the past six months of developers using git commits and
> if someone doesn't like getting emails they go to the website and
> unsubscribe _once_ which takes a minute. This is a non-issue I've no
> clue why we're dwelling on it.

I disagree with this in its *entirety* and I really don’t think it has any chance of moving forward. 

If this were to happen (and it won’t!) then developers will just send the emails to spam or some other filter because they didn’t _consent_ to being subscribed to it. And in my opinion, they’d be justified in doing that.

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
> I've mentioned it a dozen times already - you're unhappy with emails
> from bugzilla? Go there and unsubscribe. It takes a minute and we're
> talking as if it's the actual issue we are dealing with. It's not.
> Bugzilla maintenance and its up-to-date status are the issues.

I think you're working from the wrong conception that maintainers want to
receive (or even be aware) of bug reports. Maintainers want *PATCHES*, not bug
reports. You're asking senior engineers to do first-line QA.

This is why your suggestion to auto-subscribe maintainers to bug reports is
the absolutely wrong way to go about it. The maintainers will complain that
they're being inundated with spam and junk reports, and bug reporters will
complain that they are being treated rudely (because this is how a senior
engineer trying to get succinct information comes across). You know that meme
from Fallout with the words "[Everyone hated that]" on it?

> > And if we force developers to get Bugzilla spam whether they want it
> > not, and they said, "absolutely not", is it there right to have the
> > mailing list gateway disabled --- and if so, what does that do to the
> > user experience?  Thats basically the situation we have right now.
> 
> As I've said many times already: bugzilla must be an opt-out, not opt-in
> experience/option.

Please listen to the actual maintainers telling you that this won't work and
will just lead to bugzilla being added to everyone's block list (that's even
faster than logging in to bugzilla, finding where to change the default
assignee, etc).

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Al Viro]

On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:

> > Secondly, not everyone is happy with getting an e-mail message sent to
                                                                   ^^^^^^^
> > a mailing list that has a lot of bugzilla metadata associated with it,
    ^^^^^^^^^^^^^^
> > and depending on how they respond, the response might not make it back
> > to bugzilla.
> 
> I've mentioned it a dozen times already - you're unhappy with emails
> from bugzilla? Go there and unsubscribe. It takes a minute and we're
                 ^^^^^^^^^^^^^^^^^^^^^^^^
> talking as if it's the actual issue we are dealing with. It's not.
> Bugzilla maintenance and its up-to-date status are the issues.

OK, then - please tell me how to prevent e.g. linux-fsdevel@vger.kernel.org
getting spammed by that thing.  Where should I go and how do I unsubscribe
it?

The answer along the lines of "unsubscribe yourself from the maillists"
is a non-starter, obviously.

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Sun, Oct 2, 2022 at 2:49 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> The current ill-maintained semi-functional bugzilla has proven to be a
> ton more useful than random mailing lists no sane person can keep track
> of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
> is the worst of them probably.

Such a statement really needs to be backed by numbers...

> Let's operate with some examples:
>
> Bugzilla gets around two dozen bug reports weekly which encompass at
> most thirty emails, which equals to four emails daily on average.

This immediately debunks your statement above.

$ git log v5.19..linus/master | grep Fixes: | wc -l
2928

So that's 46 bugs fixed per _day_.  Most of them not reported
through bugzilla...

> LKML alone sees up to a hundred emails _daily_.
>
> Getting worked up about it? I'm dumbfounded to be honest.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

On Sun, Oct 2, 2022 at 6:08 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Sun, Oct 2, 2022 at 2:49 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> > The current ill-maintained semi-functional bugzilla has proven to be a
> > ton more useful than random mailing lists no sane person can keep track
> > of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
> > is the worst of them probably.
>
> Such a statement really needs to be backed by numbers...
>
> > Let's operate with some examples:
> >
> > Bugzilla gets around two dozen bug reports weekly which encompass at
> > most thirty emails, which equals to four emails daily on average.
>
> This immediately debunks your statement above.
>
> $ git log v5.19..linus/master | grep Fixes: | wc -l
> 2928

Sorry, this was using my grep = `grep --color=tty -i' alias.
But that caused less than 100 false-positives, thus has no impact
on the point I'm trying to make.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Joe Perches]

On Sun, 2022-10-02 at 18:08 +0200, Geert Uytterhoeven wrote:
> On Sun, Oct 2, 2022 at 2:49 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> > The current ill-maintained semi-functional bugzilla has proven to be a
> > ton more useful than random mailing lists no sane person can keep track
> > of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
> > is the worst of them probably.
> 
> Such a statement really needs to be backed by numbers...
> 
> > Let's operate with some examples:
> > 
> > Bugzilla gets around two dozen bug reports weekly which encompass at
> > most thirty emails, which equals to four emails daily on average.
> 
> This immediately debunks your statement above.

true.

> $ git log v5.19..linus/master | grep Fixes: | wc -l
> 2928
> 
> So that's 46 bugs fixed per _day_.

But not really.  Many, perhaps even most, of these "Fixes:" are for code
introduced in -rc releases and so are a typical part of a development
cycle and are not for fixes in nominally released/final versions.

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hi,

> On Oct 2, 2022, at 11:56 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> 
> OK, then - please tell me how to prevent e.g. linux-fsdevel@vger.kernel.org
> getting spammed by that thing.  Where should I go and how do I unsubscribe
> it?

Exactly. It’d be nearly impossible, not to mention that you’d have to (somehow) do it for _a large number of lists_. 

I’m sure there is a better solution, but this isn’t it…

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Willy Tarreau]

On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
> The current ill-maintained semi-functional bugzilla has proven to be a
> ton more useful than random mailing lists no sane person can keep track
> of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
> is the worst of them probably.

You seem to completely miss the point. There's no need for *someone* to
keep track of the whole mailing lists, these mailing lists are used daily
by thousands of people. It's a *collective* effort. What matters is that
there exists someone among these people who will deal with your request.
Do patches fall through the cracks ? Sure! And so what ? The important
ones are eventually noticed or resent, and there's no harm in sending a
"ping" once in a while. Actually I find bug trackers worse for this,
because they give the reporter the impression that their report is being
handled while many times there's noone reading at the other end due to
the amount of stuff that has to be triaged. With mailing lists, a sender
who gets no response starts to wonder whether anything wrong happened and
is more naturally going to ask if the message was properly received, thus
reviving it. It's extremely rare that nobody responds to a retry on a first
message.

> As I've said many times already: bugzilla must be an opt-out, not opt-in
> experience/option.

That's the best way to make sure those who feel annoyed by this spam will
just redirect the bug tracker's address to /dev/null and will never ever
receive any message from it anymore. That's quite a common pattern, I'm
surprised that it's even still proposed as a solution...

> Let's subscribe the past six months of developers using git commits and
> if someone doesn't like getting emails they go to the website and
> unsubscribe _once_ which takes a minute. This is a non-issue I've no
> clue why we're dwelling on it.

Maybe because you have not yourself been spammed by bots that each
require a different way to unsubscribe/unregister/reconfigure options ?

> Let's operate with some examples:
> 
> Bugzilla gets around two dozen bug reports weekly which encompass at
> most thirty emails, which equals to four emails daily on average.

That's roughly what I was getting from github when I disabled all
notifications.

> LKML alone sees up to a hundred emails _daily_.

With a difference that these ones are not necessarily *read*, they're
*scanned* by many of us before being archived via a single- or two-key
shortcut, with a particular focus only on some messages or series
(hence the importance of a good subject).

Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Sun, 2 Oct 2022 12:49:04 +0000
"Artem S. Tashkinov" <aros@gmx.com> wrote:

> Let's subscribe the past six months of developers using git commits and
> if someone doesn't like getting emails they go to the website and
> unsubscribe _once_ which takes a minute. This is a non-issue I've no
> clue why we're dwelling on it.

As one of the few kernel maintainers that actually likes bugzilla and I
do not mind being subscribed to it, I too find the above an awful idea
(and I agree with all those that explained why it is so).

This really comes down to a manpower issue, which is common among most
open source projects. Remember it is commonly said that the only
warrantee you get from open source projects is that if it breaks, you
get to keep the pieces.

The issue is that the users of the Linux kernel mostly got it for free.
And if they did pay for it, it is highly unlikely that they paid the
kernel maintainer that owns the subsystem that they are having issues
with. That means, for the maintainers to triage these bug reports, they
are essentially doing it for free.

Some projects are better at this, and there are developers that are
happy to give free work, but there are also other projects that have
companies actively backing the work to debug these issues.

If you are using fedora, go bug Red Hat, Ubuntu then Canonical. And
again, it comes down to if you have a paid subscription or not if you
are going to get anywhere with it.

Can this be annoying, sure. But that's how the open source ecosystem
works.

If someone is not able to figure out how to use the mailing lists, it
is unlikely that they will be able to be useful in working with the
maintainer to solve their issue. As Ted mentioned, when asked to do
something to help analyze the issue, many times there's no response
from the reporter. Maybe because the reporter had no idea what the
maintainer wanted them to do. Most kernel bugs requires a constant back
and forth between the reporter and the developer. If you don't have
that, then there's no reason to bother with trying to fix the issue.

Ideally, someone (you?) would want to be a middle man and triage the
bugzilla reports and find those that look promising to get a fix
completed, and then be the liaison between bugzilla and the kernel
maintainer, then I think that could work. But the issue comes back to
manpower. Who's going to do that?

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Sun, Oct 2, 2022 at 2:49 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> > On Sat, Oct 01, 2022 at 02:58:04PM +0000, Artem S. Tashkinov wrote:
> Bugzilla gets around two dozen bug reports weekly which encompass at
> most thirty emails, which equals to four emails daily on average.

So we're discussing about the fate of a tool through which on average
four bugs per day are submitted (some of which are not very useful
due to lack of information)? A tool which is not maintained upstream?
Perhaps that's enough reason to just kill the tool, so no one has to
worry about it, or maintain it?

However, at four emails per day, you might as well just subscribe the
subsystem mailing lists (each of which would receive only a fraction
of that, right?).  Maintainers and several developers won't even notice
that minor increase in the number of daily emails received, although
they might still complain about the contents ;-)
And that still needs someone to keep the tool working...

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Joe Perches]

On Sun, 2022-10-02 at 09:32 -0700, Joe Perches wrote:
> On Sun, 2022-10-02 at 18:08 +0200, Geert Uytterhoeven wrote:
> > On Sun, Oct 2, 2022 at 2:49 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> > > The current ill-maintained semi-functional bugzilla has proven to be a
> > > ton more useful than random mailing lists no sane person can keep track
> > > of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
> > > is the worst of them probably.
> > 
> > Such a statement really needs to be backed by numbers...
> > 
> > > Let's operate with some examples:
> > > 
> > > Bugzilla gets around two dozen bug reports weekly which encompass at
> > > most thirty emails, which equals to four emails daily on average.
> > 
> > This immediately debunks your statement above.
> 
> true.
> 
> > $ git log v5.19..linus/master | grep Fixes: | wc -l
> > 2928
> > 
> > So that's 46 bugs fixed per _day_.
> 
> But not really.  Many, perhaps even most, of these "Fixes:" are for code
> introduced in -rc releases and so are a typical part of a development
> cycle and are not for fixes in nominally released/final versions.

Unless I stuffed something up, it looks like only about 2% of these
"Fixes: " commits were for existing defects in earlier non-rc releases.

$ git log --grep="^Fixes:" --no-merges --pretty=oneline v5.18..v5.19 | wc -l
2531

$ git log --grep="^Fixes:" --no-merges --pretty=oneline v5.18..v5.19 | \
  cut -f1 -d" " | \
  while read line ; do \
    echo -n "---> $line" ; \
    echo ":$(git rev-list --max-count=2 v5.18..$line | wc -l)" ; \
  done | \
  grep ":1" | wc -l
54

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 14:35, Greg KH wrote:
> On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
>>> And if we force developers to get Bugzilla spam whether they want it
>>> not, and they said, "absolutely not", is it there right to have the
>>> mailing list gateway disabled --- and if so, what does that do to the
>>> user experience?  Thats basically the situation we have right now.
>>
>> As I've said many times already: bugzilla must be an opt-out, not opt-in
>> experience/option.
>>
>> Let's subscribe the past six months of developers using git commits and
>> if someone doesn't like getting emails they go to the website and
>> unsubscribe _once_ which takes a minute. This is a non-issue I've no
>> clue why we're dwelling on it.
>
> auto-subscribing people to anything is a sure way to get lots of people
> instantly mad at you and have them add the address to their filters.
>
> That's just not how to do things well, sorry.
>
> If you wish to be the one triaging all bugzilla bugs, wonderful, please
> start doing so.  But to volunteer others and insist that they do it is a
> non-starter for obvious reasons.

It's so weird to read this I'm just dumbfounded.

People won't even receive emails if they are simply on bugzilla. It's
only if they get CC'ed to certain bug reports they'll receive them.

And they can unsubcribe literally after getting a single email. Can
anyone even get mad because of this? To me it feels like someone
sees/creates a drama where there's none.

If you're doing kernel development it's obvious that your email address
has been revealed and people are expected to deal with it.

I receive emails about Linux from random people I don't know and it's
never freaked me out. We are talking about service emails (not spam, not
automatic subscription) about their _work_.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 14:48, Slade Watkins wrote:
> Hi,
>
>> On Oct 2, 2022, at 8:49 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
>> As I've said many times already: bugzilla must be an opt-out, not opt-in
>> experience/option.
>>
>> Let's subscribe the past six months of developers using git commits and
>> if someone doesn't like getting emails they go to the website and
>> unsubscribe _once_ which takes a minute. This is a non-issue I've no
>> clue why we're dwelling on it.
>
> I disagree with this in its *entirety* and I really don’t think it has any chance of moving forward.
>
> If this were to happen (and it won’t!) then developers will just send the emails to spam or some other filter because they didn’t _consent_ to being subscribed to it. And in my opinion, they’d be justified in doing that.
>
> -srw
>

It was a proposal from no one, i.e. me.

The other option will be what? To _mass email_ everyone asking them to
subscribe to bugzilla? Do you know what will happen? 2/3 of relevant
people will forget about/neglect this email, they will never sign up
even if they are willing to and we'll end up with a disfunction bugzilla
again.

It feels to me we are back to:

"Users are expected to break their necks finding random mailing lists
and sending their reports to them expecting feedback".

95% of users will just give up.

4.95% of users will not receive any feedback: the developer has been
busy with their work, life, past time, etc - "Sorry, missed your email".

Maybe 0.05% bug reports will be actually dealt with.

Again this does not work for serious collaborations requiring multiple
people over extended periods of time. It absolutely sucks in terms of
filling in the missing details.

I begin to sound like a broken record repeating what we've already
discussed to death a dozen times.

Let's deprecate bugzilla and just say "f it". That's what I hear. Great!

No responsibility, no bug reports, no fixes, welcome regressions.

I concur. This discussion has been a complete waste of time.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 15:05, Konstantin Ryabitsev wrote:
> On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
>> I've mentioned it a dozen times already - you're unhappy with emails
>> from bugzilla? Go there and unsubscribe. It takes a minute and we're
>> talking as if it's the actual issue we are dealing with. It's not.
>> Bugzilla maintenance and its up-to-date status are the issues.
>
> I think you're working from the wrong conception that maintainers want to
> receive (or even be aware) of bug reports. Maintainers want *PATCHES*, not bug
> reports. You're asking senior engineers to do first-line QA.
>
> This is why your suggestion to auto-subscribe maintainers to bug reports is
> the absolutely wrong way to go about it. The maintainers will complain that
> they're being inundated with spam and junk reports, and bug reporters will
> complain that they are being treated rudely (because this is how a senior
> engineer trying to get succinct information comes across). You know that meme
> from Fallout with the words "[Everyone hated that]" on it?

There's no need to exaggerate or say words like "inundated", please
refer to my previous emails about bugzilla stats.

Again, to remind everyone, bugzilla sees around ~20 bug reports
_weekly_. There are hundreds of active of kernel developers. That means
for a single bug report maybe a couple of people will receive maybe a
few emails per week.

Is this really an _issue_?

Why are people are now blowing stuff out of proportion for no reason?
This conversation alone has already seen close to three dozen emails and
no one is complaining.

>
>>> And if we force developers to get Bugzilla spam whether they want it
>>> not, and they said, "absolutely not", is it there right to have the
>>> mailing list gateway disabled --- and if so, what does that do to the
>>> user experience?  Thats basically the situation we have right now.
>>
>> As I've said many times already: bugzilla must be an opt-out, not opt-in
>> experience/option.
>
> Please listen to the actual maintainers telling you that this won't work and
> will just lead to bugzilla being added to everyone's block list (that's even
> faster than logging in to bugzilla, finding where to change the default
> assignee, etc).

It feels to me there are no actual maintainers here. Just random people
who oppose to a imaginary torrent of emails they've never actually seen.

At the same time everyone here is OK getting a hundred emails from LKML
_daily_.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 16:08, Geert Uytterhoeven wrote:
> Hi Artem,
>
> On Sun, Oct 2, 2022 at 2:49 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>> The current ill-maintained semi-functional bugzilla has proven to be a
>> ton more useful than random mailing lists no sane person can keep track
>> of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
>> is the worst of them probably.
>
> Such a statement really needs to be backed by numbers...
>
>> Let's operate with some examples:
>>
>> Bugzilla gets around two dozen bug reports weekly which encompass at
>> most thirty emails, which equals to four emails daily on average.
>
> This immediately debunks your statement above.
>
> $ git log v5.19..linus/master | grep Fixes: | wc -l
> 2928
>
> So that's 46 bugs fixed per _day_.  Most of them not reported
> through bugzilla...

I don't know what it debunks.

Bugs reported in the last 7 days:

https://bugzilla.kernel.org/buglist.cgi?chfield=%5BBug%20creation%5D&chfieldfrom=7d

Check for yourself please.

I don't see anything even remotely close to your numbers and I tend to
believe you've calculated something incorrectly.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 17:57, Willy Tarreau wrote:
> On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
>> The current ill-maintained semi-functional bugzilla has proven to be a
>> ton more useful than random mailing lists no sane person can keep track
>> of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
>> is the worst of them probably.
>
> You seem to completely miss the point. There's no need for *someone* to
> keep track of the whole mailing lists, these mailing lists are used daily
> by thousands of people. It's a *collective* effort. What matters is that
> there exists someone among these people who will deal with your request.
> Do patches fall through the cracks ? Sure! And so what ? The important
> ones are eventually noticed or resent, and there's no harm in sending a
> "ping" once in a while. Actually I find bug trackers worse for this,
> because they give the reporter the impression that their report is being
> handled while many times there's noone reading at the other end due to
> the amount of stuff that has to be triaged. With mailing lists, a sender
> who gets no response starts to wonder whether anything wrong happened and
> is more naturally going to ask if the message was properly received, thus
> reviving it. It's extremely rare that nobody responds to a retry on a first
> message.

I've given multiple reasons why mailing lists more often than not do not
work at all, I won't repeat them again.

You can open LKML right now and check all the unreplied emails where
people complain about various issues. There are tons of such messages.

>
>> As I've said many times already: bugzilla must be an opt-out, not opt-in
>> experience/option.
>
> That's the best way to make sure those who feel annoyed by this spam will
> just redirect the bug tracker's address to /dev/null and will never ever
> receive any message from it anymore. That's quite a common pattern, I'm
> surprised that it's even still proposed as a solution...

Great, it's a single action which takes at most a minute. Why are people
creating a drama out of it I've no idea.

>
>> Let's subscribe the past six months of developers using git commits and
>> if someone doesn't like getting emails they go to the website and
>> unsubscribe _once_ which takes a minute. This is a non-issue I've no
>> clue why we're dwelling on it.
>
> Maybe because you have not yourself been spammed by bots that each
> require a different way to unsubscribe/unregister/reconfigure options ?

There's just one bugzilla. We are not talking about many bugzillas yet.

>
>> Let's operate with some examples:
>>
>> Bugzilla gets around two dozen bug reports weekly which encompass at
>> most thirty emails, which equals to four emails daily on average.
>
> That's roughly what I was getting from github when I disabled all
> notifications.
>
>> LKML alone sees up to a hundred emails _daily_.
>
> With a difference that these ones are not necessarily *read*, they're
> *scanned* by many of us before being archived via a single- or two-key
> shortcut, with a particular focus only on some messages or series
> (hence the importance of a good subject).

The problem is today you're scanning it, tomorrow you don't feel like
it. An email gets lost, a problem is never addressed.

Again most bug reports sent to LKML are completely neglected, and it's
_not_ limited to LKML.

Looks like that's what people here are advocating for. "We are not paid
to deal with bug reports, so we prefer not to even hear about them".

OK, let it be, let's deprecate bugzilla.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
> On 10/2/22 12:18, Theodore Ts'o wrote:
> > On Sat, Oct 01, 2022 at 02:58:04PM +0000, Artem S. Tashkinov wrote:
> >>
> >> My expectations are actually quite low:
> >>
> >> * A central place to collect bugs (yeah, bugzilla)
> >> * Proper up to date components (they don't change too often, so there's
> >> not a lot of work to be done - you can refresh them probably every 12-24
> >> months and it's gonna be totally OK)
> >> * An ability to CC the relevant people/mailing lists (this is the only
> >> serious missing feature)
> >>
> >> That's it. It's a billion times better than random emails sent to random
> >> mailing lists. Signing up once is easier that to keep track of whom and
> >> where you've emailed or not. And of course it's a ton lot easier to find
> >> the existing bug reports.
> >
> > First of all, some of the components do CC the relevant mailing lists
> > automatically.  And this is the part of Bugzilla which is hand-hacked
> > and has no, zero, nada support upstream.  I'll defer to Konstantin
> > about how easy it is to keep that working.
> >
> > Secondly, not everyone is happy with getting an e-mail message sent to
> > a mailing list that has a lot of bugzilla metadata associated with it,
> > and depending on how they respond, the response might not make it back
> > to bugzilla.
> 
> I've mentioned it a dozen times already - you're unhappy with emails
> from bugzilla? Go there and unsubscribe. It takes a minute and we're
> talking as if it's the actual issue we are dealing with. It's not.
> Bugzilla maintenance and its up-to-date status are the issues.
> 
> >
> > Finally, in the absense of someone to actually close bugzilla entries
> > and do other necessary grooming, the bugzilla database will rapidly
> > become useless --- in which case, you might as well have a web form
> > that just helps the user send e-mail to the mailing list, and hope it
> > doesn't become a SPAM magnet.
> 
> The current ill-maintained semi-functional bugzilla has proven to be a
> ton more useful than random mailing lists no sane person can keep track
> of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
> is the worst of them probably.
> 
> >
> >> Bugzilla as it is works nearly perfectly. We have a number of developers
> >> who don't want to touch it or get emails from it - it's their right.
> >> However it would be madness to take it from users. That will make filing
> >> and following up on bug reports an absolutely poor experience for
> >> absolute most users.
> >
> > At the moment, developers aren't following up on the bug reports.
> > There is some debate as to why.  Is it because users who can't figure
> > out how to send e-mail, and who send web-form based e-mails send low
> > quality bug reports that can't be easily responded to unless someone
> > is paid $$$ and/or has the patience of a saint?  Is it because
> > components aren't being gatewayed to mailing lists?
> 
> This is not always true, some of them do, some of them actually check
> new bug reports and do a tremendous job at helping people, e.g. Mario
> Limonciello who helps resolve bugs which are not even his direct
> responsibility. BTW, I'll now CC him since he's so active over there.
> Would be great if he chimed in.
> 
> > And if we force developers to get Bugzilla spam whether they want it
> > not, and they said, "absolutely not", is it there right to have the
> > mailing list gateway disabled --- and if so, what does that do to the
> > user experience?  Thats basically the situation we have right now.
> 
> As I've said many times already: bugzilla must be an opt-out, not opt-in
> experience/option.
> 
> Let's subscribe the past six months of developers using git commits and
> if someone doesn't like getting emails they go to the website and
> unsubscribe _once_ which takes a minute. This is a non-issue I've no
> clue why we're dwelling on it.

I'm not sure that would be legal, at least in the EU.

> Let's operate with some examples:
> 
> Bugzilla gets around two dozen bug reports weekly which encompass at
> most thirty emails, which equals to four emails daily on average.
> 
> LKML alone sees up to a hundred emails _daily_.
> 
> Getting worked up about it? I'm dumbfounded to be honest.

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 18:13, Steven Rostedt wrote:
> On Sun, 2 Oct 2022 12:49:04 +0000
> "Artem S. Tashkinov" <aros@gmx.com> wrote:
>
>> Let's subscribe the past six months of developers using git commits and
>> if someone doesn't like getting emails they go to the website and
>> unsubscribe _once_ which takes a minute. This is a non-issue I've no
>> clue why we're dwelling on it.
>
> As one of the few kernel maintainers that actually likes bugzilla and I
> do not mind being subscribed to it, I too find the above an awful idea
> (and I agree with all those that explained why it is so).

Good, exactly what I've been advocating for, those who like it and can
help are welcome, others can go unsubscribe in under a minute. No drama.

>
> This really comes down to a manpower issue, which is common among most
> open source projects. Remember it is commonly said that the only
> warrantee you get from open source projects is that if it breaks, you
> get to keep the pieces.
>
> The issue is that the users of the Linux kernel mostly got it for free.
> And if they did pay for it, it is highly unlikely that they paid the
> kernel maintainer that owns the subsystem that they are having issues
> with. That means, for the maintainers to triage these bug reports, they
> are essentially doing it for free.

I perfectly understand it. I've _not_ asked anyone to do anything yet,
except maybe have their email in the bugzilla database, so that people
_could_ CC you.

They will _not_ do it right away. They first have to `git grep` commits,
find the relevant developers and then CC them.

>
> Some projects are better at this, and there are developers that are
> happy to give free work, but there are also other projects that have
> companies actively backing the work to debug these issues.
>
> If you are using fedora, go bug Red Hat, Ubuntu then Canonical. And
> again, it comes down to if you have a paid subscription or not if you
> are going to get anywhere with it.

This does not work, period. Most kernel bug reports in Fedora and Ubuntu
bug trackers linger for years, sometimes someone says, "Try the vanilla
kernel and if it's still an issue, please use the kernel bugzilla".

My Fedora kernel bug reports have been dealt with exactly this way.

RedHat does solve kernel issues in the RHEL kernel if you have a paid
subscription and you spend quite some time providing them with a perfect
reproducible test case. This is far outside this conversation.

>
> Can this be annoying, sure. But that's how the open source ecosystem
> works.
>
> If someone is not able to figure out how to use the mailing lists, it
> is unlikely that they will be able to be useful in working with the
> maintainer to solve their issue. As Ted mentioned, when asked to do
> something to help analyze the issue, many times there's no response
> from the reporter. Maybe because the reporter had no idea what the
> maintainer wanted them to do. Most kernel bugs requires a constant back
> and forth between the reporter and the developer. If you don't have
> that, then there's no reason to bother with trying to fix the issue.

Mailing lists more often than not do not work, and maybe worked in the
early 90s.

We don't need to resolve the issue right away. We don't have to deal
with it. We just need a place where people could find existing issues
and add their input. That's a lot better than chasing something in emails.

Here's the simplest example.

Person A installs kernel 6.0. They find a regression. They send an email
to maling list X. Not necessarily the relevant one and the email is
simply ignored.

Another person finds the same regression. This person B may not be aware
of the mailing list used earlier. They send a bug report elsewhere.

Now we have two completely disconnected bug reports which if luck allows
could be Googled. Oy, you must know what to google for. Not that many
people have a good Google foo.

Now with bugzilla.

Anyone opens the last seven days of bug reports and instantly sees that
something similar has already been filed and dealt with. Collaboration
ensues. Maybe just maybe some developer will join it and actually offer
a fix. If not, OK, fine, no big deal but at least it's _known_,
_visible_ and can be _found_.

Random unreplied emails God knows where? Good luck with that.

>
> Ideally, someone (you?) would want to be a middle man and triage the
> bugzilla reports and find those that look promising to get a fix
> completed, and then be the liaison between bugzilla and the kernel
> maintainer, then I think that could work. But the issue comes back to
> manpower. Who's going to do that?

I've already offered myself. The LF has no such position. And more
importantly I'm from a totalitarian country, so I'm unlikely to be ever
employed.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 18:17, Geert Uytterhoeven wrote:
> Hi Artem,
>
> On Sun, Oct 2, 2022 at 2:49 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>>> On Sat, Oct 01, 2022 at 02:58:04PM +0000, Artem S. Tashkinov wrote:
>> Bugzilla gets around two dozen bug reports weekly which encompass at
>> most thirty emails, which equals to four emails daily on average.
>
> So we're discussing about the fate of a tool through which on average
> four bugs per day are submitted (some of which are not very useful
> due to lack of information)? A tool which is not maintained upstream?
> Perhaps that's enough reason to just kill the tool, so no one has to
> worry about it, or maintain it?

Yep, there are not that many savvy Linux users out there. Filing a good
bug report takes a lot of experience to be honest.

And most bugs reports in Bugzilla are semi-automatically closed (by me)
because they are about amdgpu and i915 which have their own bug
trackers. Without such reports the volume of bug reports is basically
halved.

>
> However, at four emails per day, you might as well just subscribe the
> subsystem mailing lists (each of which would receive only a fraction
> of that, right?).  Maintainers and several developers won't even notice
> that minor increase in the number of daily emails received, although
> they might still complain about the contents ;-)
> And that still needs someone to keep the tool working..

Oh, God, finally.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 19:59, Laurent Pinchart wrote:
>> Let's subscribe the past six months of developers using git commits and
>> if someone doesn't like getting emails they go to the website and
>> unsubscribe _once_ which takes a minute. This is a non-issue I've no
>> clue why we're dwelling on it.
>
> I'm not sure that would be legal, at least in the EU.

1. That's already been done at least once AFAIK.
2. I'm talking about emails in public domain (git log). It's not a
special forces operation to exfiltrate an email database.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Sun, Oct 02, 2022 at 08:19:44PM +0000, Artem S. Tashkinov wrote:
> On 10/2/22 19:59, Laurent Pinchart wrote:
> >> Let's subscribe the past six months of developers using git commits and
> >> if someone doesn't like getting emails they go to the website and
> >> unsubscribe _once_ which takes a minute. This is a non-issue I've no
> >> clue why we're dwelling on it.
> >
> > I'm not sure that would be legal, at least in the EU.
> 
> 1. That's already been done at least once AFAIK.
> 2. I'm talking about emails in public domain (git log). It's not a
> special forces operation to exfiltrate an email database.

That may or may not matter. If we decided to go in that direction (this
doesn't seem to be the consensus so far, but that could change), we
would need legal vetting.

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Willy Tarreau]

On Sun, Oct 02, 2022 at 07:55:35PM +0000, Artem S. Tashkinov wrote:
> 
> 
> On 10/2/22 17:57, Willy Tarreau wrote:
> > On Sun, Oct 02, 2022 at 12:49:04PM +0000, Artem S. Tashkinov wrote:
> > > The current ill-maintained semi-functional bugzilla has proven to be a
> > > ton more useful than random mailing lists no sane person can keep track
> > > of. Bug "reports", i.e. random emails are neglected and forgotten. LKML
> > > is the worst of them probably.
> > 
> > You seem to completely miss the point. There's no need for *someone* to
> > keep track of the whole mailing lists, these mailing lists are used daily
> > by thousands of people. It's a *collective* effort. What matters is that
> > there exists someone among these people who will deal with your request.
> > Do patches fall through the cracks ? Sure! And so what ? The important
> > ones are eventually noticed or resent, and there's no harm in sending a
> > "ping" once in a while. Actually I find bug trackers worse for this,
> > because they give the reporter the impression that their report is being
> > handled while many times there's noone reading at the other end due to
> > the amount of stuff that has to be triaged. With mailing lists, a sender
> > who gets no response starts to wonder whether anything wrong happened and
> > is more naturally going to ask if the message was properly received, thus
> > reviving it. It's extremely rare that nobody responds to a retry on a first
> > message.
> 
> I've given multiple reasons why mailing lists more often than not do not
> work at all, I won't repeat them again.

Indeed, it just seems that many others don't share your opinion.

> You can open LKML right now and check all the unreplied emails where
> people complain about various issues. There are tons of such messages.

I don't have to "open LKML", it arrives in my mailbox and I glance over
it a few times a day in search for interesting topics or reports before
archiving it. By the way, how long do you wait before declaring a message
"unreplied" ? For me, "unreplied" starts at one week. And when I sort the
messages by thread and go back one week, I see very few such unreplied
messages. Interestingly most of the unreplied ones are LKP reports. For
sure there will be such messages lost, but it's definitely not "a ton"
like you're saying. Probably that your analysis is skewed by the fact
that many messages are patches, but one response to the cover message
is enough.

> > > As I've said many times already: bugzilla must be an opt-out, not opt-in
> > > experience/option.
> > 
> > That's the best way to make sure those who feel annoyed by this spam will
> > just redirect the bug tracker's address to /dev/null and will never ever
> > receive any message from it anymore. That's quite a common pattern, I'm
> > surprised that it's even still proposed as a solution...
> 
> Great, it's a single action which takes at most a minute. Why are people
> creating a drama out of it I've no idea.

Because if every bug tracker operator started to act like this it would
really be annoying. Why would the kernel's bugzilla have the right to
spam many people, and not all trac/gitlab/github/whatever tracker of
whatever other projects developers also follow ? It's like when spammers
used to say that spam isn't a problem, it just requires deleting a message
once in a while, why are people making a fuss about this.

> > > Let's subscribe the past six months of developers using git commits and
> > > if someone doesn't like getting emails they go to the website and
> > > unsubscribe _once_ which takes a minute. This is a non-issue I've no
> > > clue why we're dwelling on it.
> > 
> > Maybe because you have not yourself been spammed by bots that each
> > require a different way to unsubscribe/unregister/reconfigure options ?
> 
> There's just one bugzilla. We are not talking about many bugzillas yet.

This "yet" has a lot of meaning for so few letters.

> > With a difference that these ones are not necessarily *read*, they're
> > *scanned* by many of us before being archived via a single- or two-key
> > shortcut, with a particular focus only on some messages or series
> > (hence the importance of a good subject).
> 
> The problem is today you're scanning it, tomorrow you don't feel like
> it. An email gets lost, a problem is never addressed.

No. Again, mainling lists are not read by a single person but by many
thousands. Nobody's essential there. Plenty of people can sleep, visit
family, go fishing or to vacation with basically no impact. Actually I'm
way more worried about your model where you would be almost alone doing
the triage on bugzilla after most annoyed ones unsubscribe.

> Again most bug reports sent to LKML are completely neglected, and it's
> _not_ limited to LKML.

No. "Some", likely. "Many", possibly. "Most", no. The most recent one
that qualifies that I've found is "PROBLEM: Segfault in kconfig" dated
22/9 which actually I had read, and I think few people feel legitimate
about:

  https://lore.kernel.org/all/33059074b78110d4717efe09b887dd28ac77fe7f.camel@pengutronix.de/

The bug tracker would unlikely have changed this.

> Looks like that's what people here are advocating for. "We are not paid
> to deal with bug reports, so we prefer not to even hear about them".

No that's not how it works, any developer is interested with bug reports
in their respective areas. It's just that some areas are poorly covered
and that a bug tracker will not suddenly cause the missing skills to
appear out of nowhere.

Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hello,

> On Oct 2, 2022, at 2:13 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> Ideally, someone (you?) would want to be a middle man and triage the
> bugzilla reports and find those that look promising to get a fix
> completed, and then be the liaison between bugzilla and the kernel
> maintainer, then I think that could work. But the issue comes back to
> manpower. Who's going to do that?

I *really* like this idea. And I think it’s also worth a shot. And if folks think so too, then I think in terms of manpower… I may have a solution:

I’d be more than happy to step up — new system or the current in-place bugzilla. (Believe me, triaging bug reports is actually something I enjoy.) It’d take some time to get adjusted and acquainted with dealing with reports, obviously, but I think I'd be able to take it on in addition to responsibilities in my personal life. 

Consider this e-mail as me indicating my interest in stepping up (but not a commitment, just yet, I would like to hear from others on Steven's idea first.)

Thanks,
-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Linus Torvalds]

On Sun, Oct 2, 2022 at 12:27 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>
> It's so weird to read this I'm just dumbfounded.

The *real* dumbfoundedness here is how you seem to think you have the
right to tell people how to work, when you've been told over and over
how bugzilla doesn't fit in the workflow, and isn't worth the effort
and time.

And when people tell you that if you care about bugzilla, then it is
*you* who should spend the time there, you say that you are
dumbfounded. Because only you can tell people how they should work,
right?

Please go away, and look in the mirror a bit.

                  Linus

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Willy Tarreau]

On Sun, Oct 02, 2022 at 08:17:29PM +0000, Artem S. Tashkinov wrote:
> And most bugs reports in Bugzilla are semi-automatically closed (by me)
> because they are about amdgpu and i915 which have their own bug
> trackers. Without such reports the volume of bug reports is basically
> halved.

So that's fragmenting the locations where users have to look for issues
and report them, the thing you were complaining about regarding mailing
lists. So the problem is the same, except that it's based on web instead
of e-mail. At least e-mail threads don't have a "closed" status that
makes it more difficult for reporters to find what they're looking for,
and tends to be ignored by everyone since not expected to change anymore.

Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Following up on something I said earlier:

> On Oct 2, 2022, at 4:41 PM, Slade Watkins <srw@sladewatkins.net> wrote:
> 
> I’d be more than happy to step up — new system or the current in-place bugzilla. (Believe me, triaging bug reports is actually something I enjoy.) It’d take some time to get adjusted and acquainted with dealing with reports, obviously, but I think I'd be able to take it on in addition to responsibilities in my personal life. 

Honestly, despite my preferences towards Bugzilla (many years worth of bias), I think it should be ousted for something different. It seems it doesn’t really fit in many people’s workflows and seems like the consensus is it's just a hassle to deal with. And that’s fair! 

To be clear: I’d still be willing to triage bugs/issues/whatever you want to call them (on Bugzilla or off). I’ll help where I can!

Thanks,
-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Willy Tarreau]

On Sun, Oct 02, 2022 at 07:43:19PM +0000, Artem S. Tashkinov wrote:
> Again, to remind everyone, bugzilla sees around ~20 bug reports
> _weekly_. There are hundreds of active of kernel developers. That means
> for a single bug report maybe a couple of people will receive maybe a
> few emails per week.
> 
> Is this really an _issue_?
> 
> Why are people are now blowing stuff out of proportion for no reason?

Because the approach is wrong. As I explained it gives a false sense to
the reporter that their issue is being handled while the simple fact that
a message was sent to a person is in no way an engagement to do anything
about it. LKML is a broadcast area. Everyone hopes someone else will
respond and that eventually happens. When the reports are targetted, it
puts pressure on the few developers receiving the message who know that
it's unlikely anyone else will deal with that report.

> This conversation alone has already seen close to three dozen emails and
> no one is complaining.

Because it's easy to ignore. Try to setup this conversation in your
favorite bug tracker and you'll feel alone discussing with yourself.
This is a great indication that participation is much more powerful
in the mailing list model than in the bug tracker model.

Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 20:46, Linus Torvalds wrote:
> On Sun, Oct 2, 2022 at 12:27 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>>
>> It's so weird to read this I'm just dumbfounded.
>
> The *real* dumbfoundedness here is how you seem to think you have the
> right to tell people how to work, when you've been told over and over
> how bugzilla doesn't fit in the workflow, and isn't worth the effort
> and time.
>
> And when people tell you that if you care about bugzilla, then it is
> *you* who should spend the time there, you say that you are
> dumbfounded. Because only you can tell people how they should work,
> right?
>
> Please go away, and look in the mirror a bit.

Linus, I've _not_ told anyone how to work and what to do. Not in a
single email message so far.

I just want a bugzilla where I can CC _any_ developer _if_ and _only if_
they are willing to work within its confounds. That's it.


There have been no other proposals from me other than killing the entire
thing as many here have blown out of proportions the amount of emails it
generates or the amount of attention it requires.

If you want me to go away though you've actually helped _me_ solve quite
serious bugs _using_ bugzilla (which I bet would be quite difficult to
approach using emails), so be it.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Sun, Oct 02, 2022 at 02:13:20PM -0400, Steven Rostedt wrote:
> On Sun, 2 Oct 2022 12:49:04 +0000 Artem S. Tashkinov wrote:
> 
> > Let's subscribe the past six months of developers using git commits and
> > if someone doesn't like getting emails they go to the website and
> > unsubscribe _once_ which takes a minute. This is a non-issue I've no
> > clue why we're dwelling on it.
> 
> As one of the few kernel maintainers that actually likes bugzilla and I
> do not mind being subscribed to it, I too find the above an awful idea
> (and I agree with all those that explained why it is so).
> 
> This really comes down to a manpower issue, which is common among most
> open source projects. Remember it is commonly said that the only
> warrantee you get from open source projects is that if it breaks, you
> get to keep the pieces.
> 
> The issue is that the users of the Linux kernel mostly got it for free.
> And if they did pay for it, it is highly unlikely that they paid the
> kernel maintainer that owns the subsystem that they are having issues
> with. That means, for the maintainers to triage these bug reports, they
> are essentially doing it for free.
> 
> Some projects are better at this, and there are developers that are
> happy to give free work, but there are also other projects that have
> companies actively backing the work to debug these issues.
> 
> If you are using fedora, go bug Red Hat, Ubuntu then Canonical. And
> again, it comes down to if you have a paid subscription or not if you
> are going to get anywhere with it.
> 
> Can this be annoying, sure. But that's how the open source ecosystem
> works.

The dichotomy between the community/hobbyist/free side and the
commercial/professional/paid side is an argument I often hear, and often
make myself. It is not, however, ineluctable.

We have shown multiple times that this barrier doesn't have to exist.
The kernel is an impressive example of how companies and communities can
cooperate to reach a result that no single entity could have achieved.
It started with the development model and how it scaled, and other areas
were tackled along the way, such as automated testing and quality
control in general for instance. Lots of efforts went into creating
solutions that could fulfil the unique needs of our development model,
and into convincing large and small companies to invest, either money or
time.

Are we doing a great job ? Certainly not. But we are moving forward. As
Jon Corbet said several years ago in one of his talks, now that the
Linux kernel has reached a leading position in many areas, we have lost
the comfort of following other industry actors and have to innovate
ourselves. That often means (and this thought is mine, not Jon's)
winging it along the way. As impressive as some of our achievements may
be, our failures to maintain some areas of the kernel in a professional
way is also astonishing (https://xkcd.com/2347/ comes to mind). It's not
entirely surprising: the community and (part-time) volunteer model means
that everybody will tackle problems that interest them. Building a
community that can deliver professional support is not an interesting
task for everybody. It is, however, a key factor in the difference
between a kernel subsystem that strives and a subsystem that survives.

I believe the same holds true for bug tracking and support. At the end
of the day, someone will need to pay for it, but we could shatter the
traditional model here too. We could, given enough interest, bridge the
gap between all involved parties and create a support model that would
benefit everybody. It took years and huge efforts for Linux to evolve
towards more professionalism in many areas, and it would take more years
and more effort to continue and expand that, but I believe it would be
feasible.

Linux didn't start because Linus complained about existing operating
systems, ranted on usenet forums and waited for someone to fix the
problem. Someone will need to step up and lead the effrot here too. If
that person could ignore for a while that this is an impossible task, I
think they could succeed.

> If someone is not able to figure out how to use the mailing lists, it
> is unlikely that they will be able to be useful in working with the
> maintainer to solve their issue. As Ted mentioned, when asked to do
> something to help analyze the issue, many times there's no response
> from the reporter. Maybe because the reporter had no idea what the
> maintainer wanted them to do. Most kernel bugs requires a constant back
> and forth between the reporter and the developer. If you don't have
> that, then there's no reason to bother with trying to fix the issue.
> 
> Ideally, someone (you?) would want to be a middle man and triage the
> bugzilla reports and find those that look promising to get a fix
> completed, and then be the liaison between bugzilla and the kernel
> maintainer, then I think that could work. But the issue comes back to
> manpower. Who's going to do that?

On the topic of triage, I've found that distro developers often do a
pretty good job. I've received multiple bug reports of great quality
following problems initially posted to distro bug trackers, after the
distro developers took the time needed to hold reporters by the hand to
get all the required information. Kudos for that !

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 20:54, Willy Tarreau wrote:
> On Sun, Oct 02, 2022 at 07:43:19PM +0000, Artem S. Tashkinov wrote:
>> Again, to remind everyone, bugzilla sees around ~20 bug reports
>> _weekly_. There are hundreds of active of kernel developers. That means
>> for a single bug report maybe a couple of people will receive maybe a
>> few emails per week.
>>
>> Is this really an _issue_?
>>
>> Why are people are now blowing stuff out of proportion for no reason?
>
> Because the approach is wrong. As I explained it gives a false sense to
> the reporter that their issue is being handled while the simple fact that
> a message was sent to a person is in no way an engagement to do anything
> about it. LKML is a broadcast area. Everyone hopes someone else will
> respond and that eventually happens. When the reports are targetted, it

No, it doesn't happen. Should I open LKML and send you a hundred of
unreplied emails over the past year alone?

> puts pressure on the few developers receiving the message who know that
> it's unlikely anyone else will deal with that report.

"Pressure", "spam", I've completely lost you.

My first proposal was to let people _unsubscribe_ which takes a _minute_
if they hate this kind of workflow. And then I calculated how many
emails a particular developer may receive. In the worst case scenario
fewer than five in a week.

What a drama.



Just before I GTFO I will leave this bug report here (already posted it
here but maybe I need to do it again and again):

https://bugzilla.kernel.org/show_bug.cgi?id=204807

Tell me honestly how ~255 comments, and a ton of collaboration over the
span of 2.5 years can be managed using email.

>
>> This conversation alone has already seen close to three dozen emails and
>> no one is complaining.
>
> Because it's easy to ignore. Try to setup this conversation in your
> favorite bug tracker and you'll feel alone discussing with yourself.
> This is a great indication that participation is much more powerful
> in the mailing list model than in the bug tracker model.

OK, let's kill the damn thing.

Let's have random emails with duplicated issues over 50+ mailing lists
no one sees, maybe some of them are replied to. Maybe 1% of them are
actually dealt with.

After all, it's all for fun despite > 95% of kernel contributions coming
from people who are really well paid working for major corporations such
as Intel and AMD.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Linus Torvalds]

On Sun, Oct 2, 2022 at 1:56 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>
> I just want a bugzilla where I can CC _any_ developer _if_ and _only if_
> they are willing to work within its confounds. That's it.

Guess what that "add develooper to the Cc" is called?

Email.

What you do is fill in the bugzilla entry with all the data you want.

Then you use email to inform people about it.

Put enough data in the email that the developer knows whether it's
even worth looking at the bugzilla entry or not. Don't just put a link
to the bugzilla. Most developers will just go "oh, this looks like
spam"., Put the overview in the email, enough information that the
developer can go "Ahh, this is worth my time", _and_ the link to
bugzilla.

That gives you exactly what you ask for: you can CC _any_ developer.
And it doesn't force the developer to have to go to some bugzilla web
interface unless the developer thinks it actually adds value.

This is *literally* how I end up using bugzilla. As you say, I
actually do end up looking at bugzilla entries in the end, but I only
do it once it has hit my mailbox first, and I have some fairly good
indication that it's worth my time to look at it.

And yes, for some projects and for some developers you can do that
email integration from within bugzilla itself. That's how people reach
me.

But this is exactly the kind of part of bugzilla that is a TOTAL
HORROR-SHOW to manage, and it's impossible to expect every developer
to be somebody that can be listed on bugzilla, without bugzilla
becoming a prime way to send spam.

Which is why in the general case, you really should consider email to
be the "lingua franca" of kernel development communication.  It
doesn't have the fundamental limitations and management issues that
bugzilla has. If you want to add more people to the Cc in an email,
you just do it.

             Linus

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Sun, Oct 02, 2022 at 07:37:38PM +0000, Artem S. Tashkinov wrote:
> On 10/2/22 14:48, Slade Watkins wrote:
> >> On Oct 2, 2022, at 8:49 AM, Artem S. Tashkinov <aros@gmx.com> wrote:
> >> As I've said many times already: bugzilla must be an opt-out, not opt-in
> >> experience/option.
> >>
> >> Let's subscribe the past six months of developers using git commits and
> >> if someone doesn't like getting emails they go to the website and
> >> unsubscribe _once_ which takes a minute. This is a non-issue I've no
> >> clue why we're dwelling on it.
> >
> > I disagree with this in its *entirety* and I really don’t think it
> > has any chance of moving forward.
> >
> > If this were to happen (and it won’t!) then developers will just
> > send the emails to spam or some other filter because they didn’t
> > _consent_ to being subscribed to it. And in my opinion, they’d be
> > justified in doing that.
> 
> It was a proposal from no one, i.e. me.
> 
> The other option will be what? To _mass email_ everyone asking them to
> subscribe to bugzilla? Do you know what will happen? 2/3 of relevant
> people will forget about/neglect this email, they will never sign up
> even if they are willing to and we'll end up with a disfunction bugzilla
> again.
> 
> It feels to me we are back to:
> 
> "Users are expected to break their necks finding random mailing lists
> and sending their reports to them expecting feedback".
> 
> 95% of users will just give up.
> 
> 4.95% of users will not receive any feedback: the developer has been
> busy with their work, life, past time, etc - "Sorry, missed your email".
> 
> Maybe 0.05% bug reports will be actually dealt with.
> 
> Again this does not work for serious collaborations requiring multiple
> people over extended periods of time. It absolutely sucks in terms of
> filling in the missing details.
> 
> I begin to sound like a broken record repeating what we've already
> discussed to death a dozen times.
> 
> Let's deprecate bugzilla and just say "f it". That's what I hear. Great!
> 
> No responsibility, no bug reports, no fixes, welcome regressions.
> 
> I concur. This discussion has been a complete waste of time.

Do you realize how insulting this is, for all the developers and
maintainers who spend lots of their free time doing their best ? It's
all very nice to complain and rant, but if you want things to move
forward, lead the effort and work on it.

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 21:07, Linus Torvalds wrote:
> On Sun, Oct 2, 2022 at 1:56 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>>
>> I just want a bugzilla where I can CC _any_ developer _if_ and _only if_
>> they are willing to work within its confounds. That's it.
>
> Guess what that "add develooper to the Cc" is called?
>
> Email.
>
> What you do is fill in the bugzilla entry with all the data you want.
>
> Then you use email to inform people about it.
>
> Put enough data in the email that the developer knows whether it's
> even worth looking at the bugzilla entry or not. Don't just put a link
> to the bugzilla. Most developers will just go "oh, this looks like
> spam"., Put the overview in the email, enough information that the
> developer can go "Ahh, this is worth my time", _and_ the link to
> bugzilla.

That could work or could not work because e.g. my email provider,
gmx.com, is blocked by a large number of companies/people because ...
reasons. If I'm lucky I get a reply: "Mail cannot be delivered because
reasons". Sometimes I get no reply, there's no way of knowing your email
has been delivered.

More importantly there's no way of knowing an email has been read or
seen. You perfectly know that.

An email from an official bug tracker, not some random Joe on the
Internet? That will less likely be considered SPAM, fake, scam, etc.
Some people have no idea that kernel developers prefer to deal with
plain text and may simply ignore HTML messages which have long become
the default for many email providers.

You're asking people to know in advance the kernel developers email
etiquette or know English well enough to communicate your problem.
Bugzilla doesn't need that. You can dump whatever you want however you
want even in your native language.

>
> That gives you exactly what you ask for: you can CC _any_ developer.
> And it doesn't force the developer to have to go to some bugzilla web
> interface unless the developer thinks it actually adds value.
>
> This is *literally* how I end up using bugzilla. As you say, I
> actually do end up looking at bugzilla entries in the end, but I only
> do it once it has hit my mailbox first, and I have some fairly good
> indication that it's worth my time to look at it.
>
> And yes, for some projects and for some developers you can do that
> email integration from within bugzilla itself. That's how people reach
> me.
>
> But this is exactly the kind of part of bugzilla that is a TOTAL
> HORROR-SHOW to manage, and it's impossible to expect every developer
> to be somebody that can be listed on bugzilla, without bugzilla
> becoming a prime way to send spam.

There are many easier ways to SPAM people ;-) I've not heard of anyone
complaining about SPAM coming from the kernel bugzilla yet.

>
> Which is why in the general case, you really should consider email to
> be the "lingua franca" of kernel development communication.  It
> doesn't have the fundamental limitations and management issues that
> bugzilla has. If you want to add more people to the Cc in an email,
> you just do it.

Attention, Linus, the problem is attention.

Once something is filed in bugzilla, it's public, it's easily
accessible, it can easily be found, you can easily add new info.

Emails? You've flown to Japan to a conference for a week and you have
much better things than to check any email updates. A week worth of
emails have suddenly become worthless.

Here's yet another issue, how would you send a follow up if you don't
know the reference ("References" email field)? Instead of a follow up
it'll end up being a new unrelated email.

Lastly, if you're on bugzilla, your email address is a lot less likely
to be leaked. Bugzilla doesn't publish emails. Public mailing lists are
used to collect email address for SPAM databases all the time.

Since kernel developers/Linux users love privacy so much, I guess it's a
good argument for Bugzilla, not against it. Many email clients/providers
leak a ton of information about you (email client version, timezone,
even IP address) left and right.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Willy Tarreau]

On Sun, Oct 02, 2022 at 09:07:13PM +0000, Artem S. Tashkinov wrote:
> > > Why are people are now blowing stuff out of proportion for no reason?
> > 
> > Because the approach is wrong. As I explained it gives a false sense to
> > the reporter that their issue is being handled while the simple fact that
> > a message was sent to a person is in no way an engagement to do anything
> > about it. LKML is a broadcast area. Everyone hopes someone else will
> > respond and that eventually happens. When the reports are targetted, it
> 
> No, it doesn't happen. Should I open LKML and send you a hundred of
> unreplied emails over the past year alone?

If that makes you feel better, feel free to do so. I'm not scared by
only one hundred e-mails. What I'm impressed by, however, is that you're
able to spot that many unreplied e-mails because I don't see as many. If
you're that efficient at spotting them, maybe these are the ones you
should just resend to make sure they're seen, and it would require less
work (even on your side) than triaging issues.

> Just before I GTFO I will leave this bug report here (already posted it
> here but maybe I need to do it again and again):
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=204807
> 
> Tell me honestly how ~255 comments, and a ton of collaboration over the
> span of 2.5 years can be managed using email.

What makes you think it would have taken that long over e-mail ? Between
your first report and the first reply "this is not a bug", 18 months had
elapsed already. The most active part of the discussion happened grouped
on 3 days (2021-03-19 -> 22), where there were already some "I'm removing
myself from the CC because the discussion isn't productive", then a large
number of "me too" happened. Not sure how much useful this has been
overall to the involved developers, given that it's impossible to stay
focused on that long a thread and sum up all the information spanning
over that many kernel versions and that many different hardware.

My gut feeling is that handling this over the ML would have resulted in:
  - a few "sorry, no solution, try to fix your BIOS"
  - "try this" => "it works, thank you".
  - "this fix above broke for me"
  - and a few such iterations until a satisfying enough solution would
    have been found. Maybe not in 2.5 years, maybe 6 months.

But I could be wrong. I'm not claiming I know how people feel the most
efficient. Just observing what we're seeing on the lists and what I'm
used to dealing with in some bug trackers. If you want I can as well
show you a bug I reported 19 years ago that's still in state "NEW",
having seen little updates over the years. It had better been closed
since then, TBH:

   https://gcc.gnu.org/bugzilla/show_bug.cgi?format=multiple&id=11873

Pretty close to your demo above except it lasted 8 times longer and
has not seen progress by lack of interest. How's that different from
what you complain about mailing lists ? Hmm ?

Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 21:11, Laurent Pinchart wrote:
>
> Do you realize how insulting this is, for all the developers and
> maintainers who spend lots of their free time doing their best ? It's
> all very nice to complain and rant, but if you want things to move
> forward, lead the effort and work on it.
>

Considering how much I've poured into this discussion already and all my
_unpaid_ work to improve Linux over the past 25+ years, to accuse me of
this sounds like pure insult.

The part your replied to? That was thinly veiled sarcasm seeing such
strong resistance from people who have seemingly never even visited
bugzilla. I'm appalled you took it at face value.

I'm taken aback by people who oppose bugzilla which has helped resolve
criticial issues and implement important features in a manner which
wouldn't be possible using email. It's my only reaction. I'm alive and
it pains me to see a decent collaborative tool being harshly criticized
to no end with no _working_ alternatives being proposed.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Willy Tarreau]

On Sun, Oct 02, 2022 at 09:27:40PM +0000, Artem S. Tashkinov wrote:
> > Which is why in the general case, you really should consider email to
> > be the "lingua franca" of kernel development communication.  It
> > doesn't have the fundamental limitations and management issues that
> > bugzilla has. If you want to add more people to the Cc in an email,
> > you just do it.
> 
> Attention, Linus, the problem is attention.
> 
> Once something is filed in bugzilla, it's public, it's easily
> accessible, it can easily be found, you can easily add new info.
> 
> Emails? You've flown to Japan to a conference for a week and you have
> much better things than to check any email updates. A week worth of
> emails have suddenly become worthless.

Serious ? Have you ever attended a conference and looked over the
shoulder of the person in front of you ? There are 3 types of interfaces
you see:
  - code
  - slides
  - mails

The last thing people will look at during a conference definitely is a
painfully depressive bugtracker interface. However they will see bug
reports in their mailbox as they happen to read emails from their boss
or customers.

> Here's yet another issue, how would you send a follow up if you don't
> know the reference ("References" email field)? Instead of a follow up
> it'll end up being a new unrelated email.

You don't have such a problem with email. It only happens when you try
to respond via e-mail to stuff you find in a browser.

Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 21:32, Willy Tarreau wrote:
> On Sun, Oct 02, 2022 at 09:07:13PM +0000, Artem S. Tashkinov wrote:
>>>> Why are people are now blowing stuff out of proportion for no reason?
>>>
>>> Because the approach is wrong. As I explained it gives a false sense to
>>> the reporter that their issue is being handled while the simple fact that
>>> a message was sent to a person is in no way an engagement to do anything
>>> about it. LKML is a broadcast area. Everyone hopes someone else will
>>> respond and that eventually happens. When the reports are targetted, it
>>
>> No, it doesn't happen. Should I open LKML and send you a hundred of
>> unreplied emails over the past year alone?
>
> If that makes you feel better, feel free to do so. I'm not scared by
> only one hundred e-mails. What I'm impressed by, however, is that you're
> able to spot that many unreplied e-mails because I don't see as many. If
> you're that efficient at spotting them, maybe these are the ones you
> should just resend to make sure they're seen, and it would require less
> work (even on your side) than triaging issues.

So, we've been worked up about a _possible_ SPAM issue and your response
is ... create more SPAM? How does it solve all the other issues with
email I've identified?

>
>> Just before I GTFO I will leave this bug report here (already posted it
>> here but maybe I need to do it again and again):
>>
>> https://bugzilla.kernel.org/show_bug.cgi?id=204807
>>
>> Tell me honestly how ~255 comments, and a ton of collaboration over the
>> span of 2.5 years can be managed using email.
>
> What makes you think it would have taken that long over e-mail ? Between
> your first report and the first reply "this is not a bug", 18 months had
> elapsed already. The most active part of the discussion happened grouped
> on 3 days (2021-03-19 -> 22), where there were already some "I'm removing
> myself from the CC because the discussion isn't productive", then a large
> number of "me too" happened. Not sure how much useful this has been
> overall to the involved developers, given that it's impossible to stay
> focused on that long a thread and sum up all the information spanning
> over that many kernel versions and that many different hardware.

It's easy to join an existing bug report. Tell me how can I join an
existing email thread without being first subscribed to it? I certainly
can, absolute most people will not be able to.

What about sending large dump files? Should everyone on the mailing list
receive it?

A bug report is a simple plain list of messages in a single place which
could be read with a web browser. An email thread is anything but.

Searching through many emails at once? Good luck with that.

Replying to a particular email? Good luck with that.

It looks like you're under the impression that every Linux user who is
willing to ever use Linux must:

1) Subscribe to _all_ the existing mailing lists (just in case - what if
you need to work on something which was started by someone else)
2) Know the email etiquette
3) Learn to be persistent and resend (an unknown number of times) your
concerns hoping they will eventually be addressed.

Bugzilla: sign up once. Follow up. If you file a dupe, hopefully it will
be marked as a dupe. Everyone's happy. No particular skills, email
clients, formatting, referencing, etc. etc. etc.

All the developers busy and no one wants to work on your bug report?
That's Linux, you've got it for free. Submit a patch or pay someone
who'll fix your issue.

Regards,
Artem

>
> My gut feeling is that handling this over the ML would have resulted in:
>    - a few "sorry, no solution, try to fix your BIOS"
>    - "try this" => "it works, thank you".
>    - "this fix above broke for me"
>    - and a few such iterations until a satisfying enough solution would
>      have been found. Maybe not in 2.5 years, maybe 6 months.
>
> But I could be wrong. I'm not claiming I know how people feel the most
> efficient. Just observing what we're seeing on the lists and what I'm
> used to dealing with in some bug trackers. If you want I can as well
> show you a bug I reported 19 years ago that's still in state "NEW",
> having seen little updates over the years. It had better been closed
> since then, TBH:
>
>     https://gcc.gnu.org/bugzilla/show_bug.cgi?format=multiple&id=11873
>
> Pretty close to your demo above except it lasted 8 times longer and
> has not seen progress by lack of interest. How's that different from
> what you complain about mailing lists ? Hmm ?
>
> Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 21:40, Willy Tarreau wrote:
> On Sun, Oct 02, 2022 at 09:27:40PM +0000, Artem S. Tashkinov wrote:
>>> Which is why in the general case, you really should consider email to
>>> be the "lingua franca" of kernel development communication.  It
>>> doesn't have the fundamental limitations and management issues that
>>> bugzilla has. If you want to add more people to the Cc in an email,
>>> you just do it.
>>
>> Attention, Linus, the problem is attention.
>>
>> Once something is filed in bugzilla, it's public, it's easily
>> accessible, it can easily be found, you can easily add new info.
>>
>> Emails? You've flown to Japan to a conference for a week and you have
>> much better things than to check any email updates. A week worth of
>> emails have suddenly become worthless.
>
> Serious ? Have you ever attended a conference and looked over the
> shoulder of the person in front of you ? There are 3 types of interfaces
> you see:
>    - code
>    - slides
>    - mails
>
> The last thing people will look at during a conference definitely is a
> painfully depressive bugtracker interface. However they will see bug
> reports in their mailbox as they happen to read emails from their boss
> or customers.

I meant people who are at conferences or on vacation normally stop
working with their work related mailing lists. I vividly remember Linus
mailing something like this, "I've flown somewhere, I won't have
[stable] Internet, please postpone this and that". At least a couple of
times.

>
>> Here's yet another issue, how would you send a follow up if you don't
>> know the reference ("References" email field)? Instead of a follow up
>> it'll end up being a new unrelated email.
>
> You don't have such a problem with email. It only happens when you try
> to respond via e-mail to stuff you find in a browser.

That implies you've been subscribed to the mailing list earlier. Will
not work for the vast majority of people.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Sun, 2 Oct 2022 20:14:17 +0000
"Artem S. Tashkinov" <aros@gmx.com> wrote:
> > As one of the few kernel maintainers that actually likes bugzilla and I
> > do not mind being subscribed to it, I too find the above an awful idea
> > (and I agree with all those that explained why it is so).  
> 
> Good, exactly what I've been advocating for, those who like it and can
> help are welcome, others can go unsubscribe in under a minute. No drama.

No, why do they need to do something? It should be opt-in not opt-out.
What is the reason you want to burden maintainers? Your' "they can just
unsubscribe" is an insult to them.

> 
> >
> > This really comes down to a manpower issue, which is common among most
> > open source projects. Remember it is commonly said that the only
> > warrantee you get from open source projects is that if it breaks, you
> > get to keep the pieces.
> >
> > The issue is that the users of the Linux kernel mostly got it for free.
> > And if they did pay for it, it is highly unlikely that they paid the
> > kernel maintainer that owns the subsystem that they are having issues
> > with. That means, for the maintainers to triage these bug reports, they
> > are essentially doing it for free.  
> 
> I perfectly understand it. I've _not_ asked anyone to do anything yet,

Yes you are! You are making people unsubscribe. That _is_ telling
someone to do something that they do not want to do!

> except maybe have their email in the bugzilla database, so that people
> _could_ CC you.
> 
> They will _not_ do it right away. They first have to `git grep` commits,
> find the relevant developers and then CC them.
> 
> >
> > Some projects are better at this, and there are developers that are
> > happy to give free work, but there are also other projects that have
> > companies actively backing the work to debug these issues.
> >
> > If you are using fedora, go bug Red Hat, Ubuntu then Canonical. And
> > again, it comes down to if you have a paid subscription or not if you
> > are going to get anywhere with it.  
> 
> This does not work, period. Most kernel bug reports in Fedora and Ubuntu

That's because it's a free service.

> bug trackers linger for years, sometimes someone says, "Try the vanilla
> kernel and if it's still an issue, please use the kernel bugzilla".

Really? They are telling people to fill out the kernel bugzilla. Sounds
like someone needs to tell them otherwise. When I worked for Red Hat, I
don't recall anyone telling someone to fill out the kernel bugzilla
when they are told to report it upstream. It was always email the
maintainers involved.

> 
> My Fedora kernel bug reports have been dealt with exactly this way.
> 
> RedHat does solve kernel issues in the RHEL kernel if you have a paid
> subscription and you spend quite some time providing them with a perfect
> reproducible test case. This is far outside this conversation.

Why? This is exactly the type of workflow we want. We don't need to be
paid, but getting the perfect reproducer is something we ask a lot for.

> 
> >
> > Can this be annoying, sure. But that's how the open source ecosystem
> > works.
> >
> > If someone is not able to figure out how to use the mailing lists, it
> > is unlikely that they will be able to be useful in working with the
> > maintainer to solve their issue. As Ted mentioned, when asked to do
> > something to help analyze the issue, many times there's no response
> > from the reporter. Maybe because the reporter had no idea what the
> > maintainer wanted them to do. Most kernel bugs requires a constant back
> > and forth between the reporter and the developer. If you don't have
> > that, then there's no reason to bother with trying to fix the issue.  
> 
> Mailing lists more often than not do not work, and maybe worked in the
> early 90s.

Cc the maintainers along with the mailing list works much more than
just emailing the mailing list alone. And if you don't get anywhere
when you Cc the maintainer directly, what makes you think it will do
any better if it's part of bugzilla?


> 
> We don't need to resolve the issue right away. We don't have to deal
> with it. We just need a place where people could find existing issues
> and add their input. That's a lot better than chasing something in emails.
> 
> Here's the simplest example.
> 
> Person A installs kernel 6.0. They find a regression. They send an email
> to maling list X. Not necessarily the relevant one and the email is
> simply ignored.
> 
> Another person finds the same regression. This person B may not be aware
> of the mailing list used earlier. They send a bug report elsewhere.
> 
> Now we have two completely disconnected bug reports which if luck allows
> could be Googled. Oy, you must know what to google for. Not that many
> people have a good Google foo.

Isn't this what Stack Overflow is for? ;-)

> 
> Now with bugzilla.
> 
> Anyone opens the last seven days of bug reports and instantly sees that
> something similar has already been filed and dealt with. Collaboration
> ensues. Maybe just maybe some developer will join it and actually offer
> a fix. If not, OK, fine, no big deal but at least it's _known_,
> _visible_ and can be _found_.
> 
> Random unreplied emails God knows where? Good luck with that.

I don't know. I find a lot of bug issues that are fixed via searching
and getting lore links.

> 
> >
> > Ideally, someone (you?) would want to be a middle man and triage the
> > bugzilla reports and find those that look promising to get a fix
> > completed, and then be the liaison between bugzilla and the kernel
> > maintainer, then I think that could work. But the issue comes back to
> > manpower. Who's going to do that?  
> 
> I've already offered myself. The LF has no such position. And more
> importantly I'm from a totalitarian country, so I'm unlikely to be ever
> employed.

That's the issue I made in my first reply. That getting someone to do
this is the hardest part. Although, Slade seems to be volunteering.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Mon, 3 Oct 2022 00:06:58 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> > 
> > If you are using fedora, go bug Red Hat, Ubuntu then Canonical. And
> > again, it comes down to if you have a paid subscription or not if you
> > are going to get anywhere with it.
> > 
> > Can this be annoying, sure. But that's how the open source ecosystem
> > works.  
> 
> The dichotomy between the community/hobbyist/free side and the
> commercial/professional/paid side is an argument I often hear, and often
> make myself. It is not, however, ineluctable.

But is still the essence of a community. Things only get done when it
benefits those that are doing it. The point of open source is that
collaborating has a benefit for all involved. But dealing with bugs
that do affect you directly, usually goes to the bottom of your
priority list. Not that you are not willing to do so, but it may only
get done when you have time to do it. And we all know how much time
kernel developers have.

> 
> We have shown multiple times that this barrier doesn't have to exist.
> The kernel is an impressive example of how companies and communities can
> cooperate to reach a result that no single entity could have achieved.
> It started with the development model and how it scaled, and other areas
> were tackled along the way, such as automated testing and quality
> control in general for instance. Lots of efforts went into creating
> solutions that could fulfil the unique needs of our development model,
> and into convincing large and small companies to invest, either money or
> time.
> 
> Are we doing a great job ? Certainly not. But we are moving forward. As
> Jon Corbet said several years ago in one of his talks, now that the
> Linux kernel has reached a leading position in many areas, we have lost
> the comfort of following other industry actors and have to innovate
> ourselves. That often means (and this thought is mine, not Jon's)
> winging it along the way. As impressive as some of our achievements may
> be, our failures to maintain some areas of the kernel in a professional
> way is also astonishing (https://xkcd.com/2347/ comes to mind). It's not
> entirely surprising: the community and (part-time) volunteer model means
> that everybody will tackle problems that interest them. Building a
> community that can deliver professional support is not an interesting
> task for everybody. It is, however, a key factor in the difference
> between a kernel subsystem that strives and a subsystem that survives.

Exactly. The problem is that this is an issue (getting general bug
reporting from users), but it's not a fatal one if you don't get it
right. The community tends to do what it takes to survive. It's usually
not until there's a well established threat that everyone changes how
they do things. Major updates to the kernel at the end of an -rc
release is unheard of, until there's a meltdown/spectre threat.

> 
> I believe the same holds true for bug tracking and support. At the end
> of the day, someone will need to pay for it, but we could shatter the
> traditional model here too. We could, given enough interest, bridge the
> gap between all involved parties and create a support model that would
> benefit everybody. It took years and huge efforts for Linux to evolve
> towards more professionalism in many areas, and it would take more years
> and more effort to continue and expand that, but I believe it would be
> feasible.

Linus went away and came back with git. Should we ask him to go away
and come back with a better bugzilla? :-D

> 
> Linux didn't start because Linus complained about existing operating
> systems, ranted on usenet forums and waited for someone to fix the
> problem. Someone will need to step up and lead the effrot here too. If
> that person could ignore for a while that this is an impossible task, I
> think they could succeed.
> 
> > If someone is not able to figure out how to use the mailing lists, it
> > is unlikely that they will be able to be useful in working with the
> > maintainer to solve their issue. As Ted mentioned, when asked to do
> > something to help analyze the issue, many times there's no response
> > from the reporter. Maybe because the reporter had no idea what the
> > maintainer wanted them to do. Most kernel bugs requires a constant back
> > and forth between the reporter and the developer. If you don't have
> > that, then there's no reason to bother with trying to fix the issue.
> > 
> > Ideally, someone (you?) would want to be a middle man and triage the
> > bugzilla reports and find those that look promising to get a fix
> > completed, and then be the liaison between bugzilla and the kernel
> > maintainer, then I think that could work. But the issue comes back to
> > manpower. Who's going to do that?  
> 
> On the topic of triage, I've found that distro developers often do a
> pretty good job. I've received multiple bug reports of great quality
> following problems initially posted to distro bug trackers, after the
> distro developers took the time needed to hold reporters by the hand to
> get all the required information. Kudos for that !
> 

This is what I was saying about having a liaison. It could work if we
have someone to do it. We have one volunteer (Slade), perhaps this
could turn out to be something more.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 22:08, Steven Rostedt wrote:

Sometime mailing lists are worth CC'ing, sometimes individual developers
when you can easily find/bisect the bad commit.

Bugzilla hasn't been updated in a very long time so it's missing both
mailing lists and individual kernel developers.

AFAIK, some pieces of kernel have no appropriate mailing lists at all.
What about that? I've no clue.

Opt-in will work, except I've no idea how to make it work. Mass email
all the kernel developers and politely invite them to sign up? Most will
simply ignore it.

It's been mentioned here several times already that even collecting
their public email addresses in public git logs could be considered illegal.

There's too much resistance, too much "Mail will work for everyone"
except when it doesn't. I've got a strong feeling inving me to this
discussion was a bad mistake and no one even remotely likes my
proposals. Sorry, I give up.


Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Sun, 2 Oct 2022 22:20:40 +0000
"Artem S. Tashkinov" <aros@gmx.com> wrote:

> Opt-in will work, except I've no idea how to make it work. Mass email
> all the kernel developers and politely invite them to sign up? Most will
> simply ignore it.

Why not? The people who will ignore it will be the same people you are
asking to go and unsubscribe. But at least with just a single email
invite all they need to do to not be set up is to ignore a single
email. For those that want to be involved, they will make the effort to
do so.

> 
> It's been mentioned here several times already that even collecting
> their public email addresses in public git logs could be considered illegal.

I would do it with the MAINTAINERS file if you were to do that. The
purpose of the MAINTAINERS file is to send email to those listed.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

Hi Steven,

On Sun, Oct 02, 2022 at 06:18:01PM -0400, Steven Rostedt wrote:
> On Mon, 3 Oct 2022 00:06:58 +0300 Laurent Pinchart wrote:
> > > 
> > > If you are using fedora, go bug Red Hat, Ubuntu then Canonical. And
> > > again, it comes down to if you have a paid subscription or not if you
> > > are going to get anywhere with it.
> > > 
> > > Can this be annoying, sure. But that's how the open source ecosystem
> > > works.  
> > 
> > The dichotomy between the community/hobbyist/free side and the
> > commercial/professional/paid side is an argument I often hear, and often
> > make myself. It is not, however, ineluctable.
> 
> But is still the essence of a community. Things only get done when it
> benefits those that are doing it. The point of open source is that
> collaborating has a benefit for all involved. But dealing with bugs
> that do affect you directly, usually goes to the bottom of your
> priority list. Not that you are not willing to do so, but it may only
> get done when you have time to do it. And we all know how much time
> kernel developers have.

Absolutely, and that's why I think an efficient bug tracking *process*
(regardless of the tooling) would require new people to handle it
(obviously in collaboration with existing developers). I don't think we
can expect developers and maintainers to expand beyond what they're
already doing to new areas or to extend the amount of time they spend on
bug handling without expanding at the same time the scope of the
subsystems. The kernel started with technical maintainers, and over time
we have seen the rise of community builders in some subsystems. I
believe it would be possible to expand communities to cover a more
professional style of bug tracking and handling, but not by just
stretching the resources we have.

> > We have shown multiple times that this barrier doesn't have to exist.
> > The kernel is an impressive example of how companies and communities can
> > cooperate to reach a result that no single entity could have achieved.
> > It started with the development model and how it scaled, and other areas
> > were tackled along the way, such as automated testing and quality
> > control in general for instance. Lots of efforts went into creating
> > solutions that could fulfil the unique needs of our development model,
> > and into convincing large and small companies to invest, either money or
> > time.
> > 
> > Are we doing a great job ? Certainly not. But we are moving forward. As
> > Jon Corbet said several years ago in one of his talks, now that the
> > Linux kernel has reached a leading position in many areas, we have lost
> > the comfort of following other industry actors and have to innovate
> > ourselves. That often means (and this thought is mine, not Jon's)
> > winging it along the way. As impressive as some of our achievements may
> > be, our failures to maintain some areas of the kernel in a professional
> > way is also astonishing (https://xkcd.com/2347/ comes to mind). It's not
> > entirely surprising: the community and (part-time) volunteer model means
> > that everybody will tackle problems that interest them. Building a
> > community that can deliver professional support is not an interesting
> > task for everybody. It is, however, a key factor in the difference
> > between a kernel subsystem that strives and a subsystem that survives.
> 
> Exactly. The problem is that this is an issue (getting general bug
> reporting from users), but it's not a fatal one if you don't get it
> right. The community tends to do what it takes to survive. It's usually
> not until there's a well established threat that everyone changes how
> they do things. Major updates to the kernel at the end of an -rc
> release is unheard of, until there's a meltdown/spectre threat.
> 
> > I believe the same holds true for bug tracking and support. At the end
> > of the day, someone will need to pay for it, but we could shatter the
> > traditional model here too. We could, given enough interest, bridge the
> > gap between all involved parties and create a support model that would
> > benefit everybody. It took years and huge efforts for Linux to evolve
> > towards more professionalism in many areas, and it would take more years
> > and more effort to continue and expand that, but I believe it would be
> > feasible.
> 
> Linus went away and came back with git. Should we ask him to go away
> and come back with a better bugzilla? :-D

I'm sure Linus could give it a try if he was interested, but that's the
wrong approach. They key to a successful free software community is to
be prepared to do all the work yourself, not expecting that throwing
ideas around will result in someone jumping in to fix all your problems.
It may happen, but relying on it will most often lead to disappointment.
The same thing applies here. I believe something could be done to
improve bug tracking very significantly, but I have no hope that sharing
my ideas will magically result in any improvement (and when it comes to
volunteering to fix the problem myself, I can't, I'm drowning under
patch review and other tasks, in a perfect example of how things won't
scale when the amount of work prevents people from making the necessary
improvements that are key to lowering the workload).

> > Linux didn't start because Linus complained about existing operating
> > systems, ranted on usenet forums and waited for someone to fix the
> > problem. Someone will need to step up and lead the effrot here too. If
> > that person could ignore for a while that this is an impossible task, I
> > think they could succeed.
> > 
> > > If someone is not able to figure out how to use the mailing lists, it
> > > is unlikely that they will be able to be useful in working with the
> > > maintainer to solve their issue. As Ted mentioned, when asked to do
> > > something to help analyze the issue, many times there's no response
> > > from the reporter. Maybe because the reporter had no idea what the
> > > maintainer wanted them to do. Most kernel bugs requires a constant back
> > > and forth between the reporter and the developer. If you don't have
> > > that, then there's no reason to bother with trying to fix the issue.
> > > 
> > > Ideally, someone (you?) would want to be a middle man and triage the
> > > bugzilla reports and find those that look promising to get a fix
> > > completed, and then be the liaison between bugzilla and the kernel
> > > maintainer, then I think that could work. But the issue comes back to
> > > manpower. Who's going to do that?  
> > 
> > On the topic of triage, I've found that distro developers often do a
> > pretty good job. I've received multiple bug reports of great quality
> > following problems initially posted to distro bug trackers, after the
> > distro developers took the time needed to hold reporters by the hand to
> > get all the required information. Kudos for that !
> 
> This is what I was saying about having a liaison. It could work if we
> have someone to do it. We have one volunteer (Slade), perhaps this
> could turn out to be something more.

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Al Viro]

On Sun, Oct 02, 2022 at 10:20:40PM +0000, Artem S. Tashkinov wrote:

> Bugzilla hasn't been updated in a very long time so it's missing both
> mailing lists and individual kernel developers.
> 
> AFAIK, some pieces of kernel have no appropriate mailing lists at all.
> What about that? I've no clue.

There's that file, right in the root of the source tree.  Called "MAINTAINERS",
in all-caps...  Could have something to do with locating maintainers, could it not?

> Opt-in will work, except I've no idea how to make it work. Mass email
> all the kernel developers and politely invite them to sign up? Most will
> simply ignore it.

Sigh...   You really don't seem to appreciate just how deep a septic
tank you've jumped into with your combination of "it should be opt-out"
and "but unsubscribing takes just a minute, what are you unhappy about?!?"

Maybe you are not using email a lot, but for just about everyone who does...
We have heard that.  Many, many times.  From many sources - spammers,
"legitimate" companies' marketing departments, etc.

And you keep moving along the same track - the usual reaction of some
company after having pulled back a bloody stump and enjoyed the pile of
explanations of the reasons why opt-out is *NOT* *ACCEPTABLE*, *EVER*
is along the lines of "OK, we'll just spam everyone in our database once
and ask them to opt-in - that must be OK, right?"

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Mon, 3 Oct 2022 00:04:25 +0100
Al Viro <viro@zeniv.linux.org.uk> wrote:

> And you keep moving along the same track - the usual reaction of some
> company after having pulled back a bloody stump and enjoyed the pile of
> explanations of the reasons why opt-out is *NOT* *ACCEPTABLE*, *EVER*
> is along the lines of "OK, we'll just spam everyone in our database once
> and ask them to opt-in - that must be OK, right?"

But a single spam that can be ignored is so much better than being
automatically added to something that you have to do work to get out of.

We get spammed a lot to participate in surveys and such. But for those one time
emails, they never bothered me. Sometimes, I actually do participate.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hi,

> On Oct 2, 2022, at 6:18 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> 
>> 
>> I believe the same holds true for bug tracking and support. At the end
>> of the day, someone will need to pay for it, but we could shatter the
>> traditional model here too. We could, given enough interest, bridge the
>> gap between all involved parties and create a support model that would
>> benefit everybody. It took years and huge efforts for Linux to evolve
>> towards more professionalism in many areas, and it would take more years
>> and more effort to continue and expand that, but I believe it would be
>> feasible.
> 
> Linus went away and came back with git. Should we ask him to go away
> and come back with a better bugzilla? :-D

I mean, we could try and ask nicely, but I’d understand if the answer was no, haha.

> 
>> 
>> On the topic of triage, I've found that distro developers often do a
>> pretty good job. I've received multiple bug reports of great quality
>> following problems initially posted to distro bug trackers, after the
>> distro developers took the time needed to hold reporters by the hand to
>> get all the required information. Kudos for that !
>> 
> 
> This is what I was saying about having a liaison. It could work if we
> have someone to do it. We have one volunteer (Slade), perhaps this
> could turn out to be something more.

In all seriousness: I really believe having a liaison who’s — and I’m simplifying these items for right this second — 1) gathering information about a report from the reporter (by holding their hand through the process), 2) weeding out issues that aren’t actionable or reproducible, and 3) communicating all information to developers as clear and concise of a manner of possible will (hopefully) result in a better “bugzilla”. 

Sure, it won’t happen overnight, but I truly believe it has a chance (and as you mentioned, I’ve already volunteered to take this on and (hopefully) get something off the ground.)

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Sun, Oct 2, 2022 at 11:54 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> It's easy to join an existing bug report. Tell me how can I join an
> existing email thread without being first subscribed to it? I certainly
> can, absolute most people will not be able to.

lore.kernel.org

> What about sending large dump files? Should everyone on the mailing list
> receive it?

post a link

> A bug report is a simple plain list of messages in a single place which
> could be read with a web browser. An email thread is anything but.

lore.kernel.org

> Searching through many emails at once? Good luck with that.

lore.kernel.org

> Replying to a particular email? Good luck with that.

lore.kernel.org

> It looks like you're under the impression that every Linux user who is
> willing to ever use Linux must:
>
> 1) Subscribe to _all_ the existing mailing lists (just in case - what if
> you need to work on something which was started by someone else)

lore.kernel.org

> 2) Know the email etiquette

Just Be Polite

> 3) Learn to be persistent and resend (an unknown number of times) your
> concerns hoping they will eventually be addressed.
>
> Bugzilla: sign up once. Follow up. If you file a dupe, hopefully it will
> be marked as a dupe. Everyone's happy. No particular skills, email
> clients, formatting, referencing, etc. etc. etc.

Having at last the skill to provide a good rebug port would be nice...

Now, back to work. The merge window has opened, so there will be
bugs to report and/or fix...

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Christoph Hellwig]

On Fri, Sep 30, 2022 at 09:18:13PM -0400, Theodore Ts'o wrote:
> Funny thing.  I've largely given up on getting any kind of useful bug
> report from Launchpad, so I've largely ignored it.  In contast, the
> bug reports I get for e2fsprogs from Debian are generally far more
> actionable, with bug reports that have all of the data so I can
> actually root cause the problem, and help the user.

As someone who uses the Debian BTS as a bug reported only these days,
and a as maintainer (but not DD) in the past I love it.  For the
developer the email interface is perfect for quick actions, while
the website allows for a good overview.  And as a user the reportbug
tools works really nicely and tends to collect the important information
automatically.  Most importantly it does not require creating a user
account with some random entity.

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/2/22 23:04, Al Viro wrote:
> On Sun, Oct 02, 2022 at 10:20:40PM +0000, Artem S. Tashkinov wrote:
>
>> Bugzilla hasn't been updated in a very long time so it's missing both
>> mailing lists and individual kernel developers.
>>
>> AFAIK, some pieces of kernel have no appropriate mailing lists at all.
>> What about that? I've no clue.
>
> There's that file, right in the root of the source tree.  Called "MAINTAINERS",
> in all-caps...  Could have something to do with locating maintainers, could it not?
>
>> Opt-in will work, except I've no idea how to make it work. Mass email
>> all the kernel developers and politely invite them to sign up? Most will
>> simply ignore it.
>
> Sigh...   You really don't seem to appreciate just how deep a septic
> tank you've jumped into with your combination of "it should be opt-out"
> and "but unsubscribing takes just a minute, what are you unhappy about?!?"
>
> Maybe you are not using email a lot, but for just about everyone who does...
> We have heard that.  Many, many times.  From many sources - spammers,
> "legitimate" companies' marketing departments, etc.
>
> And you keep moving along the same track - the usual reaction of some
> company after having pulled back a bloody stump and enjoyed the pile of
> explanations of the reasons why opt-out is *NOT* *ACCEPTABLE*, *EVER*
> is along the lines of "OK, we'll just spam everyone in our database once
> and ask them to opt-in - that must be OK, right?"

Being on bugzilla does _not_ mean you'll receive a single email unless
someone _specifically_ CC's you.

(Except for relevant mailing lists and already specified maintainers who
are confirmed to manage certain kernel subsystems).

We've had over40 messages in this conversation and not a single person
has complained about SPAM ever coming from bugzilla.

With what I proposed that would _not_ change.

Weird to read this torrent of hatred and aggression towards purely
imaginary SPAM.

Anyways, Bugzilla is bad but it surely works. Let's have 100+ more
interchanges inventing something most users (for whom Bugzilla exists -
which people here keep forgetting all the time) will a have hard time
working with.

I repeat: Bugzilla exists for end users primarily.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/3/22 06:37, Geert Uytterhoeven wrote:
> Hi Artem,
>
> On Sun, Oct 2, 2022 at 11:54 PM Artem S. Tashkinov <aros@gmx.com> wrote:
>> It's easy to join an existing bug report. Tell me how can I join an
>> existing email thread without being first subscribed to it? I certainly
>> can, absolute most people will not be able to.
>
> lore.kernel.org
>
>> What about sending large dump files? Should everyone on the mailing list
>> receive it?
>
> post a link
>
>> A bug report is a simple plain list of messages in a single place which
>> could be read with a web browser. An email thread is anything but.
>
> lore.kernel.org
>
>> Searching through many emails at once? Good luck with that.
>
> lore.kernel.org
>
>> Replying to a particular email? Good luck with that.
>
> lore.kernel.org
>
>> It looks like you're under the impression that every Linux user who is
>> willing to ever use Linux must:
>>
>> 1) Subscribe to _all_ the existing mailing lists (just in case - what if
>> you need to work on something which was started by someone else)
>
> lore.kernel.org
>
>> 2) Know the email etiquette
>
> Just Be Polite
>
>> 3) Learn to be persistent and resend (an unknown number of times) your
>> concerns hoping they will eventually be addressed.
>>
>> Bugzilla: sign up once. Follow up. If you file a dupe, hopefully it will
>> be marked as a dupe. Everyone's happy. No particular skills, email
>> clients, formatting, referencing, etc. etc. etc.
>
> Having at last the skill to provide a good rebug port would be nice...
>
> Now, back to work. The merge window has opened, so there will be
> bugs to report and/or fix...

Lore looks alien to me and in my life I've worked with a dozen bug trackers.

* Where are open "issues"?
* Which issues are now resolved?
* What's the status of the "issue"?
* Which kernel subsystem is responsible for this or that "bug report"?
* How to change the assignee? How to know the new assignee has been
notified?

This thing looks interesting to discuss patches and merge requests
between developers who know each other and even at that it's not exactly
super intuitive. Again, you're not thinking about users who have no idea
how the kernel is developed.

If you remove bugzilla I'll never use lore.kernel.org, I promise. I'm
frightened by it.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Mike Rapoport]

On Mon, Oct 03, 2022 at 07:41:08AM +0000, Artem S. Tashkinov wrote:
> 
> 
> On 10/2/22 23:04, Al Viro wrote:
> > On Sun, Oct 02, 2022 at 10:20:40PM +0000, Artem S. Tashkinov wrote:
> > 
> > > Bugzilla hasn't been updated in a very long time so it's missing both
> > > mailing lists and individual kernel developers.
> > > 
> > > AFAIK, some pieces of kernel have no appropriate mailing lists at all.
> > > What about that? I've no clue.
> > 
> > There's that file, right in the root of the source tree.  Called "MAINTAINERS",
> > in all-caps...  Could have something to do with locating maintainers, could it not?
> > 
> > > Opt-in will work, except I've no idea how to make it work. Mass email
> > > all the kernel developers and politely invite them to sign up? Most will
> > > simply ignore it.
> > 
> > Sigh...   You really don't seem to appreciate just how deep a septic
> > tank you've jumped into with your combination of "it should be opt-out"
> > and "but unsubscribing takes just a minute, what are you unhappy about?!?"
> > 
> > Maybe you are not using email a lot, but for just about everyone who does...
> > We have heard that.  Many, many times.  From many sources - spammers,
> > "legitimate" companies' marketing departments, etc.
> > 
> > And you keep moving along the same track - the usual reaction of some
> > company after having pulled back a bloody stump and enjoyed the pile of
> > explanations of the reasons why opt-out is *NOT* *ACCEPTABLE*, *EVER*
> > is along the lines of "OK, we'll just spam everyone in our database once
> > and ask them to opt-in - that must be OK, right?"
> 
> Being on bugzilla does _not_ mean you'll receive a single email unless
> someone _specifically_ CC's you.

If I'm not mistaken, bugzilla lets CC people explicitly. How the database
of emails in bugzilla would help choosing the right people to CC better
than MAINTAINERS?

You repeated multiple times that bug reports sent to the mailing lists are
ignored, but what will make emails from bugzilla different from those bug
reports? Why do you think they will get more attention? 
 
> Anyways, Bugzilla is bad but it surely works. Let's have 100+ more
> interchanges inventing something most users (for whom Bugzilla exists -
> which people here keep forgetting all the time) will a have hard time
> working with.

You keep repeating that bugzilla is better then email, but the major point
here is not the tools, but the lack of resources to deal with initial
triage of the bugs and holding users' hand to get a meaningful report.
Until that changes, there is no point in trying to add more people CC'ed on
bugzilla reports. They won't be handled unless somebody would want to take
care of them and forcing people to receive these reports won't make anybody
more willing to help.
 
> Regards,
> Artem
> 

-- 
Sincerely yours,
Mike.

RE: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[David Laight]

...
> You repeated multiple times that bug reports sent to the mailing lists are
> ignored, but what will make emails from bugzilla different from those bug
> reports? Why do you think they will get more attention?

Most bug tracking systems I met end up being 'mostly write only'.
Sometimes management try to track the number of open bugs.
The 'solution' to that is either to get all your friends to
open low priority bugs and then ignore them (the bugs not
your friends), or to close the bugs just before the weekly
summary is done - knowing they'll get reopened later.

Oh, and bug update emails form bugzilla are entirely useless.
Unless you have time to open the link.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/3/22 08:55, Mike Rapoport wrote:
> On Mon, Oct 03, 2022 at 07:41:08AM +0000, Artem S. Tashkinov wrote:
>>
>>
>> On 10/2/22 23:04, Al Viro wrote:
>>> On Sun, Oct 02, 2022 at 10:20:40PM +0000, Artem S. Tashkinov wrote:
>>>
>>>> Bugzilla hasn't been updated in a very long time so it's missing both
>>>> mailing lists and individual kernel developers.
>>>>
>>>> AFAIK, some pieces of kernel have no appropriate mailing lists at all.
>>>> What about that? I've no clue.
>>>
>>> There's that file, right in the root of the source tree.  Called "MAINTAINERS",
>>> in all-caps...  Could have something to do with locating maintainers, could it not?
>>>
>>>> Opt-in will work, except I've no idea how to make it work. Mass email
>>>> all the kernel developers and politely invite them to sign up? Most will
>>>> simply ignore it.
>>>
>>> Sigh...   You really don't seem to appreciate just how deep a septic
>>> tank you've jumped into with your combination of "it should be opt-out"
>>> and "but unsubscribing takes just a minute, what are you unhappy about?!?"
>>>
>>> Maybe you are not using email a lot, but for just about everyone who does...
>>> We have heard that.  Many, many times.  From many sources - spammers,
>>> "legitimate" companies' marketing departments, etc.
>>>
>>> And you keep moving along the same track - the usual reaction of some
>>> company after having pulled back a bloody stump and enjoyed the pile of
>>> explanations of the reasons why opt-out is *NOT* *ACCEPTABLE*, *EVER*
>>> is along the lines of "OK, we'll just spam everyone in our database once
>>> and ask them to opt-in - that must be OK, right?"
>>
>> Being on bugzilla does _not_ mean you'll receive a single email unless
>> someone _specifically_ CC's you.
>
> If I'm not mistaken, bugzilla lets CC people explicitly. How the database
> of emails in bugzilla would help choosing the right people to CC better
> than MAINTAINERS?
>
> You repeated multiple times that bug reports sent to the mailing lists are
> ignored, but what will make emails from bugzilla different from those bug
> reports? Why do you think they will get more attention?

Maybe because they are specific? Maybe because they are not part of a
high volume mailing list such as LKML? Maybe because lots of developers
are _not_ on any mailing lists?

>
>> Anyways, Bugzilla is bad but it surely works. Let's have 100+ more
>> interchanges inventing something most users (for whom Bugzilla exists -
>> which people here keep forgetting all the time) will a have hard time
>> working with.
>
> You keep repeating that bugzilla is better then email, but the major point
> here is not the tools, but the lack of resources to deal with initial
> triage of the bugs and holding users' hand to get a meaningful report.
> Until that changes, there is no point in trying to add more people CC'ed on
> bugzilla reports. They won't be handled unless somebody would want to take
> care of them and forcing people to receive these reports won't make anybody
> more willing to help.

The initial conversation started with the fact that Bugzilla is old,
semi-deprecated, requires MySQL [no idea what's bad about it, Bugzilla
can work with MariaDB and Percona as well] and its components along with
the respective emails are extremely outdated. If I remember correctly
triaging bugs was raised much later in the discussion and is orthogonal
to the topic.

Triaging bugs could be and should be done by the people who are willing
to help [for free]. There's no problem with bugs filed under "Other" if
the reporter has no idea where to file them as long as they are visible
and searchable.

Imagine instead you send your issue to a random mailing list. What is
the chance another person with a similar issue will even find it?
Vanishingly low. The net result? Work and time wasted and no one is aware.

Again the volume of bug reports is relatively low, fewer than two dozen
a week.

Everything about Bugzilla so far has been completely blown out of
proportions:

* The insane number of emails it ostensibly sends: "OMG so much SPAM,
save me from it!"
* The privacy "issue" despite git commits and respective email addresses
being public
* The amount of work required to keep its components and email addresses
up to date - could be done maybe every 12-24 months
* The triaging "issue" which is outside the scope of this conversation

At the same time:

* Multiple reporters can perfectly find the people who have made bad
commits or who are responsible for certain drivers - it's safer to CC
them _via_ Bugzilla than to email them _privately_ or via mailing lists
which entails multiple issues including trust, SPAM, formatting,
English, net etiquette, etc. etc. etc.

You don't like Bugzilla? Fine, never touch it, never visit the website.
Never get emails from it.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Mon, Oct 3, 2022 at 11:16 AM Artem S. Tashkinov <aros@gmx.com> wrote:
> On 10/3/22 08:55, Mike Rapoport wrote:
> > On Mon, Oct 03, 2022 at 07:41:08AM +0000, Artem S. Tashkinov wrote:
> >> On 10/2/22 23:04, Al Viro wrote:
> >>> On Sun, Oct 02, 2022 at 10:20:40PM +0000, Artem S. Tashkinov wrote:
> >>>> Bugzilla hasn't been updated in a very long time so it's missing both
> >>>> mailing lists and individual kernel developers.
> >>>>
> >>>> AFAIK, some pieces of kernel have no appropriate mailing lists at all.
> >>>> What about that? I've no clue.
> >>>
> >>> There's that file, right in the root of the source tree.  Called "MAINTAINERS",
> >>> in all-caps...  Could have something to do with locating maintainers, could it not?
> >>>
> >>>> Opt-in will work, except I've no idea how to make it work. Mass email
> >>>> all the kernel developers and politely invite them to sign up? Most will
> >>>> simply ignore it.
> >>>
> >>> Sigh...   You really don't seem to appreciate just how deep a septic
> >>> tank you've jumped into with your combination of "it should be opt-out"
> >>> and "but unsubscribing takes just a minute, what are you unhappy about?!?"
> >>>
> >>> Maybe you are not using email a lot, but for just about everyone who does...
> >>> We have heard that.  Many, many times.  From many sources - spammers,
> >>> "legitimate" companies' marketing departments, etc.
> >>>
> >>> And you keep moving along the same track - the usual reaction of some
> >>> company after having pulled back a bloody stump and enjoyed the pile of
> >>> explanations of the reasons why opt-out is *NOT* *ACCEPTABLE*, *EVER*
> >>> is along the lines of "OK, we'll just spam everyone in our database once
> >>> and ask them to opt-in - that must be OK, right?"
> >>
> >> Being on bugzilla does _not_ mean you'll receive a single email unless
> >> someone _specifically_ CC's you.
> >
> > If I'm not mistaken, bugzilla lets CC people explicitly. How the database
> > of emails in bugzilla would help choosing the right people to CC better
> > than MAINTAINERS?
> >
> > You repeated multiple times that bug reports sent to the mailing lists are
> > ignored, but what will make emails from bugzilla different from those bug
> > reports? Why do you think they will get more attention?
>
> Maybe because they are specific? Maybe because they are not part of a
> high volume mailing list such as LKML? Maybe because lots of developers
> are _not_ on any mailing lists?

If they're sent only to the maintainers, not to the subsystem mailing
lists, they may be ignored, as no one but the maintainer will be aware.

> Imagine instead you send your issue to a random mailing list. What is
> the chance another person with a similar issue will even find it?

Do not underestimate the power of search engines.

> Again the volume of bug reports is relatively low, fewer than two dozen
> a week.

Which proves this tool is insignificant in the grant scheme of (Linux)
things.

> * Multiple reporters can perfectly find the people who have made bad
> commits or who are responsible for certain drivers - it's safer to CC
> them _via_ Bugzilla than to email them _privately_ or via mailing lists
> which entails multiple issues including trust, SPAM, formatting,
> English, net etiquette, etc. etc. etc.

Never send bug reports privately, unless you have a monetary
relationship with the receiving end.  Always Cc the subsystem
mailing list, so anyone involved can help.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/3/22 09:26, Geert Uytterhoeven wrote:
>
>> Imagine instead you send your issue to a random mailing list. What is
>> the chance another person with a similar issue will even find it?
>
> Do not underestimate the power of search engines.

I don't. In many situations the same issue can be described completely
differently and finding duplicates becomes near impossible. You
overestimate the power of search engines and the person's ability to use
very specific wording.

>> Again the volume of bug reports is relatively low, fewer than two dozen
>> a week.
>
> Which proves this tool is insignificant in the grant scheme of (Linux)
> things.


https://bugzilla.kernel.org/show_bug.cgi?id=204807

A very insignificant exchange of over 2 hundred comments, patches,
suggestions, etc. by the people absolute most of whom would have failed
to do that via email.

Nothing was lost, no messages were accidentally sent to SPAM, all the
people in the conversation _retained_ their privacy as Bugzilla _hides_
emails.

Hasn't privacy been raised as the cornerstone of this discussion several
times already? You're _far more private_ on Bugzilla.


>> * Multiple reporters can perfectly find the people who have made bad
>> commits or who are responsible for certain drivers - it's safer to CC
>> them _via_ Bugzilla than to email them _privately_ or via mailing lists
>> which entails multiple issues including trust, SPAM, formatting,
>> English, net etiquette, etc. etc. etc.
>
> Never send bug reports privately, unless you have a monetary
> relationship with the receiving end.  Always Cc the subsystem
> mailing list, so anyone involved can help.


I've done that on multiple occasions and in _many_ cases actually
received help vs. sending to a mailing list where my messages were
completely neglected.

For instance, I've CC'ed Linus Torvalds _privately_ from Bugzilla twice
and he _chimed_ in and _helped_ resolve the bugs. My messages to LKML
were _ignored_ by +1000 people subscribed to it.

You continue to talk as if random messages to mailing lists are
_actively_ monitored by developers. That's _not_ the case.

Absolute most developers actively monitor only messages from the people
they constantly work with. That's it.

Maybe I should start the list of "Why email sucks in terms of bug
reporting" because I keep saying the same stuff over and over again.



Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Willy Tarreau]

On Mon, Oct 03, 2022 at 07:49:38AM +0000, Artem S. Tashkinov wrote:
> Lore looks alien to me and in my life I've worked with a dozen bug trackers.
> 
> * Where are open "issues"?
> * Which issues are now resolved?
> * What's the status of the "issue"?

Very often this status is an issue by itself. It only serves to hide
the less interesting junk to discover what's left behind, but when
people continue to respond to resolved issues nobody sees that. And
if you block them from continuing to feed a closed issue, it's even
worse as it forces them to reopen a new one that causes duplication.

With e-mails you can respond to an old message if you want and keep
all the participants. And it does happen.

> If you remove bugzilla I'll never use lore.kernel.org, I promise.

It's a convincing argument, for sure!

> > > Again the volume of bug reports is relatively low, fewer than two dozen
> > > a week.
> > 
> > Which proves this tool is insignificant in the grant scheme of (Linux)
> > things.
> 
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=204807
> 
> A very insignificant exchange of over 2 hundred comments, patches,
> suggestions, etc. by the people absolute most of whom would have failed
> to do that via email.

You've already claimed that and we've also told you that it could very
well have been the opposite and have been resolved in 6 months instead
of 2.5 years, by having many more eyes on it. The thing is that you're
only listening to yourself and not to the arguments that the people you
accuse of not reading the lists gave you. At this point there's not much
more that can be done, you've completely closed that discussion by
refusing to listen, thus I think it's better to stop now.

Willy

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 29.09.22 13:33, Thorsten Leemhuis wrote:
> On 29.09.22 13:19, Thorsten Leemhuis wrote:
>>
>> TLDR: Core Linux kernel developers are unhappy with the state of
>> bugzilla.kernel.org; to improve things I plan to change a few important
>> aspects of its configuration, unless somebody comes up with better ideas
>> to tackle current problems: (1) Create a catch-all product making it
>> totally obvious to submitters that likely nobody will look into the
>> ticket. (2) Remove or hide all products & components where the subsystem
>> didn't fully commit to look into newly submitted reports. (3) Change the
>> text on the front page to make it clear that most kernel bug reports
>> need to be sent by mail.

Well, there are lots of interesting things discussed in this thread for
a time when a bugzilla replacement needs to be found. But one thing
afaics is pretty clear for the time being: we as the kernel development
community are not going to double down on bugzilla now.

Thing is: bugzilla.kernel.org is there and will be for a while, as it
provides services that some developers rely on. And it has some
problems, as widely known and outlined in my mail. Reducing those for
now by performing a few small changes (aka applying some band-aids here
and there) as outlined above IMHO is worth it to reduce the pain. There
was no opposition to that plan from Konstantin or core Linux kernel
developers afaics (please correct me if I'm wrong), so I'll likely start
working on realizing it later this week, unless I get "no, please
don't/please wait" from those people.

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Mike Rapoport]

On Mon, Oct 03, 2022 at 09:40:43AM +0000, Artem S. Tashkinov wrote:
> On 10/3/22 09:26, Geert Uytterhoeven wrote:
> 
> Nothing was lost, no messages were accidentally sent to SPAM, all the
> people in the conversation _retained_ their privacy as Bugzilla _hides_
> emails.
> 
> Hasn't privacy been raised as the cornerstone of this discussion several
> times already? You're _far more private_ on Bugzilla.

The privacy and SPAM volumes are not the cornerstones, it's the opt-out
thingy that behaves like a spam, feels like a spam and so it's treated like
a spam. 
 
> > 
> > Never send bug reports privately, unless you have a monetary
> > relationship with the receiving end.  Always Cc the subsystem
> > mailing list, so anyone involved can help.
> 
> I've done that on multiple occasions and in _many_ cases actually
> received help vs. sending to a mailing list where my messages were
> completely neglected.
> 
> For instance, I've CC'ed Linus Torvalds _privately_ from Bugzilla twice
> and he _chimed_ in and _helped_ resolve the bugs. My messages to LKML
> were _ignored_ by +1000 people subscribed to it.

Did you try CC'ing developers *and* the relevant lists? 
 
> Maybe I should start the list of "Why email sucks in terms of bug
> reporting" because I keep saying the same stuff over and over again.
 
Maybe you also need to listen what other people reply... 

I can take the point that bugzilla (or another tracker) could be more
convenient to users. But that does not mean that kernel developers and
maintainers can be forced to use it.
 
> Regards,
> Artem

-- 
Sincerely yours,
Mike.

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

Hi Thorsten,

> On Oct 3, 2022, at 6:10 AM, Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
> Thing is: bugzilla.kernel.org is there and will be for a while, as it
> provides services that some developers rely on. And it has some
> problems, as widely known and outlined in my mail. Reducing those for
> now by performing a few small changes (aka applying some band-aids here
> and there) as outlined above IMHO is worth it to reduce the pain. There
> was no opposition to that plan from Konstantin or core Linux kernel
> developers afaics (please correct me if I'm wrong), so I'll likely start
> working on realizing it later this week, unless I get "no, please
> don't/please wait" from those people.

With the band-aids you outline in place: do you think it would it be beneficial to have a liaison holding users’s hands through the process, _then_ triaging to developers by contacting them with the information they need? (This is something proposed previously on this thread[1], and something I’ve already said I’m willing to do[2][3].)

IOW, someone to bridge between end users and developers and (at least try to) help bring some order to the chaos.

Thanks,
-srw

[1] https://lore.kernel.org/lkml/20221002141321.394de676@rorschach.local.home/
[2] https://lore.kernel.org/lkml/DA4EA2D8-0EA0-41FF-B495-68EC9E92CC30@sladewatkins.net/
[3] https://lore.kernel.org/lkml/B0853C40-C19A-46D0-B151-84C9016745ED@sladewatkins.net/

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Jani Nikula]

On Fri, 30 Sep 2022, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:
> E-mail *clients* are horrible to keep track of state. E-mail itself,
> as in RFC822 (and newer), SMTP and other protocols, only handle
> transport of data. As the data within the e-mail body is free-formed,
> and wasn't meant to track items and their state, clients never evolved
> in that direction.

Email is a massively distributed software fuzzing project that lets you
transmit messages in the sideband. :p

> Bugzilla won't solve this. The huge elephant in the room is that most
> maintainers are overworked. Whether a bug report arrives in my mailbox
> as an e-mail straight from the reporter or from a bug tracker will
> make very little difference if I don't have time to look into it (I
> would even argue that bug trackers are even worse there: if I'm really
> short of time, I'm more likely to prioritize replying to e-mails
> instead of having to open a link in a web browser).

I think a bug tracker helps in quantifying the problems you have,
though, including the maintainer bandwidth. Email doesn't easily lend
itself to that kind of analysis. I can't point managers at list emails
and ask for help. And if you do get people to help, having a centralized
place for the bug data helps them.

The flip side is that it's easier for me to ignore notification mails
from a bug tracker because I know the info isn't lost in a sea of other
mails.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Graphics Center

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 03.10.22 13:18, Slade Watkins wrote:
> 
>> On Oct 3, 2022, at 6:10 AM, Thorsten Leemhuis <linux@leemhuis.info>
>> wrote:
>> 
>> Thing is: bugzilla.kernel.org is there and will be for a while, as
>> it provides services that some developers rely on. And it has some 
>> problems, as widely known and outlined in my mail. Reducing those
>> for now by performing a few small changes (aka applying some
>> band-aids here and there) as outlined above IMHO is worth it to
>> reduce the pain. There was no opposition to that plan from
>> Konstantin or core Linux kernel developers afaics (please correct
>> me if I'm wrong), so I'll likely start working on realizing it
>> later this week, unless I get "no, please don't/please wait" from
>> those people.
> 
> With the band-aids you outline in place: do you think it would it be
> beneficial to have a liaison holding users’s hands through the
> process, _then_ triaging to developers by contacting them with the
> information they need?

Well, yes and no. :-/

Thing is: up to a point that's something I do already (and will likely
continue to do at least for a while) when the reported issue is a
regression. But to be fair, I often could help way more if I wanted to,
but there are only so many hours in a day and other things to take care
of (regression tracking is only a part-time thing for me currently). So
some help there might be handy; would get load of the developers as
well, as they often are more willing to help users when a report is
about a regression.

But for other issues (aka regular bugs) I don't think it's worth it,
because why only help those users that report to bugzilla (you didn't
say that, but it sounded to me like the focus is on it)? There are
people that try to use the mailing lists, but do it badly and never get
a reply (for example because they sent their report just to LKML). They
could need help, too; maybe helping them should even be priority, as
they at least tried to do what most kernel developers want them to do,
hence their reports might be better, too.

But there is a more important reason why I think having a liaison might
not be worth it for now: It IMHO would be much better to spend the time
and effort on other things that enable users submitting better bug
reports in the first place. I have no concrete and well-thought-out
ideas at hand what to do exactly, but here are a few vague ones:

 * create an app (ideally usable locally and on the web) that guides
users through generating a good bug report (let's leave the way of
submission aside for now). That app could handle quite a few of the
steps that https://docs.kernel.org/admin-guide/reporting-issues.html
currently mentions. It for example could check if the kernel looks to be
vanilla, if the kernel is fresh, if the kernel is tainted, if an Oops is
the first one or just a follow-up error; maybe that app could even
decode stack-traces locally in some environments; and it could collect
and upload logs as well. It could also explain certain things to users
when not fulfilled, for example why it's not worth to report a problem
that happens with an old kernel.

   Sure, these apps never work perfect and doing it right is a lot of
work, but I guess one could make things a lot easier for many users
especially for our case. I assume other projects have done something
like that so that we could learn from them.

 * Improve https://docs.kernel.org/admin-guide/reporting-issues.html
further. I have some ideas there, but other things are higher on my
priority list currently. That document in the end somehow needs to
become less scary looking while still providing all important details
for situations where a reporter might need them.

 * Write new docs relevant for bug reporting. We for example still have
no well written and simple to understand text that explains bisection to
people that are new to git, bisection, or compiling kernels in general.
Speaking of which: we iirc are also missing a text that properly
explains how to quickly configure and compile a kernel using "make
localmodconfig" (I mean something like
http://www.h-online.com/open/features/Good-and-quick-kernel-configuration-creation-1403046.html)

 * Not sure, maybe a list of things that known to be broken might be
good to have? Like "yes, we know that nouveau is slow, but we can't do
anything about this" or "driver 'wifi-foo' only supports a small subset
of the features the hardware offers, so don't report bugs if bar, baz
and foobar don't work".

* Once things improved with steps like the above try to form a "kernel
tester community" where people can help each other when they run into
problems or want to report an issue. We should try to get distributions
like Arch Linux, openSUSE Tumbleweed or Fedora on board here as well, as
they and their users have an interest in ensuring new mainline releases
work well, because they regularly rebase to the latest series.

  At that point it likely would be good to have someone that is at least
somewhat paid to act as "Community Manager"; that person then could also
act as liaison between users and developers and fine-tune things (docs
etc.) further when needed.

Those were just the things that came from the top of my head that IMHO
should be a priority.

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Mon, 3 Oct 2022 09:40:43 +0000
"Artem S. Tashkinov" <aros@gmx.com> wrote:

> For instance, I've CC'ed Linus Torvalds _privately_ from Bugzilla twice
> and he _chimed_ in and _helped_ resolve the bugs.

You didn't Cc Linus _privately_, because you Cc'd him from Bugzilla. I'm
guessing that means it's a public conversation. Which is similar to Cc'ing
a maintainer and a public mailing list.

> My messages to LKML
> were _ignored_ by +1000 people subscribed to it.

LKML gets 800 emails a day. Nobody reads it (besides Jon Corbet and Andrew
Morton). But if you send email to a maintainer privately without Cc'ing any
public mailing list (or Bugzilla), then it will likely be ignored.

What we are saying is, you need to do both. Cc the maintainer _and_ a
public mailing list. That way the maintainer knows others can see it, and
could point someone else to look at it if they do not have the time, or
they know someone who can better help.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Mon, Oct 03, 2022 at 09:16:06AM +0000, Artem S. Tashkinov wrote:
> The initial conversation started with the fact that Bugzilla is old,
> semi-deprecated, requires MySQL [no idea what's bad about it, Bugzilla
> can work with MariaDB and Percona as well]

It can't, actually. It only works with MySQL 5.7 or an equally ancient MariaDB.
No, there is no official fix (because nobody is working on bugzilla).
https://bugzilla.mozilla.org/show_bug.cgi?id=1592129

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Mon, 3 Oct 2022 14:59:54 +0200
Thorsten Leemhuis <linux@leemhuis.info> wrote:

> > With the band-aids you outline in place: do you think it would it be
> > beneficial to have a liaison holding users’s hands through the
> > process, _then_ triaging to developers by contacting them with the
> > information they need?  
> 
> Well, yes and no. :-/
> 
> Thing is: up to a point that's something I do already (and will likely
> continue to do at least for a while) when the reported issue is a
> regression. But to be fair, I often could help way more if I wanted to,
> but there are only so many hours in a day and other things to take care
> of (regression tracking is only a part-time thing for me currently). So
> some help there might be handy; would get load of the developers as
> well, as they often are more willing to help users when a report is
> about a regression.

Are you asking for help in the regression tracking?

> 
> But for other issues (aka regular bugs) I don't think it's worth it,
> because why only help those users that report to bugzilla (you didn't
> say that, but it sounded to me like the focus is on it)? There are
> people that try to use the mailing lists, but do it badly and never get
> a reply (for example because they sent their report just to LKML). They
> could need help, too; maybe helping them should even be priority, as
> they at least tried to do what most kernel developers want them to do,
> hence their reports might be better, too.

Could do both.

> 
> But there is a more important reason why I think having a liaison might
> not be worth it for now: It IMHO would be much better to spend the time
> and effort on other things that enable users submitting better bug
> reports in the first place. I have no concrete and well-thought-out
> ideas at hand what to do exactly, but here are a few vague ones:
> 
>  * create an app (ideally usable locally and on the web) that guides
> users through generating a good bug report (let's leave the way of
> submission aside for now). That app could handle quite a few of the
> steps that https://docs.kernel.org/admin-guide/reporting-issues.html
> currently mentions. It for example could check if the kernel looks to be
> vanilla, if the kernel is fresh, if the kernel is tainted, if an Oops is
> the first one or just a follow-up error; maybe that app could even
> decode stack-traces locally in some environments; and it could collect
> and upload logs as well. It could also explain certain things to users
> when not fulfilled, for example why it's not worth to report a problem
> that happens with an old kernel.

Christoph mentioned Debian's reportbug utility. That does a pretty good
job at walking people through how to report a bug. It could also get
information about the current environment that would be useful too. Perhaps
something like that?

> 
>    Sure, these apps never work perfect and doing it right is a lot of
> work, but I guess one could make things a lot easier for many users
> especially for our case. I assume other projects have done something
> like that so that we could learn from them.
> 
>  * Improve https://docs.kernel.org/admin-guide/reporting-issues.html
> further. I have some ideas there, but other things are higher on my
> priority list currently. That document in the end somehow needs to
> become less scary looking while still providing all important details
> for situations where a reporter might need them.
> 
>  * Write new docs relevant for bug reporting. We for example still have
> no well written and simple to understand text that explains bisection to
> people that are new to git, bisection, or compiling kernels in general.
> Speaking of which: we iirc are also missing a text that properly
> explains how to quickly configure and compile a kernel using "make
> localmodconfig" (I mean something like
> http://www.h-online.com/open/features/Good-and-quick-kernel-configuration-creation-1403046.html)

The sad part is that most people that are going to report a bug is not
going to read a full document to figure out how to do it. Usually when
someone hits a bug, they are doing something else. And it's a burden to
report it. Obviously, they want it to be fixed, but it's viewed as a favor
to the developer and not the user to get it fixed, as it's likely seen as a
mistake by the developer that the bug exists in the first place.

Having a tool like reportbug that walks you through the steps of reporting
it would be the best way to do so. As the reporter doesn't need to think
too hard and just answer questions and let the tool do all the work.

> 
>  * Not sure, maybe a list of things that known to be broken might be
> good to have? Like "yes, we know that nouveau is slow, but we can't do
> anything about this" or "driver 'wifi-foo' only supports a small subset
> of the features the hardware offers, so don't report bugs if bar, baz
> and foobar don't work".

The tool could possibly reply with known issues, and state something like
"We are aware of this issue, and are currently trying to figure it out."


-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Mon, Oct 03, 2022 at 11:55:25AM +0300, Mike Rapoport wrote:
> If I'm not mistaken, bugzilla lets CC people explicitly. How the database
> of emails in bugzilla would help choosing the right people to CC better
> than MAINTAINERS?

It can't, actually, which is I think is the crux of misunderstanding here. I
think what Artem is proposing is to *auto-create bugzilla accounts* for anyone
who shows up in MAINTAINERS, so that they can be cc'd on a bug report.
However, everyone understood this as "add these people as default assignees,"
which is not the case.

If we auto-create accounts for MAINTAINERS, that would allow them to be cc'd
by an actual human being triaging bugs, but won't lead to any discernable
increase of bugzilla mail.

Artem, please correct me if I'm wrong.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Mon, Oct 03, 2022 at 11:26:05AM -0400, Steven Rostedt wrote:
> On Mon, 3 Oct 2022 14:59:54 +0200 Thorsten Leemhuis wrote:
> 
> > > With the band-aids you outline in place: do you think it would it be
> > > beneficial to have a liaison holding users’s hands through the
> > > process, _then_ triaging to developers by contacting them with the
> > > information they need?  
> > 
> > Well, yes and no. :-/
> > 
> > Thing is: up to a point that's something I do already (and will likely
> > continue to do at least for a while) when the reported issue is a
> > regression. But to be fair, I often could help way more if I wanted to,
> > but there are only so many hours in a day and other things to take care
> > of (regression tracking is only a part-time thing for me currently). So
> > some help there might be handy; would get load of the developers as
> > well, as they often are more willing to help users when a report is
> > about a regression.
> 
> Are you asking for help in the regression tracking?
> 
> > But for other issues (aka regular bugs) I don't think it's worth it,
> > because why only help those users that report to bugzilla (you didn't
> > say that, but it sounded to me like the focus is on it)? There are
> > people that try to use the mailing lists, but do it badly and never get
> > a reply (for example because they sent their report just to LKML). They
> > could need help, too; maybe helping them should even be priority, as
> > they at least tried to do what most kernel developers want them to do,
> > hence their reports might be better, too.
> 
> Could do both.
> 
> > But there is a more important reason why I think having a liaison might
> > not be worth it for now: It IMHO would be much better to spend the time
> > and effort on other things that enable users submitting better bug
> > reports in the first place. I have no concrete and well-thought-out
> > ideas at hand what to do exactly, but here are a few vague ones:
> > 
> >  * create an app (ideally usable locally and on the web) that guides
> > users through generating a good bug report (let's leave the way of
> > submission aside for now). That app could handle quite a few of the
> > steps that https://docs.kernel.org/admin-guide/reporting-issues.html
> > currently mentions. It for example could check if the kernel looks to be
> > vanilla, if the kernel is fresh, if the kernel is tainted, if an Oops is
> > the first one or just a follow-up error; maybe that app could even
> > decode stack-traces locally in some environments; and it could collect
> > and upload logs as well. It could also explain certain things to users
> > when not fulfilled, for example why it's not worth to report a problem
> > that happens with an old kernel.
> 
> Christoph mentioned Debian's reportbug utility. That does a pretty good
> job at walking people through how to report a bug. It could also get
> information about the current environment that would be useful too. Perhaps
> something like that?
> 
> >    Sure, these apps never work perfect and doing it right is a lot of
> > work, but I guess one could make things a lot easier for many users
> > especially for our case. I assume other projects have done something
> > like that so that we could learn from them.
> > 
> >  * Improve https://docs.kernel.org/admin-guide/reporting-issues.html
> > further. I have some ideas there, but other things are higher on my
> > priority list currently. That document in the end somehow needs to
> > become less scary looking while still providing all important details
> > for situations where a reporter might need them.
> > 
> >  * Write new docs relevant for bug reporting. We for example still have
> > no well written and simple to understand text that explains bisection to
> > people that are new to git, bisection, or compiling kernels in general.
> > Speaking of which: we iirc are also missing a text that properly
> > explains how to quickly configure and compile a kernel using "make
> > localmodconfig" (I mean something like
> > http://www.h-online.com/open/features/Good-and-quick-kernel-configuration-creation-1403046.html)
> 
> The sad part is that most people that are going to report a bug is not
> going to read a full document to figure out how to do it. Usually when
> someone hits a bug, they are doing something else. And it's a burden to
> report it. Obviously, they want it to be fixed, but it's viewed as a favor
> to the developer and not the user to get it fixed, as it's likely seen as a
> mistake by the developer that the bug exists in the first place.

It really depends on how badly the bug affects the reporter. I'm sure
that a bug that prevents GPU or audio from working alone on a shiny
brand new laptop will see lots of pings. A side issue noticed by the
user that wouldn't really affect them is more likely to end up in a
blackhole. I recently faced issues with a display controller. I sent
patches for the problems affecting my use case, and only notified the
maintainer for the other issues. Those have been "added to their todo
list (TM)". But is that really a problem ? If I'm not affected and
neither is the maintainer, there's likely better use of their time, at
least until a user who is really affected by the problem shows up.

> Having a tool like reportbug that walks you through the steps of reporting
> it would be the best way to do so. As the reporter doesn't need to think
> too hard and just answer questions and let the tool do all the work.
> 
> >  * Not sure, maybe a list of things that known to be broken might be
> > good to have? Like "yes, we know that nouveau is slow, but we can't do
> > anything about this" or "driver 'wifi-foo' only supports a small subset
> > of the features the hardware offers, so don't report bugs if bar, baz
> > and foobar don't work".
> 
> The tool could possibly reply with known issues, and state something like
> "We are aware of this issue, and are currently trying to figure it out."

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Mon, 3 Oct 2022 18:44:45 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> > The sad part is that most people that are going to report a bug is not
> > going to read a full document to figure out how to do it. Usually when
> > someone hits a bug, they are doing something else. And it's a burden to
> > report it. Obviously, they want it to be fixed, but it's viewed as a favor
> > to the developer and not the user to get it fixed, as it's likely seen as a
> > mistake by the developer that the bug exists in the first place.  
> 
> It really depends on how badly the bug affects the reporter. I'm sure
> that a bug that prevents GPU or audio from working alone on a shiny
> brand new laptop will see lots of pings. A side issue noticed by the
> user that wouldn't really affect them is more likely to end up in a
> blackhole. I recently faced issues with a display controller. I sent
> patches for the problems affecting my use case, and only notified the
> maintainer for the other issues. Those have been "added to their todo
> list (TM)". But is that really a problem ? If I'm not affected and
> neither is the maintainer, there's likely better use of their time, at
> least until a user who is really affected by the problem shows up.

I guess that's the main question. If we see hundreds of bugzilla reports
ignored, are they the one offs that nobody really cares about, or are they
the ones where it's preventing someone from using their new laptop properly?

Sometimes, even if it prevents a laptop from working properly, it could be
ignored if a workaround is in place. Like just buying an external webcam if
you can't get the internal one working.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Laurent Pinchart]

On Mon, Oct 03, 2022 at 11:51:02AM -0400, Steven Rostedt wrote:
> On Mon, 3 Oct 2022 18:44:45 +0300 Laurent Pinchart wrote:
> 
> > > The sad part is that most people that are going to report a bug is not
> > > going to read a full document to figure out how to do it. Usually when
> > > someone hits a bug, they are doing something else. And it's a burden to
> > > report it. Obviously, they want it to be fixed, but it's viewed as a favor
> > > to the developer and not the user to get it fixed, as it's likely seen as a
> > > mistake by the developer that the bug exists in the first place.  
> > 
> > It really depends on how badly the bug affects the reporter. I'm sure
> > that a bug that prevents GPU or audio from working alone on a shiny
> > brand new laptop will see lots of pings. A side issue noticed by the
> > user that wouldn't really affect them is more likely to end up in a
> > blackhole. I recently faced issues with a display controller. I sent
> > patches for the problems affecting my use case, and only notified the
> > maintainer for the other issues. Those have been "added to their todo
> > list (TM)". But is that really a problem ? If I'm not affected and
> > neither is the maintainer, there's likely better use of their time, at
> > least until a user who is really affected by the problem shows up.
> 
> I guess that's the main question. If we see hundreds of bugzilla reports
> ignored, are they the one offs that nobody really cares about, or are they
> the ones where it's preventing someone from using their new laptop properly?
> 
> Sometimes, even if it prevents a laptop from working properly, it could be
> ignored if a workaround is in place. Like just buying an external webcam if
> you can't get the internal one working.

That's an interesting example. https://lwn.net/Articles/904776/ shows
how it made lots of users *very* unhappy.

-- 
Regards,

Laurent Pinchart

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Mon, 3 Oct 2022 18:59:22 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> > Sometimes, even if it prevents a laptop from working properly, it could be
> > ignored if a workaround is in place. Like just buying an external webcam if
> > you can't get the internal one working.  
> 
> That's an interesting example. https://lwn.net/Articles/904776/ shows
> how it made lots of users *very* unhappy.

I figured you would appreciate that example ;-)

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Al Viro]

On Mon, Oct 03, 2022 at 10:20:29AM -0400, Steven Rostedt wrote:
> On Mon, 3 Oct 2022 09:40:43 +0000
> "Artem S. Tashkinov" <aros@gmx.com> wrote:
> 
> > For instance, I've CC'ed Linus Torvalds _privately_ from Bugzilla twice
> > and he _chimed_ in and _helped_ resolve the bugs.
> 
> You didn't Cc Linus _privately_, because you Cc'd him from Bugzilla. I'm
> guessing that means it's a public conversation. Which is similar to Cc'ing
> a maintainer and a public mailing list.
> 
> > My messages to LKML
> > were _ignored_ by +1000 people subscribed to it.
> 
> LKML gets 800 emails a day. Nobody reads it (besides Jon Corbet and Andrew
> Morton). But if you send email to a maintainer privately without Cc'ing any
> public mailing list (or Bugzilla), then it will likely be ignored.

Way more than 800, IME.  And I'm still subscribed to it, even though
reading through the damn thing isn't physically possible.  About 1 or 2
percents gets past the "delete unopened" pass...

Speaking of private mail... there's one case when it's warranted -
a bug that looks like a sufficiently nasty security hole in something that
would be sufficiently widely deployed.  Preferably - with something along
the lines of "off-list due to potential security impact".

Still a matter of taste - security@ is an option for those...
 
> What we are saying is, you need to do both. Cc the maintainer _and_ a
> public mailing list. That way the maintainer knows others can see it, and
> could point someone else to look at it if they do not have the time, or
> they know someone who can better help.

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Mon, 3 Oct 2022 19:24:07 +0100
Al Viro <viro@zeniv.linux.org.uk> wrote:

> Way more than 800, IME.  And I'm still subscribed to it, even though
> reading through the damn thing isn't physically possible.  About 1 or 2
> percents gets past the "delete unopened" pass...

I keep the last 10 weeks in my folder (and archive the rest.) That's 70
days worth, and I have 78,109 emails currently in that folder. OK, it's
been a while since I last took the average. It appears to be 1114 emails
per day now. I blame the extra 300 emails a day being the stable updates :-D

> 
> Speaking of private mail... there's one case when it's warranted -
> a bug that looks like a sufficiently nasty security hole in something that
> would be sufficiently widely deployed.  Preferably - with something along
> the lines of "off-list due to potential security impact".
> 
> Still a matter of taste - security@ is an option for those...

I was about to say "then include the security@ mailing list". ;-)

It's still not a private one. But for those that do not know about that
mailing list, yeah, private is fine. But that's not really the focus of
this discussion.

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Slade Watkins]

> On Oct 3, 2022, at 3:07 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> On Mon, 3 Oct 2022 19:24:07 +0100
> Al Viro <viro@zeniv.linux.org.uk> wrote:
> 
>> Way more than 800, IME.  And I'm still subscribed to it, even though
>> reading through the damn thing isn't physically possible.  About 1 or 2
>> percents gets past the "delete unopened" pass...
> 
> I keep the last 10 weeks in my folder (and archive the rest.) That's 70
> days worth, and I have 78,109 emails currently in that folder. OK, it's
> been a while since I last took the average. It appears to be 1114 emails
> per day now. I blame the extra 300 emails a day being the stable updates :-D

I keep emails under three circumstances:
1) emails pertaining to whatever window we’re in on mainline. Since that’s 6.1, I only have emails pertaining to that.
2) the two most recent stable-rc emails for current stable versions
3) anything I’ve replied to or am Cc’d on

I erase emails each window for number 1, and numbers 2+3 get their emails erased after a month of no activity.

But, I do try to at least skim through everything that comes through LKML so I’m in-the-know, so to speak. (I know, that’s strange, but I’m a fast reader and am very deeply interested in it so it’s never been hard for me to keep up on the list.)

-srw

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 03.10.22 17:37, Konstantin Ryabitsev wrote:
>
> If we auto-create accounts for MAINTAINERS, that would allow them to be cc'd
> by an actual human being triaging bugs, [...]

For the record: that would not be enough, as for bisected regressions
you often want to CC the author of the culprit who might not be a
maintainer. To catch that case as well, you'd have to create account for
everyone that contributes a change.

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Conor Dooley]

On Tue, Oct 04, 2022 at 09:37:29AM +0200, Thorsten Leemhuis wrote:
> On 03.10.22 17:37, Konstantin Ryabitsev wrote:
> >
> > If we auto-create accounts for MAINTAINERS, that would allow them to be cc'd
> > by an actual human being triaging bugs, [...]
> 
> For the record: that would not be enough, as for bisected regressions
> you often want to CC the author of the culprit who might not be a

And possibly those who provided a review for the original patch too.

> maintainer. To catch that case as well, you'd have to create account for
> everyone that contributes a change.

As someone blissfully unaware of the workings of bugzilla, it is
possible to tie multiple emails to the same account? Not that I would be
happy about having an account created for me, but I certainly would not
want more than one account..

Thanks,
Conor.

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 03.10.22 17:26, Steven Rostedt wrote:
> On Mon, 3 Oct 2022 14:59:54 +0200
> Thorsten Leemhuis <linux@leemhuis.info> wrote:
> 
>>> With the band-aids you outline in place: do you think it would it be
>>> beneficial to have a liaison holding users’s hands through the
>>> process, _then_ triaging to developers by contacting them with the
>>> information they need?  
>>
>> Well, yes and no. :-/
>>
>> Thing is: up to a point that's something I do already (and will likely
>> continue to do at least for a while) when the reported issue is a
>> regression. But to be fair, I often could help way more if I wanted to,
>> but there are only so many hours in a day and other things to take care
>> of (regression tracking is only a part-time thing for me currently). So
>> some help there might be handy; would get load of the developers as
>> well, as they often are more willing to help users when a report is
>> about a regression.
> 
> Are you asking for help in the regression tracking?

Well, like most people I have lots of ideas for things I'd like to work
on in the domain I'm active in, as the stuff I mentioned in my previous
mail show. But that's life, we all have todo and wish lists. Sooner or
later I'll likely get down to some of that stuff. But if somebody said
"we'd like you to work on this or that rather sooner than later", then
sure, help would be great. Especially help with maintaining and
improving regzbot would be great from my point of view.

But what right now would help most with regression tracking are two
things that can only be solved by educating people:

 * It would help tremendously if reporters and developers would at least
CC the regressions list on regression reports so I become aware of them;
and ideally they would tell regzbot about the report themselves of
course. Right now many people don't do either, so I have to scan various
mails every day to even get aware of most regression reports (I do this
with lei, which searches for things like "regress", "bisect", "revert",
"first bad commit", which generates many false positives :-/ ).

 * It would help a lot if way more developers would use "Link:" tags as
explained by our documentation (e.g. use them to link to reports), as I
have to track things manually if they are missing.

I'm working on educating people and I guess over time things will improve.

>> But for other issues (aka regular bugs) I don't think it's worth it,
>> because why only help those users that report to bugzilla (you didn't
>> say that, but it sounded to me like the focus is on it)? There are
>> people that try to use the mailing lists, but do it badly and never get
>> a reply (for example because they sent their report just to LKML). They
>> could need help, too; maybe helping them should even be priority, as
>> they at least tried to do what most kernel developers want them to do,
>> hence their reports might be better, too.
> Could do both.

Sure!

>> But there is a more important reason why I think having a liaison might
>> not be worth it for now: It IMHO would be much better to spend the time
>> and effort on other things that enable users submitting better bug
>> reports in the first place. I have no concrete and well-thought-out
>> ideas at hand what to do exactly, but here are a few vague ones:
>>
>>  * create an app (ideally usable locally and on the web) that guides
>> users through generating a good bug report (let's leave the way of
>> submission aside for now). That app could handle quite a few of the
>> steps that https://docs.kernel.org/admin-guide/reporting-issues.html
>> currently mentions. It for example could check if the kernel looks to be
>> vanilla, if the kernel is fresh, if the kernel is tainted, if an Oops is
>> the first one or just a follow-up error; maybe that app could even
>> decode stack-traces locally in some environments; and it could collect
>> and upload logs as well. It could also explain certain things to users
>> when not fulfilled, for example why it's not worth to report a problem
>> that happens with an old kernel.
> 
> Christoph mentioned Debian's reportbug utility. That does a pretty good
> job at walking people through how to report a bug. It could also get
> information about the current environment that would be useful too. Perhaps
> something like that?

Never came in contact with it but will put "take a closer look at it" on
my list.

>>    Sure, these apps never work perfect and doing it right is a lot of
>> work, but I guess one could make things a lot easier for many users
>> especially for our case. I assume other projects have done something
>> like that so that we could learn from them.
>>
>>  * Improve https://docs.kernel.org/admin-guide/reporting-issues.html
>> further. I have some ideas there, but other things are higher on my
>> priority list currently. That document in the end somehow needs to
>> become less scary looking while still providing all important details
>> for situations where a reporter might need them.
>>
>>  * Write new docs relevant for bug reporting. We for example still have
>> no well written and simple to understand text that explains bisection to
>> people that are new to git, bisection, or compiling kernels in general.
>> Speaking of which: we iirc are also missing a text that properly
>> explains how to quickly configure and compile a kernel using "make
>> localmodconfig" (I mean something like
>> http://www.h-online.com/open/features/Good-and-quick-kernel-configuration-creation-1403046.html)
> 
> The sad part is that most people that are going to report a bug is not
> going to read a full document to figure out how to do it. Usually when
> someone hits a bug, they are doing something else. And it's a burden to
> report it. Obviously, they want it to be fixed, but it's viewed as a favor
> to the developer and not the user to get it fixed, as it's likely seen as a
> mistake by the developer that the bug exists in the first place.
> 
> Having a tool like reportbug that walks you through the steps of reporting
> it would be the best way to do so. As the reporter doesn't need to think
> too hard and just answer questions and let the tool do all the work.

Yup, I'm all for such a tool (and would likely already have started one,
but I need to get regzbot to a more mature state first; and I actually
have no idea how to write a tool that ideally works locally and on the
web...).

But I consider explaining things like bisection and localmodconfig in
the documentation as also important, as that's likely something the tool
won't be able to automate any time soon (or never, as realizing that is
likely hard and better left to a separate tool anyway).

>>  * Not sure, maybe a list of things that known to be broken might be
>> good to have? Like "yes, we know that nouveau is slow, but we can't do
>> anything about this" or "driver 'wifi-foo' only supports a small subset
>> of the features the hardware offers, so don't report bugs if bar, baz
>> and foobar don't work".
> 
> The tool could possibly reply with known issues, and state something like
> "We are aware of this issue, and are currently trying to figure it out."

+1

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Thorsten,

On Tue, Oct 4, 2022 at 10:41 AM Thorsten Leemhuis <linux@leemhuis.info> wrote:
> But I consider explaining things like bisection and localmodconfig in
> the documentation as also important, as that's likely something the tool
> won't be able to automate any time soon (or never, as realizing that is
> likely hard and better left to a separate tool anyway).

Creating a simple Linux-specific wrapper around git bisect under
scripts/ might be useful?
The wrapper could copy .config to
$(srctree)/arch/$(ARCH)/config/bisect_defconfig, automatically run
"make bisect_defconfig" in each step, and show not only the bisected
commit, but also the impact on .config.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 04.10.22 11:20, Geert Uytterhoeven wrote:
> Hi Thorsten,
> 
> On Tue, Oct 4, 2022 at 10:41 AM Thorsten Leemhuis <linux@leemhuis.info> wrote:
>> But I consider explaining things like bisection and localmodconfig in
>> the documentation as also important, as that's likely something the tool
>> won't be able to automate any time soon (or never, as realizing that is
>> likely hard and better left to a separate tool anyway).
> 
> Creating a simple Linux-specific wrapper around git bisect under
> scripts/ might be useful?
> The wrapper could copy .config to
> $(srctree)/arch/$(ARCH)/config/bisect_defconfig, automatically run
> "make bisect_defconfig" in each step, and show not only the bisected
> commit, but also the impact on .config.

Don't worry, I still remember that trick of yours from this discussion:
https://lore.kernel.org/all/12e09497-a848-b767-88f4-7dabd8360c5e@leemhuis.info/

Something like that would be a start, but I'd say having localmodconfig
covered would be wise also, as it speeds things up tremendously for
those that start with a full-blown x86 pc distro config.

There are also people that find regressions when updating from say
v5.18.15 to v5.19.4 and want to bisect that range; never tried if that
actually works with a stable git tree, but I'd assume that approach is
unwise. I also assume a lot of people would prefer to download only the
recent history or specific stable branches when cloning the git tree
(which is possible if you know what to do, but I guess most people don't).

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Thorsten,

On Tue, Oct 4, 2022 at 12:16 PM Thorsten Leemhuis <linux@leemhuis.info> wrote:
> On 04.10.22 11:20, Geert Uytterhoeven wrote:
> > On Tue, Oct 4, 2022 at 10:41 AM Thorsten Leemhuis <linux@leemhuis.info> wrote:
> >> But I consider explaining things like bisection and localmodconfig in
> >> the documentation as also important, as that's likely something the tool
> >> won't be able to automate any time soon (or never, as realizing that is
> >> likely hard and better left to a separate tool anyway).
> >
> > Creating a simple Linux-specific wrapper around git bisect under
> > scripts/ might be useful?
> > The wrapper could copy .config to
> > $(srctree)/arch/$(ARCH)/config/bisect_defconfig, automatically run
> > "make bisect_defconfig" in each step, and show not only the bisected
> > commit, but also the impact on .config.
>
> Don't worry, I still remember that trick of yours from this discussion:
> https://lore.kernel.org/all/12e09497-a848-b767-88f4-7dabd8360c5e@leemhuis.info/

OK ;-)

> Something like that would be a start, but I'd say having localmodconfig
> covered would be wise also, as it speeds things up tremendously for
> those that start with a full-blown x86 pc distro config.

That's not that much different, as you only need to run "make localmodconfig"
once, as the first step (or as step zero, before starting the bisection).

> There are also people that find regressions when updating from say
> v5.18.15 to v5.19.4 and want to bisect that range; never tried if that
> actually works with a stable git tree, but I'd assume that approach is
> unwise. I also assume a lot of people would prefer to download only the

Yeah, you may run into issues that are fixed in v5.18.15, but not in
v5.18 itself, or in later intermediate steps.
For a short range like v5.18.15 to v5.19.4 (which are not LTS, hence
didn't receive that many updates, which can be good or bad), I don't
expect many problems, though

There are similar (but much worse) issues with bisecting between two
linux-next releases.

> recent history or specific stable branches when cloning the git tree
> (which is possible if you know what to do, but I guess most people don't).

Does it really save that much bandwidth?
How many minutes of 4K streaming video is the kernel nowadays? ;-)

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/3/22 14:20, Steven Rostedt wrote:
> On Mon, 3 Oct 2022 09:40:43 +0000
> "Artem S. Tashkinov" <aros@gmx.com> wrote:
>
>> For instance, I've CC'ed Linus Torvalds _privately_ from Bugzilla twice
>> and he _chimed_ in and _helped_ resolve the bugs.
>
> You didn't Cc Linus _privately_, because you Cc'd him from Bugzilla. I'm
> guessing that means it's a public conversation. Which is similar to Cc'ing
> a maintainer and a public mailing list.

I _did_ CC him privately by adding his _personal_ e-mail. I'm astonished
not only you don't believe me you turn my words inside out.

>
>> My messages to LKML
>> were _ignored_ by +1000 people subscribed to it.
>
> LKML gets 800 emails a day. Nobody reads it (besides Jon Corbet and Andrew
> Morton). But if you send email to a maintainer privately without Cc'ing any
> public mailing list (or Bugzilla), then it will likely be ignored.

Wow, so pretty much the vast majority of people here advocate for
deprecating Bugzilla and asking non-IT people to use something which is
essentially a ... SPAM list?

Woah.

I've given almost a dozen reasons why mailing lists simply don't work as
a bug tracker in absolute most cases.

BTW, this discussion is a perfect f-ing example of that. What could have
been easily read in a tracker needs to be repeated over and over and
over again because you didn't bother to read previous messages 'cause
you were busy, not paying attention, simply forgot and you don't want to
scroll days of messages in your inbox.

God, this is so ugly it's cringe worthy.

Most people here who advocate for killing off Bugzilla:

1) Have _never_ used it
2) Have troubles even following _this_ conversation

That' ridiculous.

>
> What we are saying is, you need to do both. Cc the maintainer _and_ a
> public mailing list. That way the maintainer knows others can see it, and
> could point someone else to look at it if they do not have the time, or
> they know someone who can better help.
>
> -- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/3/22 14:22, Konstantin Ryabitsev wrote:
> On Mon, Oct 03, 2022 at 09:16:06AM +0000, Artem S. Tashkinov wrote:
>> The initial conversation started with the fact that Bugzilla is old,
>> semi-deprecated, requires MySQL [no idea what's bad about it, Bugzilla
>> can work with MariaDB and Percona as well]
>
> It can't, actually. It only works with MySQL 5.7 or an equally ancient MariaDB.
> No, there is no official fix (because nobody is working on bugzilla).
> https://bugzilla.mozilla.org/show_bug.cgi?id=1592129
>

What do you think about Bugzilla Harmony? Works with MariaDB:

https://github.com/bugzilla/harmony

A continuation of Bugzilla.

Regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/3/22 15:37, Konstantin Ryabitsev wrote:
> On Mon, Oct 03, 2022 at 11:55:25AM +0300, Mike Rapoport wrote:
>> If I'm not mistaken, bugzilla lets CC people explicitly. How the database
>> of emails in bugzilla would help choosing the right people to CC better
>> than MAINTAINERS?
>
> It can't, actually, which is I think is the crux of misunderstanding here. I
> think what Artem is proposing is to *auto-create bugzilla accounts* for anyone
> who shows up in MAINTAINERS, so that they can be cc'd on a bug report.
> However, everyone understood this as "add these people as default assignees,"
> which is not the case.
>
> If we auto-create accounts for MAINTAINERS, that would allow them to be cc'd
> by an actual human being triaging bugs, but won't lead to any discernable
> increase of bugzilla mail.
>
> Artem, please correct me if I'm wrong.

Exactly. Only maintainers and mailing lists will be assigned
automatically as it is _now_. Other developers need to be CC'ed _manually_.

There's no SPAM issue or never has been about what I proposed. Absolute
most people will never CC anyone 'cause bug reporters are normally quite
lazy or not experienced enough to grep git logs.

You're absolutely right.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Geert Uytterhoeven]

Hi Artem,

On Tue, Oct 4, 2022 at 2:16 PM Artem S. Tashkinov <aros@gmx.com> wrote:
> On 10/3/22 14:20, Steven Rostedt wrote:
> > On Mon, 3 Oct 2022 09:40:43 +0000
> > "Artem S. Tashkinov" <aros@gmx.com> wrote:
> >> For instance, I've CC'ed Linus Torvalds _privately_ from Bugzilla twice
> >> and he _chimed_ in and _helped_ resolve the bugs.
> >
> > You didn't Cc Linus _privately_, because you Cc'd him from Bugzilla. I'm
> > guessing that means it's a public conversation. Which is similar to Cc'ing
> > a maintainer and a public mailing list.
>
> I _did_ CC him privately by adding his _personal_ e-mail. I'm astonished
> not only you don't believe me you turn my words inside out.

I think there is a misunderstanding of the meaning of "CC privately".
To me it means no public data disclosing entity (be it a public mailing
list, or a public bug tracker) was CCed as well.
To you, it seems to mean you used his personal email address instead
of a mailing list address.

> Wow, so pretty much the vast majority of people here advocate for
> deprecating Bugzilla and asking non-IT people to use something which is
> essentially a ... SPAM list?
>
> Woah.
>
> I've given almost a dozen reasons why mailing lists simply don't work as
> a bug tracker in absolute most cases.

And people disagree.  No amount of "Woah" will change that, only
facts and figures can do.

> BTW, this discussion is a perfect f-ing example of that. What could have
> been easily read in a tracker needs to be repeated over and over and
> over again because you didn't bother to read previous messages 'cause
> you were busy, not paying attention, simply forgot and you don't want to
> scroll days of messages in your inbox.
>
> God, this is so ugly it's cringe worthy.
>
> Most people here who advocate for killing off Bugzilla:
>
> 1) Have _never_ used it
> 2) Have troubles even following _this_ conversation
>
> That' ridiculous.

Thanks for your insults.
This is not the way to convince people.

P.S. I did read all of it, I may stop doing so soon...

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Tue, Oct 04, 2022 at 12:21:32PM +0000, Artem S. Tashkinov wrote:
> > It can't, actually. It only works with MySQL 5.7 or an equally ancient MariaDB.
> > No, there is no official fix (because nobody is working on bugzilla).
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1592129
> > 
> 
> What do you think about Bugzilla Harmony? Works with MariaDB:
> 
> https://github.com/bugzilla/harmony
> 
> A continuation of Bugzilla.

It doesn't look like there's enough momentum behind it, at last at this time.
We do have a plan in place to avoid the MySQL problem by moving our DB to
Postgres, so the software side of things isn't something we need to focus on
fixing at this time.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Tue, Oct 04, 2022 at 12:24:23PM +0000, Artem S. Tashkinov wrote:
> > It can't, actually, which is I think is the crux of misunderstanding here. I
> > think what Artem is proposing is to *auto-create bugzilla accounts* for anyone
> > who shows up in MAINTAINERS, so that they can be cc'd on a bug report.
> > However, everyone understood this as "add these people as default assignees,"
> > which is not the case.
> > 
> > If we auto-create accounts for MAINTAINERS, that would allow them to be cc'd
> > by an actual human being triaging bugs, but won't lead to any discernable
> > increase of bugzilla mail.
> > 
> > Artem, please correct me if I'm wrong.
> 
> Exactly. Only maintainers and mailing lists will be assigned
> automatically as it is _now_. Other developers need to be CC'ed _manually_.
> 
> There's no SPAM issue or never has been about what I proposed. Absolute
> most people will never CC anyone 'cause bug reporters are normally quite
> lazy or not experienced enough to grep git logs.

It's still not a perfect plan, because this is what tends to happen:

1. someone is cc'd via the bugzilla interface
2. the maintainer responds via replying to the email and cc's someone else
3. bugzilla doesn't automatically synchronize the email Cc: fields and the
   bug's cc list, so any responses posted via the web interface don't go to
   people who were cc'd via email

Let me ponder what can be done about that.

If you're interested in helping to get the bugzilla product list into a sane
format, I can suggest that you work on creating a mapping file between
bugzilla product/component and MAINTAINERS entry, maybe just for the
"S: Maintained" and "S: Supported" ones?

E.g., starting from the top:

    3CR990 NETWORK DRIVER -> Drivers/3CR990
    3WARE SAS/SATA-RAID SCSI DRIVERS (3W-XXXX, 3W-9XXX, 3W-SAS) -> Drivers/3WARE
    53C700 AND 53C700-66 SCSI DRIVER -> Drivers/53C700
    ... etc ...

That would allow me to automate the creation of components *and* work on a
better email bridge than what bugzilla can currently do.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Thu, Sep 29, 2022 at 01:19:24PM +0200, Thorsten Leemhuis wrote:
> Hi!
> 
> TLDR: Core Linux kernel developers are unhappy with the state of
> bugzilla.kernel.org; to improve things I plan to change a few important
> aspects of its configuration, unless somebody comes up with better ideas
> to tackle current problems: (1) Create a catch-all product making it
> totally obvious to submitters that likely nobody will look into the
> ticket. (2) Remove or hide all products & components where the subsystem
> didn't fully commit to look into newly submitted reports. (3) Change the
> text on the front page to make it clear that most kernel bug reports
> need to be sent by mail.

Here's my counter-plan, which builds on top of yours.

1. Create a Kernel/Kernel product that acts as a starting point for all bug
   submissions.
2. Create and maintain a mapping from MAINTAINER subsystem entries to
   Product/Component categories in Bugzilla (the scheme to be established).
3. Establish and maintain a team of designated triage people who are willing
   to look at incoming bugs to either:

   a. quick-close them as non-actionable (tainted kernel, distro kernel, spam)
   b. obtain missing information from the submitter as necessary
   c. figure out the correct component to assign, to the best of their ability
   d. set a "triaged" flag

4. a backend monitoring bot will track all bug changes and, when it sees a bug
   get the "triaged" state, it will:

   a. create a useful bug summary from all bug comments
   b. figure out who to notify based on the mapping (see #2 above)
   c. send out the email to everyone identified

5. the same backend monitoring bot will track responses and update the bug
   comments as needed; any comments added via the bugzilla site will be
   similarly sent out as follow-up messages.

6. the bot can also monitor commits and other discussions via lore.kernel.org
   and automatically add comments/links when it sees the bug mentioned
   elsewhere.

I'm happy to take care of everything bot-related (apparently, programming bots
is what I do now -- I just wish it was the cool and glamorous kind).

As I have stated multiple times, the hard part will be keeping a team of
people who are willing to do the bug triage work, but maybe we can start with
Greg KH using his intern funds to hire someone (assuming he's not already
using these funds for someone to help him with all the other tasks).

Does that sound like a plan for everyone?

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Greg KH]

On Tue, Oct 04, 2022 at 01:53:54PM -0400, Konstantin Ryabitsev wrote:
> As I have stated multiple times, the hard part will be keeping a team of
> people who are willing to do the bug triage work, but maybe we can start with
> Greg KH using his intern funds to hire someone (assuming he's not already
> using these funds for someone to help him with all the other tasks).

I have no interns anymore, and the ones that the LF does have in the
kernel program are using all of the remaining budget that we have, so
much so that we have a whole bunch of unpaid ones at the same time as we
have so many people applying for the process.

So I don't think you can use those funds, they are all spoken for,
sorry.

greg k-h

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Linus Torvalds]

On Tue, Oct 4, 2022 at 10:53 AM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> 2. Create and maintain a mapping from MAINTAINER subsystem entries to
>    Product/Component categories in Bugzilla (the scheme to be established).

It's probably worth asking the 0day people what they do.

Maybe they do it all manually and have no real helping infrastructure,
but I see emails from them that often (but certainly not always) seem
to get the right people involved.

And while the MAINTAINER file is useful for a fiel mapping, I'm not
convinced it's all that useful for the "product/component category"
mapping, because I doubt people will actually fill that in well (and
reliably) enough.

With actual bisection data, it's fairly easy (get the emails from the
commit that got bisected). But things like "use the backtrace in the
oops to figure out who to add to participants" is likely a bit more of
a "use clever heuristics" kind of thing.

Anyway, I do think that some kind of automation would be really good,
at least for reports that have bisection information or backtraces in
them. Without automation, people _will_ be overwhelmed on the first
level response to bug reports (ie the "try to figure out who to bring
in" front).

But if the automation is too stupid, people will start ignoring the
report emails just on the assumption that it got thihngs wrong.

Of course, if the automation is really solid enough, I think it should
work on lore.kernel.org, not on just a bugzilla thing.

             Linus

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Tue, Oct 04, 2022 at 11:03:48AM -0700, Linus Torvalds wrote:
> And while the MAINTAINER file is useful for a fiel mapping, I'm not
> convinced it's all that useful for the "product/component category"
> mapping, because I doubt people will actually fill that in well (and
> reliably) enough.

Sure, but for a best-effort first go at it, it may do more good than harm. If
someone says "this is not mine, sorry, try X", the triage team will select the
suggested component instead and retrigger the bot with a new set of
addressees.

> With actual bisection data, it's fairly easy (get the emails from the
> commit that got bisected). But things like "use the backtrace in the
> oops to figure out who to add to participants" is likely a bit more of
> a "use clever heuristics" kind of thing.
> 
> Anyway, I do think that some kind of automation would be really good,
> at least for reports that have bisection information or backtraces in
> them. Without automation, people _will_ be overwhelmed on the first
> level response to bug reports (ie the "try to figure out who to bring
> in" front).
> 
> But if the automation is too stupid, people will start ignoring the
> report emails just on the assumption that it got thihngs wrong.

Well, then at worst we'll have gone a full circle, since that's the situation
right now anyway.

> Of course, if the automation is really solid enough, I think it should
> work on lore.kernel.org, not on just a bugzilla thing.

It would be cool if we could use all those big AI projects at LF to help out
here. The trouble is that there's not really anything to train it on, because
there is no reliable mapping from message threads to subsystem components.

Maybe as the triage team goes along, it can start feeding correctly triaged
bugs to a PyTorch instance. :)

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Tue, Oct 04, 2022 at 08:02:55PM +0200, Greg KH wrote:
> On Tue, Oct 04, 2022 at 01:53:54PM -0400, Konstantin Ryabitsev wrote:
> > As I have stated multiple times, the hard part will be keeping a team of
> > people who are willing to do the bug triage work, but maybe we can start with
> > Greg KH using his intern funds to hire someone (assuming he's not already
> > using these funds for someone to help him with all the other tasks).
> 
> I have no interns anymore, and the ones that the LF does have in the
> kernel program are using all of the remaining budget that we have, so
> much so that we have a whole bunch of unpaid ones at the same time as we
> have so many people applying for the process.
> 
> So I don't think you can use those funds, they are all spoken for,
> sorry.


Yeah, I wasn't actually serious about the interns. An effort like this needs
to have a separate fund allocation that can't be cannibalized for any other
purpose.

-K
[Fixing the ksummit address]

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Jani Nikula]

On Tue, 04 Oct 2022, Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> I'm happy to take care of everything bot-related (apparently, programming bots
> is what I do now -- I just wish it was the cool and glamorous kind).

As to bots, one idea would be to go through old & unchanged bugs after
every kernel release or so, ask to reproduce on the new kernel, and auto
close if there's no response within some time frame. This could be very
relaxed for starters, but would start closing all the stale and
neglected bugs that have crept up that are of no use to anyone.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Graphics Center

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Tue, Oct 04, 2022 at 10:21:27PM +0300, Jani Nikula wrote:
> As to bots, one idea would be to go through old & unchanged bugs after
> every kernel release or so, ask to reproduce on the new kernel, and auto
> close if there's no response within some time frame. This could be very
> relaxed for starters, but would start closing all the stale and
> neglected bugs that have crept up that are of no use to anyone.

This is what Fedora does, and I've always felt it creates more annoyance than
good, but if that's the consensus, I'll be happy to implement this.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 04.10.22 19:53, Konstantin Ryabitsev wrote:
> On Thu, Sep 29, 2022 at 01:19:24PM +0200, Thorsten Leemhuis wrote:
>>
>> TLDR: Core Linux kernel developers are unhappy with the state of
>> bugzilla.kernel.org; to improve things I plan to change a few important
>> aspects of its configuration, unless somebody comes up with better ideas
>> to tackle current problems: (1) Create a catch-all product making it
>> totally obvious to submitters that likely nobody will look into the
>> ticket. (2) Remove or hide all products & components where the subsystem
>> didn't fully commit to look into newly submitted reports. (3) Change the
>> text on the front page to make it clear that most kernel bug reports
>> need to be sent by mail.
> 
> Here's my counter-plan, which builds on top of yours. 

Thx for working that out, much appreciated. You already mentioned the
hardest thing and others already raised a few few aspects that sprung to
my mind. That for now leaves one big question in my head:

Your plan would afaics mean that we invest further into a software
abandoned by its upstream and already becoming more and more of a
maintenance burden. That investment would also further increase our
dependency on that software by establishing workflows that rely on it.
Is that really wise at this point? Wouldn't it be better to spend that
time and effort to build something better that is more future proof?

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Tue, Oct 04, 2022 at 10:06:28PM +0200, Thorsten Leemhuis wrote:
> Your plan would afaics mean that we invest further into a software
> abandoned by its upstream and already becoming more and more of a
> maintenance burden. That investment would also further increase our
> dependency on that software by establishing workflows that rely on it.
> Is that really wise at this point? Wouldn't it be better to spend that
> time and effort to build something better that is more future proof?

Unfortunately, there's no such thing. ;) And maybe we'll even help tip the
course of history into the other direction -- Red Hat uses bugzilla, and so
does OpenSuse, so there's a pretty good core of well-funded companies that
would be in a position to help keep bugzilla going if it's looking like the
platform is still alive. Or that could all be wishful thinking and they'll all
migrate to Jira or something equally horrible, who knows.

I'm hoping that by keeping the bulk of this exchange relying on established
decentralized end-to-end messaging, we won't be painting ourselves into the
corner quite as much as with a tool that requires all interaction to happen
strictly via the web interface.

The alternative is to hire the folks behind patchwork to write "bugwork".

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Steven Rostedt]

On Tue, 4 Oct 2022 14:32:26 +0200
Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> > I _did_ CC him privately by adding his _personal_ e-mail. I'm astonished
> > not only you don't believe me you turn my words inside out.  
> 
> I think there is a misunderstanding of the meaning of "CC privately".
> To me it means no public data disclosing entity (be it a public mailing
> list, or a public bug tracker) was CCed as well.
> To you, it seems to mean you used his personal email address instead
> of a mailing list address.

Exactly. If I can go find it, it wasn't "private".

-- Steve

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 04.10.22 22:25, Konstantin Ryabitsev wrote:
> On Tue, Oct 04, 2022 at 10:06:28PM +0200, Thorsten Leemhuis wrote:
>> Your plan would afaics mean that we invest further into a software
>> abandoned by its upstream and already becoming more and more of a
>> maintenance burden. That investment would also further increase our
>> dependency on that software by establishing workflows that rely on it.
>> Is that really wise at this point? Wouldn't it be better to spend that
>> time and effort to build something better that is more future proof?
> 
> Unfortunately, there's no such thing. ;) And maybe we'll even help tip the
> course of history into the other direction -- Red Hat uses bugzilla, and so
> does OpenSuse, so there's a pretty good core of well-funded companies that
> would be in a position to help keep bugzilla going if it's looking like the
> platform is still alive. Or that could all be wishful thinking and they'll all
> migrate to Jira or something equally horrible, who knows.

Well, Red Hat apparently is already in the process of migrating to Jira
in the long run:
https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/thread/U7TZRWXVUGBCHS6EBJIBSFAVPFUHHV7J/

To quote that mail from March:

```
As some of you may know, Red Hat has been using both Bugzilla and Jira
via issues.redhat.com for RHEL development for several years.  Our
intention is to move to using issues.redhat.com only for the major
RHEL releases after RHEL 9.
```

Ciao, Thorsten

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Hans de Goede]

Hi,

On 10/5/22 11:00, Thorsten Leemhuis wrote:
> On 04.10.22 22:25, Konstantin Ryabitsev wrote:
>> On Tue, Oct 04, 2022 at 10:06:28PM +0200, Thorsten Leemhuis wrote:
>>> Your plan would afaics mean that we invest further into a software
>>> abandoned by its upstream and already becoming more and more of a
>>> maintenance burden. That investment would also further increase our
>>> dependency on that software by establishing workflows that rely on it.
>>> Is that really wise at this point? Wouldn't it be better to spend that
>>> time and effort to build something better that is more future proof?
>>
>> Unfortunately, there's no such thing. ;) And maybe we'll even help tip the
>> course of history into the other direction -- Red Hat uses bugzilla, and so
>> does OpenSuse, so there's a pretty good core of well-funded companies that
>> would be in a position to help keep bugzilla going if it's looking like the
>> platform is still alive. Or that could all be wishful thinking and they'll all
>> migrate to Jira or something equally horrible, who knows.
> 
> Well, Red Hat apparently is already in the process of migrating to Jira
> in the long run:
> https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/thread/U7TZRWXVUGBCHS6EBJIBSFAVPFUHHV7J/
> 
> To quote that mail from March:
> 
> ```
> As some of you may know, Red Hat has been using both Bugzilla and Jira
> via issues.redhat.com for RHEL development for several years.  Our
> intention is to move to using issues.redhat.com only for the major
> RHEL releases after RHEL 9.
> ```

That is for RHEL only though I'm not sure what the plans for Fedora are.

Also I do believe that the Red Hat bugzilla team is working on porting
bugzilla to postgresql, which would at least fix the problem of depending on
a no longer maintained mysql version.

If the postgresql port is something of interest to keep bugzilla.kernel.org
going for now, then it is probably best to just directly contact the bugzilla
maintainers @redhat.

Regards,

Hans

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/4/22 17:53, Konstantin Ryabitsev wrote:
> On Thu, Sep 29, 2022 at 01:19:24PM +0200, Thorsten Leemhuis wrote:
>> Hi!
>>
>> TLDR: Core Linux kernel developers are unhappy with the state of
>> bugzilla.kernel.org; to improve things I plan to change a few important
>> aspects of its configuration, unless somebody comes up with better ideas
>> to tackle current problems: (1) Create a catch-all product making it
>> totally obvious to submitters that likely nobody will look into the
>> ticket. (2) Remove or hide all products & components where the subsystem
>> didn't fully commit to look into newly submitted reports. (3) Change the
>> text on the front page to make it clear that most kernel bug reports
>> need to be sent by mail.
>
> Here's my counter-plan, which builds on top of yours.
>
> 1. Create a Kernel/Kernel product that acts as a starting point for all bug
>     submissions.
> 2. Create and maintain a mapping from MAINTAINER subsystem entries to
>     Product/Component categories in Bugzilla (the scheme to be established).
> 3. Establish and maintain a team of designated triage people who are willing
>     to look at incoming bugs to either:
>
>     a. quick-close them as non-actionable (tainted kernel, distro kernel, spam)
>     b. obtain missing information from the submitter as necessary
>     c. figure out the correct component to assign, to the best of their ability
>     d. set a "triaged" flag
>
> 4. a backend monitoring bot will track all bug changes and, when it sees a bug
>     get the "triaged" state, it will:
>
>     a. create a useful bug summary from all bug comments
>     b. figure out who to notify based on the mapping (see #2 above)
>     c. send out the email to everyone identified
>
> 5. the same backend monitoring bot will track responses and update the bug
>     comments as needed; any comments added via the bugzilla site will be
>     similarly sent out as follow-up messages.
>
> 6. the bot can also monitor commits and other discussions via lore.kernel.org
>     and automatically add comments/links when it sees the bug mentioned
>     elsewhere.
>
> I'm happy to take care of everything bot-related (apparently, programming bots
> is what I do now -- I just wish it was the cool and glamorous kind).
>
> As I have stated multiple times, the hard part will be keeping a team of
> people who are willing to do the bug triage work, but maybe we can start with
> Greg KH using his intern funds to hire someone (assuming he's not already
> using these funds for someone to help him with all the other tasks).
>
> Does that sound like a plan for everyone?

This looks fabulous.

Except, I'd love to let users have an option of submitting bugs to the
component they specify manually.

(I've removed a ton of people from CC because my email provider
absolutely hates when I send emails to many addresseses simultaneously -
my account was completely blocked for five days while I tried hard to
reach support).

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Artem S. Tashkinov]

On 10/3/22 14:22, Konstantin Ryabitsev wrote:
> On Mon, Oct 03, 2022 at 09:16:06AM +0000, Artem S. Tashkinov wrote:
>> The initial conversation started with the fact that Bugzilla is old,
>> semi-deprecated, requires MySQL [no idea what's bad about it, Bugzilla
>> can work with MariaDB and Percona as well]
>
> It can't, actually. It only works with MySQL 5.7 or an equally ancient MariaDB.
> No, there is no official fix (because nobody is working on bugzilla).
> https://bugzilla.mozilla.org/show_bug.cgi?id=1592129
>

(I wanted to send this message two days ago but my email account was
suspended at the time - I'm still sending it though it feels like it's
no longer relevant).

Speaking of Bugzilla Harmony:

* KDE already uses it:
https://invent.kde.org/websites/bugs-kde-org/-/merge_requests/1
* Mozilla has been using it for quite time already
* It has a docker container/image, so it's a breeze to install

I've dealt with a dozen other bug trackers and to me Bugzilla remains
the most powerful and featureful, while not being too complicated to set
up and use.

Lastly I've had a crazy idea for a second of maybe migrating the kernel
bugzilla over to RedHat/IBM (by asking them whether they are willing to
help) but on a second thought it's really really bad as the company is
very large and there's a ton of bureaucracy, so managing it would become
quite a hassle. Also, I wouldn't want the LF to hand control over it to
RedHat.

Best regards,
Artem

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Greg KH]

On Thu, Oct 06, 2022 at 10:46:29AM +0000, Artem S. Tashkinov wrote:
> Lastly I've had a crazy idea for a second of maybe migrating the kernel
> bugzilla over to RedHat/IBM (by asking them whether they are willing to
> help) but on a second thought it's really really bad as the company is
> very large and there's a ton of bureaucracy, so managing it would become
> quite a hassle. Also, I wouldn't want the LF to hand control over it to
> RedHat.

bugzilla.kernel.org was originally created by IBM way back in the 2.5
kernel days (by people at IBM at the time), and then the passed it off
to the kernel.org admins.  I doubt they want it back :)

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 05.10.22 11:57, Hans de Goede wrote:
> On 10/5/22 11:00, Thorsten Leemhuis wrote:
>> On 04.10.22 22:25, Konstantin Ryabitsev wrote:
>>> On Tue, Oct 04, 2022 at 10:06:28PM +0200, Thorsten Leemhuis wrote:
>>>> Your plan would afaics mean that we invest further into a software
>>>> abandoned by its upstream and already becoming more and more of a
>>>> maintenance burden. That investment would also further increase our
>>>> dependency on that software by establishing workflows that rely on it.
>>>> Is that really wise at this point? Wouldn't it be better to spend that
>>>> time and effort to build something better that is more future proof?
>>>
>>> Unfortunately, there's no such thing. ;) And maybe we'll even help tip the
>>> course of history into the other direction -- Red Hat uses bugzilla, and so
>>> does OpenSuse, so there's a pretty good core of well-funded companies that
>>> would be in a position to help keep bugzilla going if it's looking like the
>>> platform is still alive. Or that could all be wishful thinking and they'll all
>>> migrate to Jira or something equally horrible, who knows.
>>
>> Well, Red Hat apparently is already in the process of migrating to Jira
>> in the long run:
>> https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/thread/U7TZRWXVUGBCHS6EBJIBSFAVPFUHHV7J/
>>
>> To quote that mail from March:
>>
>> ```
>> As some of you may know, Red Hat has been using both Bugzilla and Jira
>> via issues.redhat.com for RHEL development for several years.  Our
>> intention is to move to using issues.redhat.com only for the major
>> RHEL releases after RHEL 9.
>> ```
> 
> That is for RHEL only though I'm not sure what the plans for Fedora are.

FWIW & for completeness: at least Fedora's project leader Matthew Miller
thinks that Fedora ```should plan to migrate [from bugzilla] to
something else in the next three years. There are two reasons for this,
and only one of them is that Red Hat is moving away. The second is that
Bugzilla isn’t a great tool for what we need anyway.

On the first part: yes, there’s a long, slow sunset, but I think
realistically we’re going to see business-side attention drop
significantly, and we’ll have correspondingly worse and worse service.
Right now, there are basically only two people keeping it all going,
which is heroic. I don’t think it’s too much
pulling-back-the-corporate-curtain to guess that if one or both get
tired of that and decide to go start a yak farm, Red Hat won’t
prioritize hiring backfill dedicated in the same way. We could ask CPE
to keep it going, but… there’s so much more I’d like to ask CPE. (And
non-CPE volunteers? There’s so much that’s more interesting!)

So to the second part: Bugzilla isn’t what we really need anyway.[...]```

See his post from March here for links and the rest of his msg:
https://discussion.fedoraproject.org/t/the-future-of-bugzilla-in-fedora/37735
I quoted the msg fully at the end of this mail for completeness.

Only learned about this msg now through a comment from Matthew
(https://lwn.net/Articles/910966/ ) in an article about this thread
(https://lwn.net/Articles/910740/ ). To quote from his lwn.net comment:

```
Tl;dr: many similar issues as in the above conversation. Triage is a big
deal. "Bugs" aren't really the best place to start -- it's better to
send people to a help forum first, file bugs second. [...]
```

Ciao, Thorsten

P.S.: Here is the promised full quote of:
https://discussion.fedoraproject.org/t/the-future-of-bugzilla-in-fedora/37735

```
> The Future of Bugzilla in Fedora
> Project Discussion
> council
> engineering
> quality-assurance
> Matthew Miller
> mattdm Fedora Council Member
> Mar 22
> 
> I posted this on the Fedora Development mailing list 3, in the thread
> RHEL moving to issues.redhat.com only long term 2, and am re-posting
> here for broader discussion.
> 
> Here’s what I’m thinking: we should plan to migrate to something else
> in the next three years. There are two reasons for this, and only one
> of them is that Red Hat is moving away. The second is that Bugzilla
> isn’t a great tool for what we need anyway.
> 
> On the first part: yes, there’s a long, slow sunset, but I think
> realistically we’re going to see business-side attention drop
> significantly, and we’ll have correspondingly worse and worse
> service. Right now, there are basically only two people keeping it
> all going, which is heroic. I don’t think it’s too much
> pulling-back-the-corporate-curtain to guess that if one or both get
> tired of that and decide to go start a yak farm, Red Hat won’t
> prioritize hiring backfill dedicated in the same way. We could ask
> CPE to keep it going, but… there’s so much more I’d like to ask CPE.
> (And non-CPE volunteers? There’s so much that’s more interesting!)
> 
> So to the second part: Bugzilla isn’t what we really need anyway.
> 
> I’m not saying Bugzilla has been terrible — it’s served us well, in
> fact. And I have personal fondness for it that, which I do not for,
> say, the wiki. :classic_smiley: Buzilla is it’s deeply integrated in
> a lot of our processes, and we’ve got a lot of automations and so on.
> It’s important. But… there’s a lot that could be better. I think
> specifically:
> 
> It doesn’t serve as a single place to track everything. We’ve got
> stuff scattered around Bugzilla; issue trackers in Pagure GitLab, and
> GitHub; pull requests in all of those places; and more. Not to
> mention upstreams (and inconsistent approaches to tracking upstream
> issues). I’m not arguing that we need ALL things in one place, but
> it’s important that Bugzilla isn’t that now anyway.
> 
> Bugzilla a terrible experience for end users. Usually it’s the Wrong
> Place. When a user has a problem, that may or may not be caused by a
> software defect. This is a whole topic of its own, but in short, it’s
> really better for most users to report problems at Ask Fedora, and
> then possibly after triage a bug should be filed and tracked
> somewhere.
> 
> Many of our users are advanced and recognize real defects and file
> good reports, but this leads to even more frustration, because our
> response is inconsistent. Maybe that report should actually be
> upstream. Maybe it’s in a dependency package that’s really only
> loosely maintained. For whatever reason, a lot of perfectly good
> reports end up closed EOL, which is never a good outcome.
> 
> We’re inconsistent with PRs vs issues, which is confusing and makes
> more duplication. I have seen cases where a packager complained that
> someone filed a PR that they never noticed, saying “this should be a
> bug so I’ll see it”, while others close bugs with “please send a PR”.
> We could make stronger statements about policies, but as long as
> these two things exist in separate places, that tension will keep
> coming back.
> 
> Plus plenty of minor things: Our workflow still is shoehorned around
> a bunch of RH-centric stuff (lots of fields, flags, and statuses that
> we don’t really use or want). It’s not great for the review workflow,
> or for some of the other things we’ve twisted it to. A
> component-centric approach makes it hard to track larger issues. The
> SCM workflow is … not good.
> 
> And I’m sure there’s more. But I’m not really here to complain about
> Bugzilla. It’s just actually not filling our needs. I therefore think
> that Jira / issues.redhat.com 5 wouldn’t either — although it’s got a
> ton of features on top, it’s fundamentally the same kind of thing.
> 
> We need define exactly what we do need, and figure out how to get
> that, in a sustainable way going forward. And fortunately, we DO have
> a few years, so for once we could do this before it’s a crisis.
> 
> I think we should create a project to figure this out. In fact, I
> think it should be a Fedora Objective. But, it also shouldn’t be a
> completely blue-sky initiative — we should avoid trying to develop a
> new gigantic piece of software that we own. (See past results on
> that, sigh.)
> 
> We do have some pieces already in place that should be part of the
> foundation (or at least other metaphorical bricks in the
> construction):
> 
> Ask Fedora is the place for users to report and discuss problems.
> This is our first-line of support, and it’s where we can do triage.
> Then, software defects may or may not be tracked relating to those
> conversations.
> 
> Package-specific issues should be next to the PRs. Let’s enable issue
> tracking on src.fedoraproject.org (with whatever gitforge we end up
> with that on). I think this makes sense for initial package review
> too, although that would need some workflow changes.
> 
> Bodhi and the CI results system and all of that. This is
> update-centric, but does also have a lot of issue-tracker-like
> characteristcs in the “karma” system, and it is inherently close to
> our release process. Maybe some of the process that currently goes
> through Bugzilla could move here.
> 
> The Fedora account system and Fedora message bus, plus the packager
> dashboard, the (under significant update!) Fedora notifications
> system, and so on. We have a nice underlying infrastructure for tying
> things together, and we can do more integrations. (Look at how Copr
> uses Discourse, for example. Or the Blocker Bugs app and its
> connections with Pagure and, um, Bugzilla.)
> 
> Obviously that’s not nearly sufficient. But we should look at all of
> our needs around tracking issues, plans, projects, blockers, and
> defects; consider different packaging cases with various
> relationships with upstreams; and imagine ideal workflows for users,
> packagers, project managers, spin and edition teams, QA, and all the
> other stakeholders. The last time we did this was in 2013, so… it’s a
> good 10-years-later exercise! (Infrastructure Fedora bug tracker -
> Fedora Project Wiki 2)
> 
> Then, once we have a good shared understanding of what we want, start
> fitting pieces like the above into that picture, define the gaps, and
> then find exactly what we need to fill them.
```

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Thorsten Leemhuis]

On 04.10.22 19:53, Konstantin Ryabitsev wrote:
> On Thu, Sep 29, 2022 at 01:19:24PM +0200, Thorsten Leemhuis wrote:
>> TLDR: Core Linux kernel developers are unhappy with the state of
>> bugzilla.kernel.org; to improve things I plan to change a few important
>> aspects of its configuration, unless somebody comes up with better ideas
>> to tackle current problems: (1) Create a catch-all product making it
>> totally obvious to submitters that likely nobody will look into the
>> ticket. (2) Remove or hide all products & components where the subsystem
>> didn't fully commit to look into newly submitted reports. (3) Change the
>> text on the front page to make it clear that most kernel bug reports
>> need to be sent by mail.

So, merge window is over. To avoid any doubt, I'd now like to get a
clarification what the outcome of this discussion actually is.

FWIW, as most of you likely know, lwn.net has a write-up of this thread:
https://lwn.net/Articles/910740/

> Here's my counter-plan, which builds on top of yours.

Is this the agreed on path forward by silent agreement? And if so: who
will actually shepherd this? I just wonder, as it sounded to me that
Konstantin would be happy to take care of the bot-related stuff, but
leave the rest to somebody else.

Or do we have two proposals on the table that are kind of deadlocked so
that nothing will happen until the next maintainers summit, where things
like this are usually discussed and a way forward agreed on? Then the
ugly situation with bugzilla.kernel.org would continue for afaics at
least 11 more months, which I'd call "unfortunate". :-/

Ciao, Thorsten

> 1. Create a Kernel/Kernel product that acts as a starting point for all bug
>    submissions.
> 2. Create and maintain a mapping from MAINTAINER subsystem entries to
>    Product/Component categories in Bugzilla (the scheme to be established).
> 3. Establish and maintain a team of designated triage people who are willing
>    to look at incoming bugs to either:
> 
>    a. quick-close them as non-actionable (tainted kernel, distro kernel, spam)
>    b. obtain missing information from the submitter as necessary
>    c. figure out the correct component to assign, to the best of their ability
>    d. set a "triaged" flag
> 
> 4. a backend monitoring bot will track all bug changes and, when it sees a bug
>    get the "triaged" state, it will:
> 
>    a. create a useful bug summary from all bug comments
>    b. figure out who to notify based on the mapping (see #2 above)
>    c. send out the email to everyone identified
> 
> 5. the same backend monitoring bot will track responses and update the bug
>    comments as needed; any comments added via the bugzilla site will be
>    similarly sent out as follow-up messages.
> 
> 6. the bot can also monitor commits and other discussions via lore.kernel.org
>    and automatically add comments/links when it sees the bug mentioned
>    elsewhere.
> 
> I'm happy to take care of everything bot-related (apparently, programming bots
> is what I do now -- I just wish it was the cool and glamorous kind).
> 
> As I have stated multiple times, the hard part will be keeping a team of
> people who are willing to do the bug triage work, but maybe we can start with
> Greg KH using his intern funds to hire someone (assuming he's not already
> using these funds for someone to help him with all the other tasks).
> 
> Does that sound like a plan for everyone?
> 
> -K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Konstantin Ryabitsev]

On Mon, Oct 17, 2022 at 03:57:17PM +0200, Thorsten Leemhuis wrote:
> > Here's my counter-plan, which builds on top of yours.
> 
> Is this the agreed on path forward by silent agreement? And if so: who
> will actually shepherd this? I just wonder, as it sounded to me that
> Konstantin would be happy to take care of the bot-related stuff, but
> leave the rest to somebody else.

Indeed, I need to do most of the preliminary legwork. I will start by creating
a public-inbox feed of all bugzilla comments, which is something I was
planning to do for a while anyway. Once that is in place, I can build on top
of that to add a two-way bridge to replace bugzilla's native (and rather
limited, in my experience) email bridge implementation.

Once I have this in place, we can consider what next steps should be taken.

-K

Re: Planned changes for bugzilla.kernel.org to reduce the "Bugzilla blues"

[Rafael J. Wysocki]

On Monday, October 17, 2022 10:47:31 PM CEST Konstantin Ryabitsev wrote:
> On Mon, Oct 17, 2022 at 03:57:17PM +0200, Thorsten Leemhuis wrote:
> > > Here's my counter-plan, which builds on top of yours.
> > 
> > Is this the agreed on path forward by silent agreement? And if so: who
> > will actually shepherd this? I just wonder, as it sounded to me that
> > Konstantin would be happy to take care of the bot-related stuff, but
> > leave the rest to somebody else.
> 
> Indeed, I need to do most of the preliminary legwork. I will start by creating
> a public-inbox feed of all bugzilla comments, which is something I was
> planning to do for a while anyway. Once that is in place, I can build on top
> of that to add a two-way bridge to replace bugzilla's native (and rather
> limited, in my experience) email bridge implementation.
> 
> Once I have this in place, we can consider what next steps should be taken.

Sounds good to me.

Cheers!