Re: Linux 2.6.21

Re: Linux 2.6.21

[Tomasz Chmielewski]

Linus Torvalds wrote:

> On Sun, 29 Apr 2007, Adrian Bunk wrote:
>> 
>> The kernel Bugzilla currently contains 1600 open bugs.
> 
> Adrian, why do you keep harping on this, and ignoring reality?
> 
> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> 
> How many of those are interesting and valid? How many of them are 
> relevant? How many of them are duplicates?

And - how many of these bug reports have kernel's bugzilla ever 
forwarded to lkml so that other people could see them?

Is that number zero (because kernel's bugzilla is configured this way)?


-- 
Tomasz Chmielewski

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 11:04:10PM +0200, Tomasz Chmielewski wrote:
> Linus Torvalds wrote:
>
>> On Sun, 29 Apr 2007, Adrian Bunk wrote:
>>> The kernel Bugzilla currently contains 1600 open bugs.
>> Adrian, why do you keep harping on this, and ignoring reality?
>> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
>> How many of those are interesting and valid? How many of them are 
>> relevant? How many of them are duplicates?
>
> And - how many of these bug reports have kernel's bugzilla ever forwarded 
> to lkml so that other people could see them?
>
> Is that number zero (because kernel's bugzilla is configured this way)?

Andrew forwards incoming Bugzilla bugs to the responsible maintainers 
(if there are any).

If it is considered useful it shouldn't be a problem to automatically 
forward all incoming Bugzilla bugs to linux-kernel.

> Tomasz Chmielewski

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Bernd Eckenfels]

In article <20070430030213.GB23336@stusta.de> you wrote:
> If it is considered useful it shouldn't be a problem to automatically 
> forward all incoming Bugzilla bugs to linux-kernel.

Yes, most of it to linux-kernel, some components (netdev@, architecture) to
a more specific list.

Gruss
Bernd

Re: Linux 2.6.21

[Tomasz Chmielewski]

Adrian Bunk schrieb:
> On Sun, Apr 29, 2007 at 11:04:10PM +0200, Tomasz Chmielewski wrote:
>> Linus Torvalds wrote:
>>
>>> On Sun, 29 Apr 2007, Adrian Bunk wrote:
>>>> The kernel Bugzilla currently contains 1600 open bugs.
>>> Adrian, why do you keep harping on this, and ignoring reality?
>>> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
>>> How many of those are interesting and valid? How many of them are 
>>> relevant? How many of them are duplicates?
>> And - how many of these bug reports have kernel's bugzilla ever forwarded 
>> to lkml so that other people could see them?
>>
>> Is that number zero (because kernel's bugzilla is configured this way)?
> 
> Andrew forwards incoming Bugzilla bugs to the responsible maintainers 
> (if there are any).
> 
> If it is considered useful it shouldn't be a problem to automatically 
> forward all incoming Bugzilla bugs to linux-kernel.

Why isn't it done yet? If the bugs were already forwarded to 
linux-kernel (and perhaps, to linux-<subsystem> when possible, too), we 
would save at least two days of this long "Linux 2.6.21" thread...

For I somehow feel that most people here dislike bugzilla because of 
misconceptions - which only arose as bugzilla.kernel.org is *really* 
misconfigured.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[Andrew Morton]

(various cc's reestablished.  Please don't remove cc's when dealing with
kernel people).

On Mon, 30 Apr 2007 06:57:08 +0200 Bernd Eckenfels <ecki@lina.inka.de> wrote:

> In article <20070430030213.GB23336@stusta.de> you wrote:
> > If it is considered useful it shouldn't be a problem to automatically 
> > forward all incoming Bugzilla bugs to linux-kernel.
> 
> Yes, most of it to linux-kernel, some components (netdev@, architecture) to
> a more specific list.
> 

I think it would be a net good thing to send bugme-new notifications to lkml,
actually.  There are only maybe 5-10 per day (mad guess).

Re: Linux 2.6.21

[Gene Heskett]

On Monday 30 April 2007, Tomasz Chmielewski wrote:
>For I somehow feel that most people here dislike bugzilla because of
>misconceptions - which only arose as bugzilla.kernel.org is *really*
>misconfigured.

Bugzilla was indeed miss-conceived.  It shoulda been on birth control pills.

I'm not claiming that we don't need something, but bugzilla sure as heck isn't 
it.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Where's the Coke machine?  Tell me a joke!!

Re: Linux 2.6.21

[Lennart Sorensen]

On Sun, Apr 29, 2007 at 09:40:07PM +0200, Diego Calleja wrote:
> So far, it seems that most of people's opinion WRT to bug reporting and trackingcan
> be divided into 2 groups:
> 
> - People who argues (and they're right) that bugzilla and web interfaces in general
>   suck and that email + an "Adrian-like" solution works better
> 
> - People who argues that a bug tracker better than a mailing list is absolutely
>   needed (and they're right). They also argue that while bugzilla sucks, it's
>   better than nothing.
> 
> There's a common point between both groups: bugzilla sucks. The ideal
> solution would be to replace bugzilla with some alternative and better
> opensource bug tracking software, but I doubt it exists (there must be a
> reason why everybody uses bugzilla). A good bug tracker should feel like
> it makes your work easier, instead of making you feel like you're wasting
> time (which is what bugzilla does)

Debian has a bug track system which interacts mainly with the users
through email.  Seems rather nice to use and doesn't make you sign up to
submit things, and has no issues with mailing lists being "subscribed"
to a bug.  Probably a bit complicated to setup though.

> I don't see why a web interface bug tracker should be bad for bug tracking,
> as long as it's good and integrates 100% in the mailing lists. In my humble
> opinion the "perfect" bug tracker for Linux should be something like this:
> - Has a email interface (like the Debian bug tracking database).
> - Has a web interface that completely follows the email threads
>    (reading/posting), but make the comments real emails, not just
>     database fields.
> If done well (unlike the current bugzilla-to-email hack), it should possible
> to do many nice things, like add a lkml bug report to the bug tracking
> database (which shouldn't be a "real" database, but just an lkml mail
> archive with a list of message IDs that are considered a bug and its state)
> by just replying the thread, CCing the bug tracker and telling him to include
> the thread in the database.

So in other words, basically the debian bug track system, except
perhaps with an ability to submit bugs through a web interface too
rather than just email and reportbug (which I believe uses email).

> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

--
Len Sorensen

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Monday, 30 April 2007 08:30, Andrew Morton wrote:
> On Mon, 30 Apr 2007 00:09:06 +0200 "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > On Sunday, 29 April 2007 22:52, Alexey Dobriyan wrote:
> > > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > > of the relevant message, so why to require the reporter to file the report with
> > > > the bugzilla himself?]
> > > 
> > > Last time I did this, bugzilla at osdl.org won't let me add original
> > > reporter to goddamn CC list. It would be el neat, because not everyone
> > > followed instructions and forwarding emails between reporter and
> > > bugzilla sucks.
> > 
> > That's related to what I said before.  The requirement that the addresses on
> > the CC list must be 'known' to bugzilla is deadly wrong in every case I can
> > imagine.
> 
> But unknown-to-bugzilla email addresses are accepted when they're sent to
> bugme-daemon@kernel-bugs.osdl.org.  This is why I'll very often switch a
> bug report to email, copying individuals and mailing lists and
> bugme-daemon.  Then bugzilla just sits silently in the background recording
> everything.

If I wanted to do this, what would I have to do?  I mean, assume I have a bug
report that I want to send to someone whom the bugzilla doesn't like.
What's the right procedure in such a case?

> But once a bug has switched to email, it needs to stay there - it would be
> bad if someone were to update the bug via the web UI because none of the
> emailed participants would know of the update.  So i'll often explicitly
> ask "please follow up via emailed reply-to-all".
> 
> It's not great, but there's certainly enough material here for people to get
> in and work on the bug, should they be so inclined.
> 
> My overall approach with this stuff is: short-term bugs are handled via email
> and long-term ones are tracked in buzilla.

I think that's reasonable, I've always done it this way, AFAIR.

> Hence someone (Hi, Mom) needs to track all the emailed-only bug reports and get
> them filed in bugzilla once they go stale.

I can do that with suspend/hibernation-related bug reports, but sometimes
I'm not sure who is the right person to notify in the first place.

Re: Linux 2.6.21

[Vegard Nossum]

On Mon, April 30, 2007 8:57 pm, Adrian Bunk wrote:
> I never expected the reality to be come as white as my ideal or the
> washed things in washing powder ads.

This reminds me very much of what the brilliant computing scientist Edsger
W. Dijkstra more than once wrote:

`Confusing "love of perfection" with "claim of perfection", people will
accuse you of the latter and then blame you for the first.' (EWD709)

Also relevant to the discussion, I think, but in another way, is this:

`One of them is the dogma that striving for perfection is
counterproductive in the sense that it would make software development
much too expensive. But what are the main causes of the soaring costs of
software development? A major cost, in terms of both manpower and
unforeseen delays, is debugging, and one can save a lot by investing more
in preventing the bugs from entering the design in the first place. Since
the errors are so expensive, in general the high-quality design is also by
far the cheaper. Another major cause is that many systems are built on
shifting foundations in the sense that the underlying software of
operating systems and compilers is too shaky to be stable, with the result
that each new release of that underlying software requires possibly
extensive adaptation of what has been built on top of it. Finally, many of
the tools the programmer is supposed to work with are so poorly documented
that they force him to find out by experiment what they might be able to
do for him. Since these experiments can be pretty expensive and
time-consuming and —inductive reasoning being what it is— an educated
guess is the best the poor programmer can hope for, the poor programmer is
really in a miserable position. So here you see three major sources of
cost explosion traced down to someone's assumption that striving for
perfection is counterproductive!' (EWD952)

For the complete documents, see
http://www.cs.utexas.edu/users/EWD/transcriptions/EWD07xx/EWD709.html and
http://www.cs.utexas.edu/users/EWD/transcriptions/EWD09xx/EWD952.html,
respectively.


Regards from Vegard's

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 11:20:46AM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 30 Apr 2007, Adrian Bunk wrote:
> > 
> > My ideal was always that reported bugs should be fixed.
> 
> ..and this is where we differ.
> 
> OF COURSE bugs should be fixed. But you seem to think that there is 
> something magical and special about every single bug-report.
> 
> You have a new home assignment: watch the "every sperm is sacred" thing 
> from Monty Python's "Meaning of Life". Google for it.

I like the Flying Circus and the other Monty Python films (including 
"The Crimson Permanent Assurance"), but "Meaning of Life" didn't impress 
me. But I have the song somewhere if this counts.  ;-)

> And if you cannot appreciate the absurdity and humor of that thing, maybe 
> you should think about it a bit more.
> 
> And once you _can_ appreciate the humor of that song/skit, look yourself 
> in the mirror, and ask yourself: "is every bug report sacred?"
> 
> Really?
> 
> > If you accept that this is anyway impossible because more bugs get added 
> > than could get fixed you might not need any tracking at all.
> 
> That's a TOTALLY IDIOTIC argument.
> 
> That goes from "every sperm is sacred" to "sperm doesn't count at all".
> 
> Can you not see how stupid that statement of yours really is? Can you not 
> see that anybody who thinks in those kinds of black-and-white terms is 
> simply not FUNCTIONAL!
> 
> Bugs are neither sacred, _nor_ should they be ignored. 
> 
> Ponder that, grasshopper. And until you can see that things are not 
> "either-or", "black-and-white", "all or nothing", I don't think I really 
> can have anything worthwhile to add in this discussion to you. People who 
> think in absolutes are simply not worth talking to.

I never expected the reality to be come as white as my ideal or the 
washed things in washing powder ads.

My ideal was white, and the shade of grey of the current reality is 
darker than I think it should be.

At least theoretically reachable are things like:
- every incoming bug report is quickly handled by one or more
  kernel developers who know the drivers and subsystems involved
- there's a last -rc kernel published for a few days of testing,
  and except for the Makefile change the -final is identically to it
  (or a new -rc published)

There are sometimes nonsensical bug reports and "handling" could be 
rejecting a bug e.g. due to a tainted kernel.

And sometimes mysterious bugs are more or less undebuggable.

But these two points would have resulted in 2.6.21 being released more 
or less at the same time, but with a dozen regressions less.

> 			Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1023 bytes --]



On Mon, 30 Apr 2007, Linus Torvalds wrote:

> Ponder that, grasshopper. And until you can see that things are not 
> "either-or", "black-and-white", "all or nothing", I don't think I really 
> can have anything worthwhile to add in this discussion to you. People who 
> think in absolutes are simply not worth talking to.

This is another case of "perfect is the enemy of good".

Tryng to reach perfect is not only guaranteed to fail, but trying to reach 
it AND NOT REALIZING that it's stupid and wrong is actually much WORSE 
than just trying to do a reasonable job.

And if you put some _totally_idiotic_ expectation that all bugreports can 
be fixed, and should always be totally blocking, that's guaranteed to just 
cause a totally unusuable bug reporting system.

And your bugzilla arguments seem to be exactly that. A naïve and totally 
unrealistic expectation of "every bugreport is sacred" is BAD.

In other words: perfection not only isn't even possible, BUT IT IS NOT 
EVEN WORTH TRYING TO REACH FOR!

			Linus

Re: Linux 2.6.21

[Linus Torvalds]

On Mon, 30 Apr 2007, Adrian Bunk wrote:
> 
> My ideal was always that reported bugs should be fixed.

..and this is where we differ.

OF COURSE bugs should be fixed. But you seem to think that there is 
something magical and special about every single bug-report.

You have a new home assignment: watch the "every sperm is sacred" thing 
from Monty Python's "Meaning of Life". Google for it.

And if you cannot appreciate the absurdity and humor of that thing, maybe 
you should think about it a bit more.

And once you _can_ appreciate the humor of that song/skit, look yourself 
in the mirror, and ask yourself: "is every bug report sacred?"

Really?

> If you accept that this is anyway impossible because more bugs get added 
> than could get fixed you might not need any tracking at all.

That's a TOTALLY IDIOTIC argument.

That goes from "every sperm is sacred" to "sperm doesn't count at all".

Can you not see how stupid that statement of yours really is? Can you not 
see that anybody who thinks in those kinds of black-and-white terms is 
simply not FUNCTIONAL!

Bugs are neither sacred, _nor_ should they be ignored. 

Ponder that, grasshopper. And until you can see that things are not 
"either-or", "black-and-white", "all or nothing", I don't think I really 
can have anything worthwhile to add in this discussion to you. People who 
think in absolutes are simply not worth talking to.

			Linus

Re: Linux 2.6.21

[Borislav Petkov]

On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> On Thu, Apr 26, 2007 at 02:13:08PM -0700, Linus Torvalds wrote:
> >...
> > (I've said this before, but I'll say it again: one thing that would 
> > already make bugzilla better is to just always drop any bug reports that 
> > are more than a week old and haven't been touched. It wouldn't need *much* 
> > touching, but if a reporter cannot be bothered to say "still true with 
> > current snapshot" once a week, then it shouldn't be seen as being somehow 
> > up to those scare resources we call "developers" to have to go through 
> > it).
> >...
> 
> And if it's a bug in an unmaintained subsystem, a user could do this for 
> 100 weeks without any effect.
> There's no value in keeping reporters busy with useless tasks. [1]
> 
> Don't forget:
> A good bug report is an important contribution.
> 
> We are already quite good at ignoring bug reports that come through 
> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> than 1600 open bugs because this tells how bad we are at handling bugs.
> How many thousand bug reports have been ignored during the same time on 
> linux-kernel?

... and it gets better: I've been trying to fix/improve stuff I can, with
my limited abilities as coder for some time now in the linux kernel, but there
are simply cases where, although you're trying to even come up with a proper fix
and adhere to all the kernel coding standards and _even_ produce a patch which
can serve as rough cut for a probable fix, your mail simply disappears into the
void unanswered. Then you send again, and again, yet it remains unreplied and then you
give up and turn to something else. Some of those patches, for example, were the
rework of the debugging scheme of libata i did more than a year ago which got
reviewed and then .. forgotten, some kernel janitor cleanups, etc...

-- 
Regards/Gruß,
    Boris.

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 12:01:38AM -0700, David Miller wrote:
>...
> If bugs should be reported to the mailing list, then they
> should just get rid of bugzilla because it's aparently
> serving as a garbage bin.

The first question is not "Bugzilla" but "Does bug tracking make sense?".

Many bug reports to linux-kernel get zero attention and are lost without 
tracking. Is losing bug reports OK or do we need a tracking of bugs?

Tracking means:
- not to miss bug reports
- record discussions and the status of the bugs
- the real value comes from:
  provide useful reports like "all open ACPI bugs" or "all 2.6.21-rc 
  regressions" to people who are actually working on fixing the bugs

I was tracking 2.6.21-rc regressions.
Some people said this was useful.

Manually tracking of three dozen regressions plus sending regular sorted 
reports was at the limit of what I am able to handle manually without 
any tracking tool.

There are two different questions that seem to often be mixed in 
discussions:

The first question is:
Does it make sense to track kernel bugs?

If no, there's no need to discuss Bugzilla or other tools.

If yes, the second question is:
Which tracking tool would make sense for the Linux kernel?
This could be Bugzilla, the Debian bug tracking system, or even 
something written from scratch.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 02:24:03PM -0700, Linus Torvalds wrote:
> 
> 
> On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > > 
> > > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> > 
> > OK, how do you suggest to track bugs in a way that doesn't suck?
> 
> I've tried to explain.
> 
> Bugzilla can be one _part_ of it, but anybody who thinks it's the "main 
> part" is really not being realistic. It's too cumbersome, and it's too 
> stupid. 

I do completely agree with you on this.

The main parts are people doing some sorting and forwarding of an 
incoming bug (currently mostly Andrew) and someone with deeper subsystem 
knowledge looking deeper into a bug (currently often missing).

> Quite frankly, "lkml + google" is probably in many ways a *better* way to 
> search for problems. But yes, some manual smarts (and the _occasional_ 
> pointer to bugzilla) is probably currently the only option.
> 
> Exactly because I don't think anybody has shown any better automation than 
> bugzilla. But that doesn't make bugzilla "the One Choice". That's not how 
> it works. If there is no automation, manual tracking is still better than 
> *crap* automation.

I had the regressions stored in a plain textfile.

For getting regressions reasonably grouped for my regression emails, I 
used paper, pen and scissors - and this is not a joke.

That really didn't scale when we had 36 regressions.

So some tool is needed if the bug numbers are bigger - no matter whether 
it's Bugzilla or speaking plain SQL to a database, or anything else.

> > Bug reports to linux-kernel have the big problem that they are lost if 
> > no developer immediately picks them up.
> 
> ..and this is different from bugzilla exactly _how_?
> 
> Those things are lost too. As you yourself have pointed out. The fact that 
> you can search for them is _exactly_ as relevant as the fact that you can 
> search for lkml on google.

It depends on how you look at bugs.

My ideal was always that reported bugs should be fixed.

If you accept that this is anyway impossible because more bugs get added 
than could get fixed you might not need any tracking at all.

> > What I do know is that the majority of them has never been proper 
> > debugged by a kernel developer knowing the subsystem in question.
> 
> And you blame the developers, but not bugzilla? Why are you so unable to 
> see bugzilla as part of the *cause* of the problem? You're perfectly happy 
> to blame other things, but bugzilla is somehow above blame?

If Andrew forwarded a bug reported in Bugzilla to a developer, and 
the developer doesn't answer, is this Bugzilla's fault? Or in any other 
way worse than a bug report direct to the developer?

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[David Lang]

On Sun, 29 Apr 2007, Theodore Tso wrote:

> On Sun, Apr 29, 2007 at 03:15:42PM +0200, Andi Kleen wrote:
>> This means we need people who figure out who to assign bugs too.
>> Aka bugmasters.
>>
>> BTW one big problem in our current bugzilla is that a lot of people
>> cannot reassign bugs they don't own. I sometimes see bugs that I don't
>> own bug I know who is responsible, but bugzilla doesn't allow me to do it.
>>
>> So I think what would help:
>>
>> - Ask more people to just categorize and reassign bugs (anybody interested?)
>> - Give more people in bugzilla the power to reassign arbitary bugs
>> (bugzilla maintainers would need to do that)
>
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.
>
> More importantly, anyone is allowed to recategorize and reassign bugs.
> If someone does so maliciously or incorrectly, you can always revert
> it, and if someone is being truly malicious, you can always blacklist
> that one person.  It this respect, it is far more wiki-like than
> bugzilla, which has always been too much like a straightjacket.
>
> It's not perfect, but it's better than bugzilla --- but then again,
> just about *anything* would be better than bugzilla.  (Hmm, except
> maybe SourceForge's very tragic bug tracking system... :-)
>
> Of course, as Linus has said, it's not a complete solution --- you
> still need humans to be smart about things --- but if the goal is to
> make it easier to archive and track information about a bug, at
> *least* with the Debian BTS, when you reply to an e-mail message, the
> reply is automatically appended to the bug log!

this, and the fact that anyone can add to the bug log by just sending an e-mail 
are a nice feature

however, I had a reason to take a look at the debian BTS late last week to see 
if the bugs and patch that I sent to the sysklog maintainer (both debian and 
upstream) got included in debian 4.0.

talk about depressing.

there are about a dozen bugs _with_ patches sitting in the queue for several 
years

David Lang

Re: Linux 2.6.21

[Nicolas Mailhot]

Johannes Stezenbach wrote :
> On Sun, Apr 29, 2007, Adrian Bunk wrote:
> > On Sun, Apr 29, 2007 at 01:33:16PM -0700, Linus Torvalds wrote:
> > > On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > > > 
> > > > The kernel Bugzilla currently contains 1600 open bugs.
> > > 
> > > Adrian, why do you keep harping on this, and ignoring reality?
> > > 
> > > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> > 
> > OK, how do you suggest to track bugs in a way that doesn't suck?
> > 
> > Bug reports to linux-kernel have the big problem that they are lost if 
> > no developer immediately picks them up.
> 
> I suspect some bug reports get ignored deliberately.

[...]

You're probably not the only one thinking this but you know? This kind
of message makes me barf

> Y'know, a lot of developers don't just work for the money, they
> work for the fun of it. Thus they try to avoid pain and grief. :-)

Y'know, a lot of bug reporters are not paid at all, they try to help by
reporting problems.

> it's hard to write good bug reports,
[...]
> Thus they try to avoid pain and grief. :-)

It's way easier (and rewarding) to flame Linux on MLs, forums, articles
and make it a reputation darker than the darkest night than to have to
engage some developers on a bug report.

> Thus it's
> just natural that you learn from the experience to ignore every
> bug report which even remotely smells fishy ;-/

Thus it's just natural that you learn from the experience to stop
reporting problems and let some other poor sod do it, and instead spend
your time productively somewhere else

> Bugzilla just makes this visible because it doesn't forget.
> Which is stupid and discouraging for both (potential) bug
> reporters and developers.

Are you kidding? Do you think having a painfully drafted e-mail report
ignored is any less discouraging? Why do you think Adrian Bunk is
blowing a fuse today?

> And I also think that ignoring bad bug reports _increases_
> the software quality, because you can use the saved time
> working on something productive. 

All it increases is the lack of testing since ignored people stop
reporting bugs, so you have :
1. a small minority of hardened bug reporters (seems even Adrian is not
hardened enough to keep on it)
2. a majority of newbie reporters that don't know how to do clean
reports yet, and will vanish from the circuit before learning as soon as
they get the "ignored" treatment.

> And it makes developers happier :-)

That's the actual reason and it's a bad one. Unless developers have the
hardware, time and energy to do their own testing. Which is not the case
(read Andrew Morton's periodical complaints)

Now, here is what this bug reporter thinks:

1. bugzilla sucks for all the reasons written before

2. e-mail reports suck more:
- unless you post at the right time you get ignored
- you spend your time re-sending the same info because people don't read
thread histories
- you can't attach significant debugging material such as logs because
you'll hit random list size limits
- you get your mail address harversted by spammers (and if you create a
one-shut address people complain you don't consult it often enough)
- list filters will randomly decide your message is spam, and your
report will be lost
- you can't easily reassign reports once a group told you the bug is
somewhere else, you have to restart from step 1
- you don't even have the sucky target list you have in bugzilla - you
spend more time finding maintainer and list addresses than writing the
report (and then you have to monitor a gazillon of list to check if
someone read your stuff)
- people will write you "this looks the same as foo" and you'll spend
some more hours locating foo instead of having a clear pointer to
another report

e-mail is all nice and dandy for people who:
- have setup the infrastructure on their desktop to process large mail
volumes
- are behind strong spam filters
- have the clout to be listened to at once
- have all the relevant list & maintainers pointers in their head
- have helpers like Adrian to sort their mail for them
- have some personal http space where they can put bulky info the lists
won't accept

For the average bug reporter it's a very hostile system. Most people
don't have any of this, that's what bugzilla provides (and of course if
you have your own personnal infrastructure bugzilla may seem
unnecessary)

Don't get me wrong I had bugs solved through e-mail (or irc), and I had
bugs solved through bugzilla, and every time having a motivated fixer
and a good bug report was more important than the method used. These
days I only report through bugzilla because it makes it easier to attach
stuff and create good reports.

Now, the critical part on any bug reporting system is people like
Adrian. You can have reporter-oriented systems (bugzilla) or
developer-oriented systems (e-mail) what makes them work are the
go-betweens (who happen to do the less sexy and hardest part of the
process). They're the ones who mediate between:
1. reporters who just want to report a problem without spending a month
learning the habits of the people in charge of the code
2. developers who want gold-plated reports

So IMHO:

1. people like Adrian should be the ones choosing the reporting method
so it makes *their* job not the reporter or developer ones easier

2. when they bend backwards to send reports in the developer-preferred
form (like Adrian did) it's criminal to ignore them like a random e-mail
or bugzilla report. The bug may be more work than the technical problem
is worth but that's not the point. People like Adrian are not
replaceable. When they've invested so much time producing nice reports
you have to act on them if only to mark your respect for people doing a
thankless but necessary job.

-- 
Nicolas Mailhot

Re: Linux 2.6.21

[Johannes Stezenbach]

On Sun, Apr 29, 2007, Theodore Tso wrote:
> 
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.

Plus it has the very user-friendly reportbug and querybts
commandline interfaces. No going to web pages to query
bugs, and you can just download the email thread for
a bug report as an mbox file and then reply via email.

(Querybts currently only works if you know the package
name, it can't search across all packages so it wouldn't
be that useful for the kernel. But for Debian packages
this tool is gold.)


Johannes

Re: Linux 2.6.21

[Matthias Andree]

On Mon, 30 Apr 2007, Indan Zupancic wrote:

> I don't know, but what about telling the hapless person who went
> through the process of posting a bug what's wrong with the bug report?

It's a tedious process you keep doing over and over and over and over again,
and my experience shows it's sheer luck if people can actually fill in
the missing bits given the list.

Usually you have to ask thrice to obtain even the most essential
information such as version. Let alone vendor patches.

Anyways, the solution to this problem is someone _politely_ asking
reporters to provide necessary information and also point out that they
cannot ever hope to have their bug fixed without making a best-effort
attempt at answering all questions the first time they're being asked.

There are notable exceptions, people pinpointing code fragments at fault
and everything, but those are usually tech people and not end users.

> That said, if someone is an obvious idiot, ignoring saves time. But I
> think that's quite rare, and in general you should give the reporter
> feedback, and then ignore the bug report. (Until it improves.)

And that is what happens all too often (not in absolute figures, but in
the developer's perception of it) - insufficient information to debug.

Yes I know, some of the bugs hide themselves so well you actually need
four or five reports by different people to actually pinpoint the bug,
perhaps accompanied by insufficient interface documentation that make it
difficult to verify assumptions/expectations or assess potential
solutions (such as the res_init() issue in fetchmail, or probably the khubd going
south issue in Linux), but that's not the point.

-- 
Matthias Andree

Re: Linux 2.6.21

[Anton Altaparmakov]

On 29 Apr 2007, at 22:24, Linus Torvalds wrote:
> Exactly because I don't think anybody has shown any better  
> automation than
> bugzilla. But that doesn't make bugzilla "the One Choice". That's  
> not how
> it works. If there is no automation, manual tracking is still  
> better than
> *crap* automation.

Have you seen/used RT? -> http://bestpractical.com/rt

We use it here at work and it works great.  People can report bugs  
both by email or via web interface.  We get everything that comes in  
emailed to us and we can respond by email and RT recognizes the  
responses being in the same thread and lumps them into the same bug   
(and when the origin was by email that is even without evil bug  
numbers appearing in the subject with the help of some perl scrip  
magic (aka RT action script)).  The only time I ever go into the web  
interface is about once a week to have a look at my list of open bugs  
and to do some tidying like merging bug reports and things like that.

It also has some cool features like "extract this into the FAQ" and  
there is a "FAQ" in RT that contains an autogenerated FAQ from what  
people have pulled out in that way.

Only problem is for the kernel we would need a beefy system (needs  
fast database or it gets very slow when you get into 100k+ bugs  
region) and someone who knows RT well and has a lot of spare time to  
set it up to your liking and then to maintain it...  (RT takes a  
while to set up because you can tweak just about everything and you  
can add/modify/remove functionality at will as it is very modular and  
written in Perl so pretty much anyone can adapt it to do exactly what  
they want without even needing to wait for lengthy recompiles to  
happen...)

You could for example automate sorting of bug reports into queues  
(e.g. SCSI, Net, FS, etc) by grepping the emailed bug report (or  
website generated one although on the website people can choose the  
queue by hand if they want) and sorting appropriately.  Admittedly  
for this to be at all useful someone would have to spend some time  
working out intelligent things to grep for or all bugs would match to  
all queues when they contain dmesg output for example... (-;

This might not be perfect but in comparison to bugzilla it is  
actually usable and at least we here at work like it so much we  
adopted it into our workflow voluntarily which says something...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Linux 2.6.21

[Matthias Andree]

On Sun, 29 Apr 2007, Linus Torvalds wrote:

> What works is somebody who is a bugmaster, and it doesn't really matter 
> *what* bug tracker he points to (bugzilla being one of the possibilities, 
> although not necessarily the best, and absolutely NOT the only choice), 
> and turn them into emails. Once they are emails, bugzilla can track them.

While I'm not reading this entire thread for lack of time:

This looks exactly like the kind of bug tracking that Timo Sirainen of
Dovecot fame concocted and talked about on the dovecot-users list,
quote:

| Dovecot BTS
| -----------
|
|The preferred way to report bugs is to send them to dovecot-bugs at
|dovecot.org. The only thing it does is prefix the subject line with [BUG
|#nnn] and forward it to dovecot at dovecot.org.
|
|Now everyone can reply to it just as it was a normal mailing list mail.
|As long the subject contains the "[BUG #nnn]" prefix, it's part of the
|bug.
|
|Existing mailing list threads can also be turned into bugs by replying
|to the thread's root message with To: dovecot-bugs at dovecot.org. This
|again causes the new reply to contain [BUG #nnn] prefix.
|
|Then comes the web part. [...]"      -- quoting Timo Sirainen
<http://www.dovecot.org/list/dovecot/2007-January/018786.html>

Perhaps it's a stripped-down sibling of the Debian BTS without itself
knowing :-) anyways, it's E-Mail centric, low ceremony, devised from the
currently implemented workflow.

I haven't followed its details, portability, shape, state or anything,
but the requirements appear very similar to Linus's -- at least to me
with entirely outside view (I've never used kernel bugzilla), so it
might be a starting point, conceptionally or with some luck even
implementation-wise.

(Yes I know it's going to be tough to obtain all the precioussssssss
bugssssssssssss from bugzzzzzzzzzzzzzzzilla but anyways, if nobody likes
to use it, something will be done and if only neglect...)

HTH

-- 
Matthias Andree

Re: Linux 2.6.21

[David Miller]

From: Tomasz Chmielewski <mangoo@wpkg.org>
Date: Mon, 30 Apr 2007 08:09:13 +0200

> Why didn't you do it then? Why didn't you send your patch to the main 
> developer?
> Wouldn't be your problem fixed if you did it?

Because all the directions say to report bugs to the bugzilla via
bugs.freedesktop.org, which I did.

I went out of my way to go through the hassle of creating an
account, figuring out which buttons to push, how to attach
a patch, etc.

And my reward for all that effort is that either my work gets
ignored or I should send it to the mailing list?

That's garbage.

If bugs should be reported to the mailing list, then they
should just get rid of bugzilla because it's aparently
serving as a garbage bin.

Re: Linux 2.6.21

[Andrew Morton]

On Mon, 30 Apr 2007 00:09:06 +0200 "Rafael J. Wysocki" <rjw@sisk.pl> wrote:

> On Sunday, 29 April 2007 22:52, Alexey Dobriyan wrote:
> > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > of the relevant message, so why to require the reporter to file the report with
> > > the bugzilla himself?]
> > 
> > Last time I did this, bugzilla at osdl.org won't let me add original
> > reporter to goddamn CC list. It would be el neat, because not everyone
> > followed instructions and forwarding emails between reporter and
> > bugzilla sucks.
> 
> That's related to what I said before.  The requirement that the addresses on
> the CC list must be 'known' to bugzilla is deadly wrong in every case I can
> imagine.

But unknown-to-bugzilla email addresses are accepted when they're sent to
bugme-daemon@kernel-bugs.osdl.org.  This is why I'll very often switch a
bug report to email, copying individuals and mailing lists and
bugme-daemon.  Then bugzilla just sits silently in the background recording
everything.

But once a bug has switched to email, it needs to stay there - it would be
bad if someone were to update the bug via the web UI because none of the
emailed participants would know of the update.  So i'll often explicitly
ask "please follow up via emailed reply-to-all".

It's not great, but there's certainly enough material here for people to get
in and work on the bug, should they be so inclined.

My overall approach with this stuff is: short-term bugs are handled via email
and long-term ones are tracked in buzilla.

Hence someone (Hi, Mom) needs to track all the emailed-only bug reports and get
them filed in bugzilla once they go stale.

Re: Linux 2.6.21

[Tomasz Chmielewski]

David Miller wrote:

>> > I reported a bug that eats people's hard disks due to a bug
>> > in the X.ORG PCI support code on sparc, NOBODY has fixed
>> > the bug in 2 years even though a full bugzilla entry with
>> > even a full patch fix is in there.
>> 
>> Well but at least they could find it again if they wanted.
>> If you sent it by email and it had gotten lost for some reason
>> (nobody interested, which seems to be the real issue here) then
>> it would be lost forever.
> 
> WRONG!  If I have sent it to the main developer list the damn patch
> would be applied by now.

Why didn't you do it then? Why didn't you send your patch to the main 
developer?
Wouldn't be your problem fixed if you did it?


> WHY?
> 
> BECAUSE EMAIL ENGAGES PEOPLE AND BUGZILLA DOES NOT!

-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[Willy Tarreau]

On Sun, Apr 29, 2007 at 07:37:25PM +0200, Andi Kleen wrote:
> > My personal experience with bugzilla is that it's very unfriendly to
> > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > debug patches, system information etc., but not for _reporting_ new ones.
> 
> What did you find unfriendly? 

It is too much complicated for new reporters.

I remember I sent a patch by mail for NTP to people at ISC, and they asked
me to pass through bugzilla because it was important for them to track it.
What initially was a 5 minutes email turned to a 30 minutes nightmare with
doubts at every click, and it was even difficult for me to attach the patch.

Later they replied to me by mail, which I consulted from another address, I
replied and was rejected because it was not the same address... I had to be
very very motivated to use such a crap.

Definitely the tool we need if we want to reduce the number of bug reports!

Just my experience as a bug reporter...

Willy

Re: Linux 2.6.21

[Gene Heskett]

On Monday 30 April 2007, Bernd Eckenfels wrote:
>In article <200704292150.22956.gene.heskett@gmail.com> you wrote:
>> You can't have it even do a search to see if it already has something
>> similar without creating an account and logging in.  Since I'm out of wall
>> space, and the missus is bugging me to paint over all that, I left.
>
>Well, thats not a bugzilla problem. upstream bugzilla allows anonymous
>search.
>
>Infact bugme.osdl.org allows search right on the frontpage. And if you want
>to dig deeper, use the query function.
>
>This is the quicksearch on "USB":
>
><http://bugme.osdl.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug
>_status=OPEN&bug_status=ASSIGNED&bug_status=REOPENED&field0-0-0=product&type
>0-0-0=substring&value0-0-0=usb&field0-0-1=component&type0-0-1=substring&valu
>e0-0-1=usb&field0-0-2=short_desc&type0-0-2=substring&value0-0-2=usb&field0-0
>-3=status_whiteboard&type0-0-3=substring&value0-0-3=usb>
>
And that output I'll have to admit, is much more usefull.  However, in your 
wildest dreams I couldn't recompose that link line if I was stuck in a pool 
drain 10 feet down and that turned on the air hose I had in my hands.

>> I repeat:  Usefull?  For what?
>
>Try again.
>
>Gruss
>Bernd



-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
He who is content with his lot probably has a lot.

Re: Linux 2.6.21

[Kyle Moffett]

It might not be bad to write up an email-based BTS-alike bug-tracking  
system just for the Linux kernel.  It should probably even be  
implemented 100% via email at first, with a web-based status viewer  
as a later add-on.  Here's a possible email format:

[kbugger: action1 arg1 arg2 ..., action2 arg1 arg2 ...]

Make it almost totally message-id and thread based, and make it an  
implicit part of LKML (IE: subscribe the kbugger program to LKML).   
People who are flagged "admin" may ban/unban users and make certain  
large-scale changes.  Supported actions:

create, create-parent, create-thread:
   Create a new bug associated with this message.
   The arguments specify the title.
   This would automatically happen for new threads with titles like  
"[BUG] foo: It's broken"

merge:
   Merges the current bug and/or email thread into an existing bug.
   The arguments are a list of bug numbers and/or message-IDs to  
merge together with this one.

prune, prune-parent, prune-thread:
   Prunes a given thread from the current bug.
   Optional argument specifies a referential message-ID

settitle:
   Change the title of the current bug

fixed, broken:
   Mark the bug as fixed or broken in a particular version/configuration
   Arguments are used as opaque strings representing configurations  
where it is known to be fixed or broken.  For example [kbugger: fixed  
2.6.16 2.6.20-x86, broken 2.6.20-ppc] would just store the list of  
strings and statuses.  If the bug was auto-created with a title like  
"[BUG ppc] foo: It's broken" or "[BUG 2.6.20] bar: I dunno", then the  
argument to the [BUG] title portion will be auto-passed to [kbugger:  
broken].

status:
   Get a brief status report on the current bug.

info:
   Get a detailed status report on the current bug.

history:
   Get detailed information about the history of the current bug.   
This only sends the reply to the author.

stop:
   Stop parsing the rest of this email.  Useful when teaching  
somebody about kbugger commands via an email sent to kbugger itself.

The results of the kbugger statements in an email will be sent as a  
reply to the original message and "To:" the original sender, "CC:"- 
ing the targets of the message, so if the original [kbugger: create]  
post went to LKML then the reply will go there as well for people to  
read and for it to be archived by kbugger as part of the status  
history of that bug.  All emails it receives will be autoparsed for  
commands, however it should be coded to ignore all text in emailed  
patches, and it should support the [kbugger: stop] command to halt  
parsing for cases where you need to send plain kbugger commands via  
email.

If somebody was feeling unusually brave they might add some  
keyword<=>author bayesian tracking so that it could automatically  
discover the keywords in emails that particular authors reply to and  
helpfully forward a kbugger info email to that person with a link to  
the archives for the original thread.  If somebody didn't want to  
receive such info messages they could click a link or send a  
[kbugger: no-auto-forward] command to blacklist themselves from  
receiving automatic forwards ([kbugger: auto-forward] to re-enable).   
Maybe even [kbugger: not-for-me ...] and [kbugger: for-me] commands  
which take bug numbers and message-IDs to tune the keyword lists.

I may try working on something like that this week if I get the time.

Cheers,
Kyle Moffett

Re: Linux 2.6.21

[Bernd Eckenfels]

In article <200704292150.22956.gene.heskett@gmail.com> you wrote:
> You can't have it even do a search to see if it already has something similar 
> without creating an account and logging in.  Since I'm out of wall space, and 
> the missus is bugging me to paint over all that, I left.

Well, thats not a bugzilla problem. upstream bugzilla allows anonymous
search.

Infact bugme.osdl.org allows search right on the frontpage. And if you want
to dig deeper, use the query function.

This is the quicksearch on "USB":

<http://bugme.osdl.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=OPEN&bug_status=ASSIGNED&bug_status=REOPENED&field0-0-0=product&type0-0-0=substring&value0-0-0=usb&field0-0-1=component&type0-0-1=substring&value0-0-1=usb&field0-0-2=short_desc&type0-0-2=substring&value0-0-2=usb&field0-0-3=status_whiteboard&type0-0-3=substring&value0-0-3=usb>

> I repeat:  Usefull?  For what?

Try again.

Gruss
Bernd

Re: Linux 2.6.21

[Gene Heskett]

On Sunday 29 April 2007, Michal Piotrowski wrote:
>Hi Diego,
>
>On 29/04/07, Diego Calleja <diegocg@gmail.com> wrote:
>[..]
>
>> So unless someone is willing to write such tool (which I doubt, since it
>> doesn't looks easy), all this discussion seems pointless, and we should
>> stick with this http://kernelnewbies.org/known_regressions page
>> which is showing to be quite useful :)

Usefull?  For what?  Having seen the link at the time I was fighting with yet 
another rearranged USB setup, and I swear the boot process uses Schrodingers 
Cat to determine which device found in the usb scan gets the first ttyUSB, 
and which gets the second, so I went to this site to see if it could be more 
productive than bugzilla (I'd like to toss about 10 pounds of C4 into the 
only code repository on the planet that thing is built from), but no, sorry, 
no bisquit.

Why?  First, its yet another password I have to scribble on the wall along 
with enough surrounding data so I might be able to find it the next time.

You can't have it even do a search to see if it already has something similar 
without creating an account and logging in.  Since I'm out of wall space, and 
the missus is bugging me to paint over all that, I left.

I repeat:  Usefull?  For what?

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Simon: (to Jayne) "Enemies?  You?  No, how can it be?"
				--Episode #7, "Jaynestown"

Re: Linux 2.6.21

[Andi Kleen]

Theodore Tso <tytso@mit.edu> writes:
> 
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.

Bugzilla can do that too. But I'm not convinced this is a good idea.
We had this some years ago with the jitterbug experiment and usually
it tended to faithfully keep track of very long off topic threads
that had drifted long from the original bug. The resulting collections
of emails usually not very useful.

While other interfaces might be a culture shock for some people
at least they force them to concentrate on the particular issue --
contribute to a specific bug.

-Andi

Re: Linux 2.6.21

[Björn Steinbrink]

[Oops, the first try of this mail got out from my local address, sorry]

On 2007.04.29 19:55:35 -0400, Theodore Tso wrote:
> On Sun, Apr 29, 2007 at 03:15:42PM +0200, Andi Kleen wrote:
> > This means we need people who figure out who to assign bugs too.
> > Aka bugmasters.
> > 
> > BTW one big problem in our current bugzilla is that a lot of people
> > cannot reassign bugs they don't own. I sometimes see bugs that I don't
> > own bug I know who is responsible, but bugzilla doesn't allow me to do it.
> > 
> > So I think what would help:
> > 
> > - Ask more people to just categorize and reassign bugs (anybody interested?)
> > - Give more people in bugzilla the power to reassign arbitary bugs
> > (bugzilla maintainers would need to do that)
> 
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.
> 
> More importantly, anyone is allowed to recategorize and reassign bugs.
> If someone does so maliciously or incorrectly, you can always revert
> it, and if someone is being truly malicious, you can always blacklist
> that one person.  It this respect, it is far more wiki-like than
> bugzilla, which has always been too much like a straightjacket.
> 
> It's not perfect, but it's better than bugzilla --- but then again,
> just about *anything* would be better than bugzilla.  (Hmm, except
> maybe SourceForge's very tragic bug tracking system... :-)
> 
> Of course, as Linus has said, it's not a complete solution --- you
> still need humans to be smart about things --- but if the goal is to
> make it easier to archive and track information about a bug, at
> *least* with the Debian BTS, when you reply to an e-mail message, the
> reply is automatically appended to the bug log!

I've started hacking on some bash scripts to do the e-mail part, based
on the requirements I've gathered from reading this thread. So far, I
got one script that can be plugged into procmail and processes mails to
do the following (right now):

 - Create a bug
 - Set the bug type to "regression"
 - Update the timestamp of the last action for that bug
 - Assign the bug to a subsystem
 - Track the submitter and whoever grabs the bug

More (hopefully) to come...

Those commands are just added to any email in the thread discussing a
bug like this:
@bugthing
mine		# Mark myself as owner
regression	# Mark this bug as a regression (used for reports)
subsystem mm	# Assign the bug to mm
needinfo	# Tell the tracker to bug the reporter if nothing happens
Thanks

That block can appear anywhere in the email, so if you're discussing
some problem on lkml and want to track that bug, you can just add such a
block to your email and turn an untracked email conversion into a
tracked bug.

Tracking does _not_ mean that all emails are stored, those can be looked
up on lkml.org or MARC, where the created reports will probably contain
URLs to the latter, because it supports lookups based on message ids.

Tracking does just mean that the state of the bug is stored somewhere.
That means (currently, suggestions welcome):

 - What's its name? (E-Mail subject)
 - Who reported it?
 - Who (if any) stepped up to own that bug?
 - What type of bug is it?
 - Which subsystem does it belong to?
 - What's its current state? (new, owned, fixed, ...)
 - When did the last action on this bug happen?

Based on that information, I've started writing some scripts that create
"reports" (all of them currently being pretty incomplete):

 - Bugs that belong to a specific subsystem (on request, currently
   through a procmail triggered script; this is meant to satisfy
   Adrian's request of asking for example for all SATA bugs.)

 - Bugs that are not owned and didn't see any action for one week
   (cronjob; to lkml, the submitter, the recipients of the bug mail and
   maybe folks who subscribed to such mails)

 - All bugs marked as regressions (complete list to lkml, personalised
   lists to each owner of any regressions, probably sent on request)

 - Annoy the bug submitter if someone tagged the bug with "moreinfo" and
   there wasn't any reply from the submitter within one week

 - Annoy the bug owner if he didn't do anything about his bug within
   one week

Those "reports" should push the relevant parties into looking at those
bugs again while being terse, like Adrian's regression list and not
annoying anyone but those who are most likely to be interested.

Bugs in which noone shows interest (i.e. nothing happens at all), should
probably be removed after 2-3 weeks, with a notice going to the
submitter and recipients of the bug's emails as well as those subscribed
to receiving reports of non-owned inactive bugs. Well, unless those bugs
are regressions, we should probably bug about them until they get
closed.


That's my plan so far. It's probably going to be ugly and all, but I
wanted to get something done quick (and learn a bit about bash scripting
along the way) to see if it works at all, and what you guys think about
that approach. Maybe I can write a less ugly version later, or someone
picks up the approach and writes a better implementation himself. But
maybe the preliminary one (once it is somewhat useable) could be tested
by a few brave folks to see if the approach works out at all...

I don't know about the Debian BTS internals, but IMHO, the mentioned
capability of being email-based isn't enough for Linux.  Things might
start out on lkml as some small issue, but suddenly there's some reason
to track a bug that is discussed in an already existing thread. If you
file it into an email based system, you loose the entire history of that
discussion if you look at it from the BTS.  With the tracker-only
approach, you simply stay email based and if required, you just lookup
the right thread using the information from the tracker and go back to
email, if you don't have those emails, the public archives most probably
do.

And far more important are IMHO the reports, which should distribute the
work of tracking bugs a bit. If a regression is found, the reporter, the
owner or anyone reading the thread, can mark it as such and noone is
forced to track _all_ regressions. The wiki helps with this, too, but
IMHO it's neither easy nor fast enough to look up and change that patch
manually for anyone to enjoy that. Also, the automatic reminders should
help (a bit) to not loose too many bugs, just because the person
responsible for it got distracted (and forgot about it) or because the
bug submitter forgets that he needed to provide more info, which he
couldn't do at the time the request hit his inbox (I do that all the
time).

Does this sound like something usable in the long run? Do I just waste
my time? Do I miss something very important?

Björn

Re: Linux 2.6.21

[Dave Jones]

On Sun, Apr 29, 2007 at 07:55:35PM -0400, Theodore Tso wrote:

 > but if the goal is to
 > make it easier to archive and track information about a bug, at
 > *least* with the Debian BTS, when you reply to an e-mail message, the
 > reply is automatically appended to the bug log!

bugzilla does that too.

	Dave

-- 
http://www.codemonkey.org.uk

Re: Linux 2.6.21

[Indan Zupancic]

On Mon, April 30, 2007 01:41, Johannes Stezenbach wrote:
> Developers are just humans and if they have no incentive to
> act on a bug report they will ignore it. I think this is a
> fact that you have to deal with.

Reporters are just humans too and if they have no incentive to
post bugs they won't. So it's a lose/lose situation, really.
With a group of people working together they should try to
motivate each other, not demoralize everyone.

(I know, each bug report is a pain, voicing someone's failure.
So ignoring it might make people feel better, but it doesn't
fix anything.)


> It's also not necessarily the fault of the reporter if
> a bug report gets ignored, but for every report a developer
> has to make a decision to handle it or not, and there are
> lots of reasons why he may decide to not handle it, or
> at least not now (and then forget about it).

True. There's also a difference between a bad bug report and
one that a specific developer won't handle. In the former case
anyone could recognize it and tell the reporter about it. The
latter is a bit trickier, but if the developer thinks about
looking at it later, he better can tell the reporter just that.
A short "I'll take a look at it, later, when I've more time."
is so much better than plain silence.


> But I'm quite sure that an important bug would be reported
> again until fixed.

I wouldn't be so sure about that. What's worse, why would the
reporter bother telling that the bug is fixed in version N+1?
No one cared about it anyway, so there's no one to tell it to.
That would explain a lot open bugs too.

Greetings,

Indan

Re: Linux 2.6.21

[Theodore Tso]

On Sun, Apr 29, 2007 at 03:15:42PM +0200, Andi Kleen wrote:
> This means we need people who figure out who to assign bugs too.
> Aka bugmasters.
> 
> BTW one big problem in our current bugzilla is that a lot of people
> cannot reassign bugs they don't own. I sometimes see bugs that I don't
> own bug I know who is responsible, but bugzilla doesn't allow me to do it.
> 
> So I think what would help:
> 
> - Ask more people to just categorize and reassign bugs (anybody interested?)
> - Give more people in bugzilla the power to reassign arbitary bugs
> (bugzilla maintainers would need to do that)

Folks might want to take a look at the Debian Bug Tracking System
(BTS).  It has a web interface which you can use to query history, but
*everything* is e-mail driven, and the way you submit, close, update,
tag/classfy bugs --- everything --- is via e-mail.

More importantly, anyone is allowed to recategorize and reassign bugs.
If someone does so maliciously or incorrectly, you can always revert
it, and if someone is being truly malicious, you can always blacklist
that one person.  It this respect, it is far more wiki-like than
bugzilla, which has always been too much like a straightjacket.

It's not perfect, but it's better than bugzilla --- but then again,
just about *anything* would be better than bugzilla.  (Hmm, except
maybe SourceForge's very tragic bug tracking system... :-)

Of course, as Linus has said, it's not a complete solution --- you
still need humans to be smart about things --- but if the goal is to
make it easier to archive and track information about a bug, at
*least* with the Debian BTS, when you reply to an e-mail message, the
reply is automatically appended to the bug log!
	
						- Ted

Re: Linux 2.6.21

[Johannes Stezenbach]

On Mon, Apr 30, 2007, Indan Zupancic wrote:
> On Mon, April 30, 2007 00:36, Johannes Stezenbach wrote:
> > The lesson to learn is that there are some very valid
> > reasons why bug reports get ignored (some not mentioned here),
> > and there's nothing you can do about it. And it has nothing to
> > do with the method or tool used for reporting or tracking.
> >
> > And I also think that ignoring bad bug reports _increases_
> > the software quality, because you can use the saved time
> > working on something productive. And it makes developers
> > happier :-)
> >
> > [Just to avoid misunderstandings: By no means do I advocate
> > ignoring *every* bug report. But ignoring the bad ones
> > is just the sane thing to do.]
> 
> I don't know, but what about telling the hapless person who went
> through the process of posting a bug what's wrong with the bug report?
> Or is software quality the only thing you care about, and you don't
> want to waste time on learning people to write better bug reports?
> 
> If you want to scare away bug reporters, just ignore their first (and
> thus likely bad) bug report they write. There isn't much less motivating
> than a deafening silence. Just compare it with writing a patch.
> 
> Though, in a sense, if the software quality is measured by the number of
> bug reports, your tactic might "improve" it indeed.
> 
> That said, if someone is an obvious idiot, ignoring saves time. But I
> think that's quite rare, and in general you should give the reporter
> feedback, and then ignore the bug report. (Until it improves.)

Well, I'm a moody bastard, and I would hope other people handle
this better than I. However, all the bugzillas of this world
are full with old, ignored bugs, amd I thought this might
serve as an explanation why.

Developers are just humans and if they have no incentive to
act on a bug report they will ignore it. I think this is a
fact that you have to deal with.

It's also not necessarily the fault of the reporter if
a bug report gets ignored, but for every report a developer
has to make a decision to handle it or not, and there are
lots of reasons why he may decide to not handle it, or
at least not now (and then forget about it).

But I'm quite sure that an important bug would be reported
again until fixed.


Johannes

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 23:51, Diego Calleja wrote:
> El Sun, 29 Apr 2007 23:10:28 +0200, Adrian Bunk <bunk@stusta.de> escribió:
> 
> > What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> 
> AFAIK, submitting its contents to the list periodically CCing the developers,
> like you did with your lists.
> 
> If developers care to fix it or not or how much Linus cares about that list before
> releasing a new version is another question. I think it's useful because it makes
> those bugs look more important than the 1600 stored in the bugzilla...it won't
> help to fix those 1600, but it attracts some attention over the "release critical"
> ones and encourages developers to fix them, even if not all of them get fixed.
> 
> I don't think you can do many other things to get as much bugs fixed as possible,
> unless we reward bug fixers with weekends in the Playboy mansion. I think the
> fundamental question here is: is there a way to make hackers follow and fix _all_
> the bugs? I'd love it was possible, but AFAIK all the projects that have tried to
> be ultra-stable and have adopted a policy to fullfill such goal have fallen behind
> of competing projects that cared more about working in improving their software.

Apart from this many bugs are found and get fixed in the process of developing
new code, so the 'ultra stable' approach is not really practical.

Greetings,
Rafael

Re: Linux 2.6.21

[Indan Zupancic]

Hello,

On Mon, April 30, 2007 00:36, Johannes Stezenbach wrote:
> The lesson to learn is that there are some very valid
> reasons why bug reports get ignored (some not mentioned here),
> and there's nothing you can do about it. And it has nothing to
> do with the method or tool used for reporting or tracking.
>
> And I also think that ignoring bad bug reports _increases_
> the software quality, because you can use the saved time
> working on something productive. And it makes developers
> happier :-)
>
> [Just to avoid misunderstandings: By no means do I advocate
> ignoring *every* bug report. But ignoring the bad ones
> is just the sane thing to do.]

I don't know, but what about telling the hapless person who went
through the process of posting a bug what's wrong with the bug report?
Or is software quality the only thing you care about, and you don't
want to waste time on learning people to write better bug reports?

If you want to scare away bug reporters, just ignore their first (and
thus likely bad) bug report they write. There isn't much less motivating
than a deafening silence. Just compare it with writing a patch.

Though, in a sense, if the software quality is measured by the number of
bug reports, your tactic might "improve" it indeed.

That said, if someone is an obvious idiot, ignoring saves time. But I
think that's quite rare, and in general you should give the reporter
feedback, and then ignore the bug report. (Until it improves.)

Greetings,

Indan

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Monday, 30 April 2007 00:00, Adrian Bunk wrote:
> On Mon, Apr 30, 2007 at 12:00:28AM +0200, Rafael J. Wysocki wrote:
> > On Sunday, 29 April 2007 22:43, Adrian Bunk wrote:
> > > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > > >...
> > > > But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> > > > From lkml.org, for example.  Why don't we use that?  The only missing piece
> > > > is the 'keep a list' thing, but that's not a rocket science, IMHO.
> > > > 
> > > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > > of the relevant message, so why to require the reporter to file the report with
> > > > the bugzilla himself?]
> > > > 
> > > > _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> > > > keep track of each thread separately, so you can browse any of them at any
> > > > time.  In particular, you can see the _history_ of each bug report sent to LKML
> > > > if you have a link to any message in its thread.
> > > > 
> > > > Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> > > > you'll even be able to use the lkml.org archives plus wget and a couple of
> > > > shell scripts to cherry pick the links to all bug reports sent to the list
> > > > within a given time interval.
> > > > 
> > > > All of this functionality is out there already.
> > > >...
> > > 
> > > How can I get the functionality "show me all unfixed SATA bugs"?
> > > 
> > > That's one of the important functionalities of every bug tracking 
> > > system.
> > 
> > That's the missing piece, obviously.
> > 
> > BTW, I didn't want to say that one could entirely replace a bug-tracking system
> > with tracking the LKML archives.  What I wanted to say was that the email
> > messages sent to the LKML were easily trackable and could be hooked up into a
> > bug-tracking system, for example with the help of URLs.
> > 
> > In such a setup people could send initial reports to the LKML and the links to
> > these messages might be put into a bug-tracking system as soon as it turned
> > out that the bugs were worthy of tracking.
> 
> Who is doing this "might be put", and why don't you start with asking 
> the submitter to submit bugs in a bug tracking system and forward the 
> bug report from the bug tracking system (manually or automatically) to 
> the developers and linux-kernel?

Because quite often I know what the problem is after having asked the reporter
a couple of simple questions or I can forward his report to a list on which
there are the right people and the problem gets fixed quickly, so it just
doesn't need to be formally registered.

The problem with the bugzilla is that it requires a considerable setup for each
bug which is a loss of time if the bug is trivial.  Using the bugzilla for
handling trivial bugs just doesn't make sense, IMHO.  It's too heavywieght
for that.  [It _is_ useful nevertheless.  For example, the suspend metabug that
you've created for me is really a nice thing, so thanks a lot again. :-)
Perhaps something like this might be used for tracking the regressions ...]

Unfortunately, you can't say whether or not the bug is trivial until you see
the report.  If you've got it from bugzilla, you're stuck with it, so it's just
more efficient to use something else in the initial phase of resolving
problems.

Greetings,
Rafael

Re: Linux 2.6.21

[Andi Kleen]

> The actual list of known regressions is wiki based. Everyone can
> update bug status, add references etc.

Well do they know about it? 

Also something a little more structured would seem better for this.
How do you query a wiki? 

-Andi

Re: Linux 2.6.21

[Michal Piotrowski]

On 30/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Mon, Apr 30, 2007 at 12:33:30AM +0200, Thomas Gleixner wrote:
> > On Mon, 2007-04-30 at 00:19 +0200, Adrian Bunk wrote:
> >...
> > > And it failed because many regressions still stayed unfixed and some
> > > even undebugged.
> >
> > No it failed not. It is not perfect. Way more bugs, which have been
> > fixed or are in the debugging process, would have been unnoticed and
> > ignored otherwise.
> >...
>
> It depends on what you consider failure and what you consider success.

I hope that this discussion about bugs will change something in Linux
regressions front.

Huge thanks to you for that.

>
> For me, it failed. Not because it wasn't perfect, but because we could
> have done much better with fixing the known regressions, and also by not
> introducing several regressions between the last -rc and the final
> kernel (and people who did test -rc7 and would most likely also have
> tested an -rc8 ran into them).
>
> >       tglx
>
> cu
> Adrian

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Michal Piotrowski]

On 30/04/07, Andi Kleen <andi@firstfloor.org> wrote:
> > Right. Simply because these lists are assembled by someone
> >
> > - who knows how to pick that reports from the mailinglists
> > - who knows how to sort them in a useful way
> > - who knows how to add the relevant folks on CC
>
> That all needs to be done by someone initially yes. But then tracking what happens
> afterwards is something that can be distributed. A difficult bug can take a long
> time to resolve and generate a lot of messages; you don't want to require
> the initial sorter to handle all that too.
>
> It's much more scalable to let the developers update the state themselves
> then once they handle the bug.

The actual list of known regressions is wiki based. Everyone can
update bug status, add references etc.

>From time to time, I'll send this list to right people.

>
> -Andi
>

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 12:33:30AM +0200, Thomas Gleixner wrote:
> On Mon, 2007-04-30 at 00:19 +0200, Adrian Bunk wrote:
>...
> > And it failed because many regressions still stayed unfixed and some 
> > even undebugged.
> 
> No it failed not. It is not perfect. Way more bugs, which have been
> fixed or are in the debugging process, would have been unnoticed and
> ignored otherwise.
>...

It depends on what you consider failure and what you consider success.

For me, it failed. Not because it wasn't perfect, but because we could 
have done much better with fixing the known regressions, and also by not 
introducing several regressions between the last -rc and the final 
kernel (and people who did test -rc7 and would most likely also have 
tested an -rc8 ran into them).

> 	tglx

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Andi Kleen]

> Right. Simply because these lists are assembled by someone
> 
> - who knows how to pick that reports from the mailinglists
> - who knows how to sort them in a useful way
> - who knows how to add the relevant folks on CC

That all needs to be done by someone initially yes. But then tracking what happens
afterwards is something that can be distributed. A difficult bug can take a long
time to resolve and generate a lot of messages; you don't want to require
the initial sorter to handle all that too.

It's much more scalable to let the developers update the state themselves
then once they handle the bug.

-Andi

Re: Linux 2.6.21

[Johannes Stezenbach]

On Sun, Apr 29, 2007, Adrian Bunk wrote:
> On Sun, Apr 29, 2007 at 01:33:16PM -0700, Linus Torvalds wrote:
> > On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > > 
> > > The kernel Bugzilla currently contains 1600 open bugs.
> > 
> > Adrian, why do you keep harping on this, and ignoring reality?
> > 
> > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> 
> OK, how do you suggest to track bugs in a way that doesn't suck?
> 
> Bug reports to linux-kernel have the big problem that they are lost if 
> no developer immediately picks them up.

I suspect some bug reports get ignored deliberately.

Why? Because it's hard to write good bug reports, and thus
a lot of them suck.

In some cases you are lucky and get a test case which makes
the bug easily reproducible, or you get a nice, comprehensible
Oops message, and can pinpoint the bug right away. But usually you
have to interact with the user to get more information, have her
try various patches or do a bisect. And sometimes this interaction
drives you crazy because the user is slow to answer, gives you
incomplete or even false information, and doesn't do what you
ask her to do to narrow the bug, or you just have no clue
what the bug could be etc.

Probably every developer went through this experience a couple of
times, wasting a lot of time and causing a lot of grief. Thus it's
just natural that you learn from the experience to ignore every
bug report which even remotely smells fishy ;-/
Y'know, a lot of developers don't just work for the money, they
work for the fun of it. Thus they try to avoid pain and grief. :-)

Bugzilla just makes this visible because it doesn't forget.
Which is stupid and discouraging for both (potential) bug
reporters and developers.


The lesson to learn is that there are some very valid
reasons why bug reports get ignored (some not mentioned here),
and there's nothing you can do about it. And it has nothing to
do with the method or tool used for reporting or tracking.

And I also think that ignoring bad bug reports _increases_
the software quality, because you can use the saved time
working on something productive. And it makes developers
happier :-)

[Just to avoid misunderstandings: By no means do I advocate
ignoring *every* bug report. But ignoring the bad ones
is just the sane thing to do.]


Johannes

Re: Linux 2.6.21

[Thomas Gleixner]

On Mon, 2007-04-30 at 00:19 +0200, Adrian Bunk wrote:
> > Don't be silly, did any of the developers say, that he has spare time to
> > read your regression lists ?
> 
> It worked because several people (including Linus) emphasized that 
> fixing regressions from this list was important.

Right. Simply because these lists are assembled by someone

- who knows how to pick that reports from the mailinglists
- who knows how to sort them in a useful way
- who knows how to add the relevant folks on CC
- ....

Can you see the pattern ?

> And it failed because many regressions still stayed unfixed and some 
> even undebugged.

No it failed not. It is not perfect. Way more bugs, which have been
fixed or are in the debugging process, would have been unnoticed and
ignored otherwise.

> > So what are you complaining about ? Folks stepped up and built a
> > regression list and posted it to LKML. What's wrong with that ?
> 
> If it works it's perfectly fine.

It will work not much different from your lists. It'll be not perfect
either.

	tglx

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 11:52:01PM +0200, Thomas Gleixner wrote:
> On Sun, 2007-04-29 at 23:21 +0200, Adrian Bunk wrote:
> > >> > It's not great but it's the best clone of you we've found 8)
> > >>
> > >> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> > >
> > > It's for -stable team.
> > >...
> > 
> > Did Greg or Chris say they have spare time for going through this list?
> 
> Don't be silly, did any of the developers say, that he has spare time to
> read your regression lists ?

It worked because several people (including Linus) emphasized that 
fixing regressions from this list was important.

And it failed because many regressions still stayed unfixed and some 
even undebugged.

> Michal posted it to LKML with the relevant developers in CC including
> the stable team, so they are in the loop for updates, which is a Good
> Thing!
> 
> Hey, you did this yourself. It is a great help and as your lists did,
> Michals list pointed me to a bug, which I would have missed otherwise. 
> 
> So what are you complaining about ? Folks stepped up and built a
> regression list and posted it to LKML. What's wrong with that ?

If it works it's perfectly fine.

> 	tglx

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Andi Kleen]

> BECAUSE EMAIL ENGAGES PEOPLE AND BUGZILLA DOES NOT!
> 
> Nobody looks at the bugzilla because there is too much junk in there
> to make the signal any useful to search for, there's simply too much
> noise.

That means just x.org doesn't have a working bugmaster setup.
Again a technical solution doesn't fix misorganization or missing people;
it just pushes some work humans are at to computers if you have the right structure

The normal bugzilla workflow is that some people categorize the bugs, 
ask the necessary questions and then figure out which developer
to assign it to. Then the developer doesn't end up with "too much noise"
but just a limited set of bugs to look at.

And the responsible developer then gets an email and looks at the bug.

This already happens to some limited fashion in kernel.org bugzilla:
I get bugs assigned occasionally and while it's slow I tend to look
at (near) all of them and try to improve things there.

If a single developer ends up with too many bugs this way or there
is nobody to assign a bug to or nobody processes the incoming bugs
then the project has a problem. Yes bugzilla doesn't work then 
if the project is not well organized.

But that's in no way different from what would happen with your
email sent to a mailing list if it had the same problem.

-Andi (who finds it a bit bizarre he has to explain the concept
of "things can get easier when some work is pushed to computers"
concept to a hacker) 

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 05:27:56PM -0400, Dave Jones wrote:
> On Sun, Apr 29, 2007 at 02:05:42AM +0200, Adrian Bunk wrote:
>  > On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
>  > >...
>  > > What's the difference between bugzilla and lkml.org? Both have search 
>  > > buttons. Both archive the old stuff. Both can be pointed to.
>  > 
>  > Mailing lists don't track bugs.
> 
> There's no reason they can't.  Store them in folders 'fixed' 'pending'
> 'notabug' etc. Move mails between them when states change.
> reply-to them when necessary. Bounce them. Add people to Ccs. etc. etc.
> 
> The only remaining piece of the puzzle is "how does everyone see the
> states of the various pools", which could be solved in a number of
> ways, daily uploads to a place on kernel org for eg.
> 
> Hell, you could store them in a git tree if it came to it.
> That would also solve the problem for those with an aversion
> to web browsers to see bugs :-)

That would be an implementation of a git based bug tracker.  :-)

> 	Dave

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 22:52, Alexey Dobriyan wrote:
> On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > of the relevant message, so why to require the reporter to file the report with
> > the bugzilla himself?]
> 
> Last time I did this, bugzilla at osdl.org won't let me add original
> reporter to goddamn CC list. It would be el neat, because not everyone
> followed instructions and forwarding emails between reporter and
> bugzilla sucks.

That's related to what I said before.  The requirement that the addresses on
the CC list must be 'known' to bugzilla is deadly wrong in every case I can
imagine.

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 22:43, Adrian Bunk wrote:
> On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> >...
> > But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> > From lkml.org, for example.  Why don't we use that?  The only missing piece
> > is the 'keep a list' thing, but that's not a rocket science, IMHO.
> > 
> > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > of the relevant message, so why to require the reporter to file the report with
> > the bugzilla himself?]
> > 
> > _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> > keep track of each thread separately, so you can browse any of them at any
> > time.  In particular, you can see the _history_ of each bug report sent to LKML
> > if you have a link to any message in its thread.
> > 
> > Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> > you'll even be able to use the lkml.org archives plus wget and a couple of
> > shell scripts to cherry pick the links to all bug reports sent to the list
> > within a given time interval.
> > 
> > All of this functionality is out there already.
> >...
> 
> How can I get the functionality "show me all unfixed SATA bugs"?
> 
> That's one of the important functionalities of every bug tracking 
> system.

That's the missing piece, obviously.

BTW, I didn't want to say that one could entirely replace a bug-tracking system
with tracking the LKML archives.  What I wanted to say was that the email
messages sent to the LKML were easily trackable and could be hooked up into a
bug-tracking system, for example with the help of URLs.

In such a setup people could send initial reports to the LKML and the links to
these messages might be put into a bug-tracking system as soon as it turned
out that the bugs were worthy of tracking.

Greetings,
Rafael

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 12:00:28AM +0200, Rafael J. Wysocki wrote:
> On Sunday, 29 April 2007 22:43, Adrian Bunk wrote:
> > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > >...
> > > But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> > > From lkml.org, for example.  Why don't we use that?  The only missing piece
> > > is the 'keep a list' thing, but that's not a rocket science, IMHO.
> > > 
> > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > of the relevant message, so why to require the reporter to file the report with
> > > the bugzilla himself?]
> > > 
> > > _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> > > keep track of each thread separately, so you can browse any of them at any
> > > time.  In particular, you can see the _history_ of each bug report sent to LKML
> > > if you have a link to any message in its thread.
> > > 
> > > Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> > > you'll even be able to use the lkml.org archives plus wget and a couple of
> > > shell scripts to cherry pick the links to all bug reports sent to the list
> > > within a given time interval.
> > > 
> > > All of this functionality is out there already.
> > >...
> > 
> > How can I get the functionality "show me all unfixed SATA bugs"?
> > 
> > That's one of the important functionalities of every bug tracking 
> > system.
> 
> That's the missing piece, obviously.
> 
> BTW, I didn't want to say that one could entirely replace a bug-tracking system
> with tracking the LKML archives.  What I wanted to say was that the email
> messages sent to the LKML were easily trackable and could be hooked up into a
> bug-tracking system, for example with the help of URLs.
> 
> In such a setup people could send initial reports to the LKML and the links to
> these messages might be put into a bug-tracking system as soon as it turned
> out that the bugs were worthy of tracking.

Who is doing this "might be put", and why don't you start with asking 
the submitter to submit bugs in a bug tracking system and forward the 
bug report from the bug tracking system (manually or automatically) to 
the developers and linux-kernel?

> Greetings,
> Rafael

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Thomas Gleixner]

On Sun, 2007-04-29 at 23:21 +0200, Adrian Bunk wrote:
> >> > It's not great but it's the best clone of you we've found 8)
> >>
> >> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> >
> > It's for -stable team.
> >...
> 
> Did Greg or Chris say they have spare time for going through this list?

Don't be silly, did any of the developers say, that he has spare time to
read your regression lists ?

Michal posted it to LKML with the relevant developers in CC including
the stable team, so they are in the loop for updates, which is a Good
Thing!

Hey, you did this yourself. It is a great help and as your lists did,
Michals list pointed me to a bug, which I would have missed otherwise. 

So what are you complaining about ? Folks stepped up and built a
regression list and posted it to LKML. What's wrong with that ?

	tglx

Re: Linux 2.6.21

[Diego Calleja]

El Sun, 29 Apr 2007 23:10:28 +0200, Adrian Bunk <bunk@stusta.de> escribió:

> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?

AFAIK, submitting its contents to the list periodically CCing the developers,
like you did with your lists.

If developers care to fix it or not or how much Linus cares about that list before
releasing a new version is another question. I think it's useful because it makes
those bugs look more important than the 1600 stored in the bugzilla...it won't
help to fix those 1600, but it attracts some attention over the "release critical"
ones and encourages developers to fix them, even if not all of them get fixed.

I don't think you can do many other things to get as much bugs fixed as possible,
unless we reward bug fixers with weekends in the Playboy mansion. I think the
fundamental question here is: is there a way to make hackers follow and fix _all_
the bugs? I'd love it was possible, but AFAIK all the projects that have tried to
be ultra-stable and have adopted a policy to fullfill such goal have fallen behind
of competing projects that cared more about working in improving their software.

Re: Linux 2.6.21

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: Sun, 29 Apr 2007 23:26:43 +0200

> > I reported a bug that eats people's hard disks due to a bug
> > in the X.ORG PCI support code on sparc, NOBODY has fixed
> > the bug in 2 years even though a full bugzilla entry with
> > even a full patch fix is in there.
> 
> Well but at least they could find it again if they wanted.
> If you sent it by email and it had gotten lost for some reason
> (nobody interested, which seems to be the real issue here) then
> it would be lost forever.

WRONG!  If I have sent it to the main developer list the damn patch
would be applied by now.

WHY?

BECAUSE EMAIL ENGAGES PEOPLE AND BUGZILLA DOES NOT!

Nobody looks at the bugzilla because there is too much junk in there
to make the signal any useful to search for, there's simply too much
noise.

Re: Linux 2.6.21

[Arkadiusz Miskiewicz]

On Sunday 29 of April 2007, Tomasz Chmielewski wrote:

> > I reported a bug that eats people's hard disks due to a bug
> > in the X.ORG PCI support code on sparc, NOBODY has fixed
> > the bug in 2 years even though a full bugzilla entry with
> > even a full patch fix is in there.
>
> And how fast was the bug fixed when you posted it to the X.ORG list?

Not so long ago I got such reply on xorg@ ml:

,,Sorry, without a bugzilla entry, I lost your bug report. ''

xorg people want bugzilla reports (well, they don't fix them anyway, too).

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

Re: Linux 2.6.21

[Francois Romieu]

Diego Calleja <diegocg@gmail.com> :
[...]
> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

The rtl8139 entry is outdated.

-- 
Ueimor

Re: Linux 2.6.21

[Dave Jones]

On Sun, Apr 29, 2007 at 02:05:42AM +0200, Adrian Bunk wrote:
 > On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
 > >...
 > > What's the difference between bugzilla and lkml.org? Both have search 
 > > buttons. Both archive the old stuff. Both can be pointed to.
 > 
 > Mailing lists don't track bugs.

There's no reason they can't.  Store them in folders 'fixed' 'pending'
'notabug' etc. Move mails between them when states change.
reply-to them when necessary. Bounce them. Add people to Ccs. etc. etc.

The only remaining piece of the puzzle is "how does everyone see the
states of the various pools", which could be solved in a number of
ways, daily uploads to a place on kernel org for eg.

Hell, you could store them in a git tree if it came to it.
That would also solve the problem for those with an aversion
to web browsers to see bugs :-)

	Dave

-- 
http://www.codemonkey.org.uk

Re: Linux 2.6.21

[David Lang]

On Sun, 29 Apr 2007, Dave Jones wrote:

> On Sun, Apr 29, 2007 at 02:05:42AM +0200, Adrian Bunk wrote:
> > On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
> > >...
> > > What's the difference between bugzilla and lkml.org? Both have search
> > > buttons. Both archive the old stuff. Both can be pointed to.
> >
> > Mailing lists don't track bugs.
>
> There's no reason they can't.  Store them in folders 'fixed' 'pending'
> 'notabug' etc. Move mails between them when states change.
> reply-to them when necessary. Bounce them. Add people to Ccs. etc. etc.

and archive off the old reports into folders as well. While I don't think that 
closeing a report becouse it hasn't seen an update in a week is reasonable, 
closeing a report that hasn't seen an update or retest in a couple 2.6.x 
releases definantly is (this is a 4-6 month period at the rate of change in the 
kernel the odds are pretty good that the code is no longer the same at that 
point)

> The only remaining piece of the puzzle is "how does everyone see the
> states of the various pools", which could be solved in a number of
> ways, daily uploads to a place on kernel org for eg.

if people are interested in doing this it's not a technicly hard problem.

get bugs to a 'bug maintainer' e-mail address. this can either be a copy of the 
full lkml firehose, or volunteers who pull selected messages from the list, with 
a server that discards duplicates it's safe to have multiple people bounce the 
same message to the bug list, if they forward the message (to add a comment on 
who they think may need to work on it) then it will take manual weeding out of 
duplicates

have an IMAP server available for this address. make it read-only to the world 
and read-write to a list of volunteers who will sort the messages.

sort the messages into the different catagories, with a subfolder for each bug 
(note: the structure at this point is arbtrary, the volunteers can further 
orginise the folders)

with a good IMAP server you can add the address of the particular bug folder to 
the cc list if you want (bugs+drivers.ethernet.3com.123@kernel.org for a complex 
example) to have the follow-ups go directly into that folder.

now the problem with this is that developers would still have to look at it for 
an overview (the volunteers would still need to copy the developers when they 
create a new bug)

the cyrus mail server will scale well for this sort of thing, and I beleive that 
it can also export the mailboxes as news feeds as well (I haven't ever 
configured this so I can't say the exact details)

for searching, just make it available to google (I'm sure google would cooperate 
with this sort of thing to find the best way to do it)

the key is to find people to volunteer to do the sorting. the hosting of this is 
pretty simple (for that matter, I'll offer to host it if nobody else steps up)

David Lang

> Hell, you could store them in a git tree if it came to it.
> That would also solve the problem for those with an aversion
> to web browsers to see bugs :-)

git is not particulary good for searching the contents of things.

on that note, I wonder if google has a git:// search as part of theit git code 
project? if not they should.

Re: Linux 2.6.21

[Andi Kleen]

> I reported a bug that eats people's hard disks due to a bug
> in the X.ORG PCI support code on sparc, NOBODY has fixed
> the bug in 2 years even though a full bugzilla entry with
> even a full patch fix is in there.

Well but at least they could find it again if they wanted.
If you sent it by email and it had gotten lost for some reason
(nobody interested, which seems to be the real issue here) then
it would be lost forever.

Of course a database is not a silver bullet by itself, it still
needs people to use it correctly and then actually work on the bugs.

A database just a tool to move some work humans are bad at (keeping track 
of a lot of data and deriving trends out of it) to a a computer which is 
much better at this.

-Andi

Re: Linux 2.6.21

[Michal Piotrowski]

On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Sun, Apr 29, 2007 at 11:16:51PM +0200, Michal Piotrowski wrote:
> > On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> >> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> >
> > It's for -stable team.
> >...
>
> Did Greg or Chris say they have spare time for going through this list?

It's not a list "hello -stable team, here are a new regressions, please fix it".

> cu
> Adrian

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > 
> > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> 
> OK, how do you suggest to track bugs in a way that doesn't suck?

I've tried to explain.

Bugzilla can be one _part_ of it, but anybody who thinks it's the "main 
part" is really not being realistic. It's too cumbersome, and it's too 
stupid. 

Quite frankly, "lkml + google" is probably in many ways a *better* way to 
search for problems. But yes, some manual smarts (and the _occasional_ 
pointer to bugzilla) is probably currently the only option.

Exactly because I don't think anybody has shown any better automation than 
bugzilla. But that doesn't make bugzilla "the One Choice". That's not how 
it works. If there is no automation, manual tracking is still better than 
*crap* automation.

> Bug reports to linux-kernel have the big problem that they are lost if 
> no developer immediately picks them up.

..and this is different from bugzilla exactly _how_?

Those things are lost too. As you yourself have pointed out. The fact that 
you can search for them is _exactly_ as relevant as the fact that you can 
search for lkml on google.

> What I do know is that the majority of them has never been proper 
> debugged by a kernel developer knowing the subsystem in question.

And you blame the developers, but not bugzilla? Why are you so unable to 
see bugzilla as part of the *cause* of the problem? You're perfectly happy 
to blame other things, but bugzilla is somehow above blame?

		Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 11:16:51PM +0200, Michal Piotrowski wrote:
> On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
>> On Sun, Apr 29, 2007 at 10:56:57PM +0200, Diego Calleja wrote:
>> > El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> 
>> escribió:
>> >
>> > > Bugzilla might not be perfect, but it works and it's better than doing
>> > > it by hand.
>> >
>> > The good thing about the wiki is that it doesn't exclude bugzilla. It's
>> > just a "regressions list", it doesn't intends to replace bugzilla. If a 
>> bug
>> > doesn't gets fixed for a while, I don't think it's very useful to keep 
>> it
>> > forever in the list like you do in the bugzilla, because I don't think
>> > it's possible to fix every single bug, and it steals you time to fix 
>> bugs
>> > that you are able to fix.
>> >
>> > It's not great but it's the best clone of you we've found 8)
>>
>> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
>
> It's for -stable team.
>...

Did Greg or Chris say they have spare time for going through this list?

> Regards,
> Michal

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Michal Piotrowski]

On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Sun, Apr 29, 2007 at 10:56:57PM +0200, Diego Calleja wrote:
> > El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> escribió:
> >
> > > Bugzilla might not be perfect, but it works and it's better than doing
> > > it by hand.
> >
> > The good thing about the wiki is that it doesn't exclude bugzilla. It's
> > just a "regressions list", it doesn't intends to replace bugzilla. If a bug
> > doesn't gets fixed for a while, I don't think it's very useful to keep it
> > forever in the list like you do in the bugzilla, because I don't think
> > it's possible to fix every single bug, and it steals you time to fix bugs
> > that you are able to fix.
> >
> > It's not great but it's the best clone of you we've found 8)
>
> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?

It's for -stable team.

I didn't saw any 2.6.22-stuff regression yet.

>
> cu
> Adrian

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 10:56:57PM +0200, Diego Calleja wrote:
> El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> escribió:
> 
> > Bugzilla might not be perfect, but it works and it's better than doing 
> > it by hand.
> 
> The good thing about the wiki is that it doesn't exclude bugzilla. It's
> just a "regressions list", it doesn't intends to replace bugzilla. If a bug
> doesn't gets fixed for a while, I don't think it's very useful to keep it
> forever in the list like you do in the bugzilla, because I don't think
> it's possible to fix every single bug, and it steals you time to fix bugs
> that you are able to fix.
> 
> It's not great but it's the best clone of you we've found 8)

What exactly is the purpose of the 2.6.21 regressions list in the Wiki?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 01:33:16PM -0700, Linus Torvalds wrote:
> 
> 
> On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > 
> > The kernel Bugzilla currently contains 1600 open bugs.
> 
> Adrian, why do you keep harping on this, and ignoring reality?
> 
> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.

OK, how do you suggest to track bugs in a way that doesn't suck?

Bug reports to linux-kernel have the big problem that they are lost if 
no developer immediately picks them up.

> How many of those are interesting and valid? How many of them are 
> relevant? How many of them are duplicates?
> 
> You don't know. Nobody does. So why do you bother reporting that number?

What I do know is that the majority of them has never been proper 
debugged by a kernel developer knowing the subsystem in question.

> That number is exactly as relevant as the number of dog-hairs on our couch 
> ("in the millions"). An impressively large number, definitely uncountable, 
> and definitely also not relevant to anythign at all. Not at all unlike 
> that "1600 open bugs" number that you bring up.
> 
> Do you think the number of dog-hairs on our couch is an argument for or 
> against people trying to track regressions? If not, why do you keep 
> bringing up bugzilla?

I tracked 2.6.21-rc regressions, and I do not scale for higher numbers 
of bugs. When I had to track 36 known regressions it was a real 
nightmare.

Bugzilla tracks regressions and scales for higher numbers of bugs.

Let me ask you two questions:
- Do you think regression tracking makes any sense at all?
- If yes, which scalable way of regression tracking would in your
  opinion not suck?

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Diego Calleja]

El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> escribió:

> Bugzilla might not be perfect, but it works and it's better than doing 
> it by hand.

The good thing about the wiki is that it doesn't exclude bugzilla. It's
just a "regressions list", it doesn't intends to replace bugzilla. If a bug
doesn't gets fixed for a while, I don't think it's very useful to keep it
forever in the list like you do in the bugzilla, because I don't think
it's possible to fix every single bug, and it steals you time to fix bugs
that you are able to fix.

It's not great but it's the best clone of you we've found 8)

Re: Linux 2.6.21

[Alexey Dobriyan]

On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> [For example, you can create a bugzilla entry with a link to the lkml.org copy
> of the relevant message, so why to require the reporter to file the report with
> the bugzilla himself?]

Last time I did this, bugzilla at osdl.org won't let me add original
reporter to goddamn CC list. It would be el neat, because not everyone
followed instructions and forwarding emails between reporter and
bugzilla sucks.

Re: Linux 2.6.21

[Tomasz Chmielewski]

David Miller wrote:

>> Don't think that's true. There are plenty of projects who only
>> accept bugs through bugzilla (mozilla, various distributions, etc.) 
>> and I don't see any evidence of your claim being true.
> 
> That explains why my bugs don't get looked at for months if
> not years when I submit them to such projects.
> 
> I reported a bug that eats people's hard disks due to a bug
> in the X.ORG PCI support code on sparc, NOBODY has fixed
> the bug in 2 years even though a full bugzilla entry with
> even a full patch fix is in there.

And how fast was the bug fixed when you posted it to the X.ORG list?


> Bugzilla sucks, emails rules because it is in your face and
> gets people to work on things.

Bugzilla can be configured to send emails, too (to the list for a newly 
reported bug for example).


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
>...
> But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> From lkml.org, for example.  Why don't we use that?  The only missing piece
> is the 'keep a list' thing, but that's not a rocket science, IMHO.
> 
> [For example, you can create a bugzilla entry with a link to the lkml.org copy
> of the relevant message, so why to require the reporter to file the report with
> the bugzilla himself?]
> 
> _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> keep track of each thread separately, so you can browse any of them at any
> time.  In particular, you can see the _history_ of each bug report sent to LKML
> if you have a link to any message in its thread.
> 
> Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> you'll even be able to use the lkml.org archives plus wget and a couple of
> shell scripts to cherry pick the links to all bug reports sent to the list
> within a given time interval.
> 
> All of this functionality is out there already.
>...

How can I get the functionality "show me all unfixed SATA bugs"?

That's one of the important functionalities of every bug tracking 
system.

> Greetings,
> Rafael

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Simon Arlott]

On 29/04/07 19:09, Andi Kleen wrote:
>>  - a lot of reporters will not use bugzilla, because it's damn 
>>    inconvenient even for reporting. If you propose something that uses 
> 
> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.

These projects are probably losing plenty of trivial bug reports 
from people who shouldn't have to register with a bug tracker and 
fill in a long form every time to submit a short bug report.

> Sure there will be always people who cannot be bothered
> to use any kind of interface for bugs, but then

Most of the time bugzilla appears to be a great way to ignore 
bugs. Every large project seems to have one with bugs that 
stay open for years - bugzilla's own inability to quick search 
without javascript[1] being a good example (it's fixed now). 
If no one can get around to fixing reported bugs why should 
anyone bother submitting more?

They even released their software with a link to the bug if 
you tried to do a quick search without javascript:
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=70907
"THIS BUG IS FIXED IN BUGZILLA 2.22. D..."

> these are unlikely to stay on board during a longer 
> remote debugging q'n'a session either. So those people
> can be just ignored; they essentially don't exist in
> the bug report universe.

How will the people you ignore get their bugs fixed?

> I don't think the "keep it in Andrew's/Adrian's head" method
> is going to scale longer term at least (and one of them has 
> already thrown in the towel) 

People are doing something about this, at least for tracking 
the regressions at http://kernelnewbies.org/known_regressions 
(which is read-only unless you register, people have already 
commented on this).

> The "send it to a gigantic mailing list and hope someone catches
> it" method also doesn't seem to be that great. At least there
> are lots of lost reports in my experience this way.

I have had experience of this with the dvb mailing list (this was 
a couple of months before their news post referencing bugzilla 
too), even when I included a patch to fix it.

Someone suggested another way to fix a bug and I replied that it 
worked - that fix never went anywhere else and other people could 
be having the same problem today because it's still not been 
applied anywhere. (I will submit a -1 +1 patch myself when 
I have time this week).

> I suspect the real reason is more "Linus doesn't like web interfaces
> for no particular good reason". Not much can be done about that.
> Well perhaps someone can write a gopher based bugzilla interface 
> or something to solve that instead @) 

Bugzilla's advanced search interface is far too crowded, that 
should be clear to anyone. The simpler search usually isn't much 
use because either it finds too many bugs or "zarro boogs" and 
you're left wondering if the bug is there or not.

-- 
Simon Arlott

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> 
> The kernel Bugzilla currently contains 1600 open bugs.

Adrian, why do you keep harping on this, and ignoring reality?

Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.

How many of those are interesting and valid? How many of them are 
relevant? How many of them are duplicates?

You don't know. Nobody does. So why do you bother reporting that number?

That number is exactly as relevant as the number of dog-hairs on our couch 
("in the millions"). An impressively large number, definitely uncountable, 
and definitely also not relevant to anythign at all. Not at all unlike 
that "1600 open bugs" number that you bring up.

Do you think the number of dog-hairs on our couch is an argument for or 
against people trying to track regressions? If not, why do you keep 
bringing up bugzilla?

		Linus

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 21:14, Andi Kleen wrote:
> On Sun, Apr 29, 2007 at 08:50:55PM +0200, Rafael J. Wysocki wrote:
> > On Sunday, 29 April 2007 19:37, Andi Kleen wrote:
> > > > My personal experience with bugzilla is that it's very unfriendly to
> > > > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > > > debug patches, system information etc., but not for _reporting_ new ones.
> > > 
> > > What did you find unfriendly? 
> > 
> > - You are required to select a category and 'component' for your report, which
> > often is difficult (especially if you're not a kernel expert)
> 
> Usually there is other and then someone else figures it out.
> 
> > - You need to have a bugzilla account (or to create one, if you don't)
> > - If you want to add an address to the CC list, it must be known to bugzilla
> > and there's no (obvious) way to check which addresses are known (bugzilla
> > rejects the report if there's a 'wrong' email address in the list)  [IMO this is
> > really really broken.]
> 
> The Novell bugzilla actually has that fixed. You have a search email button
> to look up addresses.  Perhaps that feature will be ported someday into
> the kernel.org one (I would like to have it too) 
> 
> > - You are asked to provide many details that need not be relevant and casual
> > reporters don't know that they can skip this part
> > - Attaching files is tedious and referring to attachments unintuitive
> 
> Anyways that are mostly all detail (except the registration requirement) that
> could be probably all easily fixed.
> 
> > And I think they are two _totally_ conceptually different things.  You report
> > a bug to let somebody know that there's a problem and this doesn't necessarily
> 
> The problem is we need a way to route those reports to the right people.
> Routing it to a single person or broadcasting it just doesn't scale.
> And the best way I know of is to use some database that keeps track of the state.
> 
> That is what bugzilla is essentially.
> 
> > For this reason there should be a simple means of filing initial bug reports
> > with someone to look at them and forward them to appropriate people who will
> > decide if the problem needs to be tracked.  If they do, it's time to use
> > bugzilla.  Not earlier.
> 
> The only sane way to do that would be to save them somewhere and keep
> a list and then let a group of people process them.

But emailed reports _are_ saved anyway and we _know_ how to get a copy.
>From lkml.org, for example.  Why don't we use that?  The only missing piece
is the 'keep a list' thing, but that's not a rocket science, IMHO.

[For example, you can create a bugzilla entry with a link to the lkml.org copy
of the relevant message, so why to require the reporter to file the report with
the bugzilla himself?]

_Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
keep track of each thread separately, so you can browse any of them at any
time.  In particular, you can see the _history_ of each bug report sent to LKML
if you have a link to any message in its thread.

Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
you'll even be able to use the lkml.org archives plus wget and a couple of
shell scripts to cherry pick the links to all bug reports sent to the list
within a given time interval.

All of this functionality is out there already.

> Hmm, wait... sounds like bugzilla, doesn't it?
> 
> > You are right, email is not suitable for tracking bugs.

Well, now, I think that even this statement is not exactly right. :-)

> > Still, it works quite well as a means of sending initial reports.
>
> I disagree. It works small scale but does not really scale well.

IMO that depends on how you handle it.

Greetings,
Rafael

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 09:40:07PM +0200, Diego Calleja wrote:
>...
> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

This list currently contains 29 known regressions.
Someone has to manually add them.
Someone has to manually track the status of all of them.
Someone has to manually group related regressions (e.g. in the same 
subsystem) together.

The kernel Bugzilla currently contains 1600 open bugs.

Maintaining a regression list by hand was really a pain when during 
2.6.21-rc we had 36 known regressions.

Any approach that does not involve some kind of tracker with some kind 
of database simply doesn't scale.

Bugzilla might not be perfect, but it works and it's better than doing 
it by hand.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: Sun, 29 Apr 2007 20:09:09 +0200

> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.

That explains why my bugs don't get looked at for months if
not years when I submit them to such projects.

I reported a bug that eats people's hard disks due to a bug
in the X.ORG PCI support code on sparc, NOBODY has fixed
the bug in 2 years even though a full bugzilla entry with
even a full patch fix is in there.

Bugzilla sucks, emails rules because it is in your face and
gets people to work on things.

Re: Linux 2.6.21

[Michal Piotrowski]

Hi Diego,

On 29/04/07, Diego Calleja <diegocg@gmail.com> wrote:
[..]
> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

Thanks for your help with this!

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Diego Calleja]

So far, it seems that most of people's opinion WRT to bug reporting and trackingcan
be divided into 2 groups:

- People who argues (and they're right) that bugzilla and web interfaces in general
  suck and that email + an "Adrian-like" solution works better

- People who argues that a bug tracker better than a mailing list is absolutely
  needed (and they're right). They also argue that while bugzilla sucks, it's
  better than nothing.

There's a common point between both groups: bugzilla sucks. The ideal
solution would be to replace bugzilla with some alternative and better
opensource bug tracking software, but I doubt it exists (there must be a
reason why everybody uses bugzilla). A good bug tracker should feel like
it makes your work easier, instead of making you feel like you're wasting
time (which is what bugzilla does)

I don't see why a web interface bug tracker should be bad for bug tracking,
as long as it's good and integrates 100% in the mailing lists. In my humble
opinion the "perfect" bug tracker for Linux should be something like this:
- Has a email interface (like the Debian bug tracking database).
- Has a web interface that completely follows the email threads
   (reading/posting), but make the comments real emails, not just
    database fields.
If done well (unlike the current bugzilla-to-email hack), it should possible
to do many nice things, like add a lkml bug report to the bug tracking
database (which shouldn't be a "real" database, but just an lkml mail
archive with a list of message IDs that are considered a bug and its state)
by just replying the thread, CCing the bug tracker and telling him to include
the thread in the database.

So unless someone is willing to write such tool (which I doubt, since it
doesn't looks easy), all this discussion seems pointless, and we should
stick with this http://kernelnewbies.org/known_regressions page
which is showing to be quite useful :)

Re: Linux 2.6.21

[Russell King]

On Sun, Apr 29, 2007 at 08:09:09PM +0200, Andi Kleen wrote:
> >  - a lot of reporters will not use bugzilla, because it's damn 
> >    inconvenient even for reporting. If you propose something that uses 
> 
> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.

There's an ARM category in the kernel.org bugzilla.  Folk are completely
free to submit ARM bugs to either the (closed) mailing list or bugzilla.

99.99999999% of bug reports in the ARM community come via the mailing
list.  I think to date there's been about 10 bugzilla entries since
Feb 2004.

This is inspite of me linking to the kernel.org bugzilla from my
website.

So it seems that virtually all the folk involved with the ARM kernel,
given a completely free choice, _prefer_ to send email-based bug
reports over touching bugzilla.

That's quite a different metric to projects forcing bugzilla on people,
and I'd say is a more valid metric to gauge whether bugzilla is really
suitable.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Linux 2.6.21

[Andi Kleen]

On Sun, Apr 29, 2007 at 08:50:55PM +0200, Rafael J. Wysocki wrote:
> On Sunday, 29 April 2007 19:37, Andi Kleen wrote:
> > > My personal experience with bugzilla is that it's very unfriendly to
> > > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > > debug patches, system information etc., but not for _reporting_ new ones.
> > 
> > What did you find unfriendly? 
> 
> - You are required to select a category and 'component' for your report, which
> often is difficult (especially if you're not a kernel expert)

Usually there is other and then someone else figures it out.

> - You need to have a bugzilla account (or to create one, if you don't)
> - If you want to add an address to the CC list, it must be known to bugzilla
> and there's no (obvious) way to check which addresses are known (bugzilla
> rejects the report if there's a 'wrong' email address in the list)  [IMO this is
> really really broken.]

The Novell bugzilla actually has that fixed. You have a search email button
to look up addresses.  Perhaps that feature will be ported someday into
the kernel.org one (I would like to have it too) 

> - You are asked to provide many details that need not be relevant and casual
> reporters don't know that they can skip this part
> - Attaching files is tedious and referring to attachments unintuitive

Anyways that are mostly all detail (except the registration requirement) that
could be probably all easily fixed.

> And I think they are two _totally_ conceptually different things.  You report
> a bug to let somebody know that there's a problem and this doesn't necessarily

The problem is we need a way to route those reports to the right people.
Routing it to a single person or broadcasting it just doesn't scale.
And the best way I know of is to use some database that keeps track of the state.

That is what bugzilla is essentially.

> For this reason there should be a simple means of filing initial bug reports
> with someone to look at them and forward them to appropriate people who will
> decide if the problem needs to be tracked.  If they do, it's time to use
> bugzilla.  Not earlier.

The only sane way to do that would be to save them somewhere and keep
a list and then let a group of people process them.

Hmm, wait... sounds like bugzilla, doesn't it?

> You are right, email is not suitable for tracking bugs.  Still, it works quite
> well as a means of sending initial reports.

I disagree. It works small scale but does not really scale well.

-Andi

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 20:09, Andi Kleen wrote:
> >  - a lot of reporters will not use bugzilla, because it's damn 
> >    inconvenient even for reporting. If you propose something that uses 
> 
> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.
> 
> Sure there will be always people who cannot be bothered
> to use any kind of interface for bugs, but then
> these are unlikely to stay on board during a longer 
> remote debugging q'n'a session either. So those people
> can be just ignored; they essentially don't exist in
> the bug report universe.
> 
> Anyways it only works if people are willing to use it too and there
> are enough people who maintain bugs (aka ask questions to find out
> who to reassign, prune old bugs etc.)  If that's not there then
> it won't work well obviously, like it is currently the case.
> 
> I don't think the "keep it in Andrew's/Adrian's head" method
> is going to scale longer term at least (and one of them has 
> already thrown in the towel) 
> 
> The "send it to a gigantic mailing list and hope someone catches
> it" method also doesn't seem to be that great. At least there
> are lots of lost reports in my experience this way.

This, actually, might work if the report is 'flagged' in a specific way.

For example, if there's a message sent to LKML with a combination of '[BUG]'
and 'suspend' in the subject, I have no problems whatsoever with spotting it. ;-)

Greetings,
Rafael

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Rafael J. Wysocki wrote:
>
> - You are required to select a category and 'component' for your report, which
>   often is difficult (especially if you're not a kernel expert)
> - You need to have a bugzilla account (or to create one, if you don't)

Amen.

Both of those are show-stoppers. It may be ok for kernel developers, but 
it's not good for random people. It's one of the reasons I don't generally 
use bugzilla for other projects - if they require me to sign up etc, they 
can take care of their own bugs.

(For the same reason I consider closed mailing lists to be useless for 
bugreports. If you have to be a member to send email, it's not a 
bug-report thing, it's just a secret society).

I realize that spam is a problem, but there are better spam solutions than 
alienating the people who just want to send a report.

Also, I don't know if anybody has ever tried to avoid sending duplicate 
bugs, but every time I use bugzilla to send a bug-report (which I do for 
things like FC7 live CD's breaking etc where I care enough, and I have a 
bugzilla account on that bugzilla _anyway_ for other reasons), I am ready 
to _kill_ somebody whenever I see that "humorous" 

	zarro bugs found

message, after it made it almost impossible for me to even figure out how 
to do a good search in the first place!

So I gnash my teeth, and fill in the bug report anyway, and if it's a 
duplicate becasue bugzilla didn't have any sane way to get any kind of 
overview at all, hey, it's a duplicate. Nothing I can do about it.

And the reason I gnash my teeth is that I realize that because I can't 
even get any overview of the bugzilla entries as a random submitter (and 
dammit, I probably have a better idea of what the problem might be than 
most people), I sure as hell can't expect a random user to be better. So I 
bet my bad reports are a lot better than the average, and a lot of people 
probably just give up and don't report anything at all.

		Linus

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 19:37, Andi Kleen wrote:
> > My personal experience with bugzilla is that it's very unfriendly to
> > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > debug patches, system information etc., but not for _reporting_ new ones.
> 
> What did you find unfriendly? 

- You are required to select a category and 'component' for your report, which
often is difficult (especially if you're not a kernel expert)
- You need to have a bugzilla account (or to create one, if you don't)
- If you want to add an address to the CC list, it must be known to bugzilla
and there's no (obvious) way to check which addresses are known (bugzilla
rejects the report if there's a 'wrong' email address in the list)  [IMO this is
really really broken.]
- You are asked to provide many details that need not be relevant and casual
reporters don't know that they can skip this part
- Attaching files is tedious and referring to attachments unintuitive

> While I also cannot say I love it I don't find it any less unfriendly
> than an email.

That depends (please see below).

> Besides the primary point of bug tracking is not to be friendly
> to someone, but to (a) fix the bugs and (b) know how many bugs
> there for a given release. Any replacement would need to solve
> this problem too.

For _tracking_ bugs, the bugzilla is more-or-less suitable.
For _reporting_ bugs, IMHO, it's not.

And I think they are two _totally_ conceptually different things.  You report
a bug to let somebody know that there's a problem and this doesn't necessarily
mean that the problem has to be tracked.  It may be very simple and immediately
resolvable, in which case registering it in bugzilla is a loss of time and
resources.  If the problem _turns_ _out_ to be difficult, _then_ it'll need to
be tracked, but usually it's not known whether or not this is the case until
someone 'in the know' looks at the initial report.

For this reason there should be a simple means of filing initial bug reports
with someone to look at them and forward them to appropriate people who will
decide if the problem needs to be tracked.  If they do, it's time to use
bugzilla.  Not earlier.

> Email does not solve it as far as I can see.

You are right, email is not suitable for tracking bugs.  Still, it works quite
well as a means of sending initial reports.

Greetings,
Rafael

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> 
> I suspect the real reason is more "Linus doesn't like web interfaces
> for no particular good reason". Not much can be done about that.

Right. Dig your head in the sand, and ignore all the other people who 
piped up and said they hate bugzilla too, and find email much more 
convenient. It's "just Linus" in your dreamworld.

		Linus

Re: Linux 2.6.21

[Andi Kleen]

>  - a lot of reporters will not use bugzilla, because it's damn 
>    inconvenient even for reporting. If you propose something that uses 

Don't think that's true. There are plenty of projects who only
accept bugs through bugzilla (mozilla, various distributions, etc.) 
and I don't see any evidence of your claim being true.

Sure there will be always people who cannot be bothered
to use any kind of interface for bugs, but then
these are unlikely to stay on board during a longer 
remote debugging q'n'a session either. So those people
can be just ignored; they essentially don't exist in
the bug report universe.

Anyways it only works if people are willing to use it too and there
are enough people who maintain bugs (aka ask questions to find out
who to reassign, prune old bugs etc.)  If that's not there then
it won't work well obviously, like it is currently the case.

I don't think the "keep it in Andrew's/Adrian's head" method
is going to scale longer term at least (and one of them has 
already thrown in the towel) 

The "send it to a gigantic mailing list and hope someone catches
it" method also doesn't seem to be that great. At least there
are lots of lost reports in my experience this way.

I suspect the real reason is more "Linus doesn't like web interfaces
for no particular good reason". Not much can be done about that.
Well perhaps someone can write a gopher based bugzilla interface 
or something to solve that instead @)

-Andi

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> 
> Besides the primary point of bug tracking is not to be friendly
> to someone, but to (a) fix the bugs and (b) know how many bugs
> there for a given release. Any replacement would need to solve
> this problem too.
> 
> Email does not solve it as far as I can see.

Email fixes a _lot_ more bugs than bugzilla does. 

End of story. I don't think anybody who cannot accept that UNDENIABLE FACT 
should even participate in this discussion. Wake up and look at all the 
bugs we fix - most of them have never been in bugzilla.

That's a FACT.

Don't go around ignoring reality.

		Linus

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> On Sun, Apr 29, 2007 at 09:07:43AM -0700, Linus Torvalds wrote:
> >
> > Yes. But not using bugzilla.
> 
> And what would you use instead? 

Didn't you even *read* my email?

I already told you: we have real bugs getting reported and fixed that 
don't hit bugzilla or any bugtracker AT ALL.

This is not a "all or nothing" situation. There is absolutely _zero_ real 
reason to think that everything has to go through bugzilla.

> > I don't know if you've noticed already, but I'm not the only one that 
> > doesn't have a very high opinon of bugzilla ;)
> 
> Sure, but do you have alternatives? 

Yes. The ones that *work*. Plain email is preferably over bugzilla 90% of 
the time. 

But quite frankly, if you think you can make bugzilla work (and realize 
that a lot of people will _not_ be looking at it or reporting bugs into 
it), go ahead. I don't care. The only think I care about is *REALITY*, and 
that means:

 - a lot of reporters will not use bugzilla, because it's damn 
   inconvenient even for reporting. If you propose something that uses 
   _only_ bugzilla, you'd better also have the people who enter other 
   peoples bugreports into there.

 - a lot of developers will not use bugzilla, because it's even more 
   inconvenient for developers, with no sane ways to interact with the 
   right people. So if you propose using bugzilla, you'd really better 
   have the manpower to turn bugzilla into emails (and no, the bugzilla 
   cc list etc is _not_ the primary one - the email cc's are the primary 
   ones, because that's where it is much easier to bring in new people)

In other words, anything that thinks that bugzilla is "primary" is just 
broken. It can be a _part_ of the thing, but drop the belief of it being a 
primary tracker. It's just too inconvenient.

		Linus

Re: Linux 2.6.21

[Andi Kleen]

> My personal experience with bugzilla is that it's very unfriendly to
> reporters.  IMHO it's suitable for tracking unresolved problems along with
> debug patches, system information etc., but not for _reporting_ new ones.

What did you find unfriendly? 

While I also cannot say I love it I don't find it any less unfriendly
than an email.

Besides the primary point of bug tracking is not to be friendly
to someone, but to (a) fix the bugs and (b) know how many bugs
there for a given release. Any replacement would need to solve
this problem too.

Email does not solve it as far as I can see.

-Andi

Re: Linux 2.6.21

[Andi Kleen]

On Sun, Apr 29, 2007 at 09:07:43AM -0700, Linus Torvalds wrote:
> 
> 
> On Sun, 29 Apr 2007, Andi Kleen wrote:
> > Linus Torvalds <torvalds@linux-foundation.org> writes:
> > > 
> > > The thing is, bugzilla is totally broken because it's designed to help 
> > > track bugs, but it's *not* designed to actually handle the much harder 
> > > problem, which is to actually get the *right* developers to be aware 
> > > of the *right* bugs!
> > 
> > This means we need people who figure out who to assign bugs too.
> > Aka bugmasters.
> 
> Yes. But not using bugzilla.

And what would you use instead? 

> 
> I don't know if you've noticed already, but I'm not the only one that 
> doesn't have a very high opinon of bugzilla ;)

Sure, but do you have alternatives? 

> What works is somebody who is a bugmaster, and it doesn't really matter 
> *what* bug tracker he points to (bugzilla being one of the possibilities, 
> although not necessarily the best, and absolutely NOT the only choice), 
> and turn them into emails. Once they are emails, bugzilla can track them.

So you want to do the work that should be done by a computer in
a database done by a human.  Does not sound particularly efficient.

> > In theory it could be nearly automated. Figure out what files related
> > to the bug and assign to the last 5 people who submitted patches
> > for them and/or signed off. 
> 
> I do think you're pretty optimistic. I think it's true for trivial driver 
> bugs, but even for trivial driver bugs the initial report is often not 
> enough to pinpoint the driver. 
> 
> Let's take the sis900 bug as an example (not because I want to rag on that 
> being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's 
> a good case of something really really easy - and if that easy case isn't 
> handled trivially and obviously, then the bug-reporting doesn't work).
> 
> In that case, the initial report was (condensed version, but fairly 
> accurate): "system hangs on boot at random points. 2.6.21-rc7 worked 
> well".
> 
> Now, realistically, if that entry had been in bugzilla, what would you do?

You need a few people in bugzilla that ask the questions to narrow it
down (= bugmasters). e.g. the opensuse bugzilla works this way :- 
everything new gets assigned to a few screening people who 
ask some questions and then reassign to the right people.

> Would bugzilla have helped? HELL NO. It would have been a disaster. It 
> would have wasted reporter time, it would have wasted developer time, and

An unmaintained bugzilla yes. A well maintained one would have someone
asking them the (often quite repetive questions) to narrow it down
to a subsystem or a driver. And then it could have been assigned.

> I think both of these are good ideas, but I don't think it's enough. There 
> must be some better form of bug tracking than bugzilla. Some relly 
> distributed way of letting people work together, *without* having to 
> congregate on some central web-site kind of thing. A "git for bugs", where 
> you can track bugs locally and without a web interface.
> 
> And I think it would be something much closer to "distributed email mbox 
> with status tracking facilities", _not_ a web clicky interface.

There have been a couple of email thread trackers; like jitterbug -- 
in fact bugzilla can do that with an email interfaces. But in my experience 
they don't work well because a bug tracking system has slightly different
requirements than normal emails (e.g. it wants you to roughly
stay on topic for the current bug). With email people always
forget that and in the end you end up with lots of stuff in there
not related to the bug at all.

Also with distributed solutions it would be hard to get
a global "how many regressions do we really have right now" statistic,
which is fairly important.

The web interface is slow and ugly but at least it puts
the people in the right mind set for this unlike email.
And it gives you a central point to get an overview of the bugs.

Anyways I'm sure bugzilla could be improved, I just don't know
of anything better currently.

-Andi

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 18:07, Linus Torvalds wrote:
> 
> On Sun, 29 Apr 2007, Andi Kleen wrote:
> > Linus Torvalds <torvalds@linux-foundation.org> writes:
> > > 
> > > The thing is, bugzilla is totally broken because it's designed to help 
> > > track bugs, but it's *not* designed to actually handle the much harder 
> > > problem, which is to actually get the *right* developers to be aware 
> > > of the *right* bugs!
> > 
> > This means we need people who figure out who to assign bugs too.
> > Aka bugmasters.
> 
> Yes. But not using bugzilla.
> 
> I don't know if you've noticed already, but I'm not the only one that 
> doesn't have a very high opinon of bugzilla ;)
> 
> What works is somebody who is a bugmaster, and it doesn't really matter 
> *what* bug tracker he points to (bugzilla being one of the possibilities, 
> although not necessarily the best, and absolutely NOT the only choice), 
> and turn them into emails. Once they are emails, bugzilla can track them.
> 
> Andrew does this, and yes, Adrian did it.
> 
> > In theory it could be nearly automated. Figure out what files related
> > to the bug and assign to the last 5 people who submitted patches
> > for them and/or signed off. 
> 
> I do think you're pretty optimistic. I think it's true for trivial driver 
> bugs, but even for trivial driver bugs the initial report is often not 
> enough to pinpoint the driver. 
> 
> Let's take the sis900 bug as an example (not because I want to rag on that 
> being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's 
> a good case of something really really easy - and if that easy case isn't 
> handled trivially and obviously, then the bug-reporting doesn't work).
> 
> In that case, the initial report was (condensed version, but fairly 
> accurate): "system hangs on boot at random points. 2.6.21-rc7 worked 
> well".
> 
> Now, realistically, if that entry had been in bugzilla, what would you do?
> 
> Equally realistically, let's ignore bugzilla for a moment, and ask what 
> the best method for handling something like this would be? Have an open 
> mind, no rules on "have to use bug tracker XYZ". 
> 
> You know what? The report went to me and the kernel mailing list as email. 
> And that was the *right* thing in that case. There was no good sign of who 
> it should go to, and while there wasn't a whole lot of information there, 
> there *was* a very tight timeframe (ie it could pinpoint to within about a 
> week when it started). But the only thing I could really ask for was for 
> the person to bisect it.
> 
> Would bugzilla have helped? HELL NO. It would have been a disaster. It 
> would have wasted reporter time, it would have wasted developer time, and 
> it would likely have been ignored because the bug report wasn't specific 
> enough to really trigger any good queries or trigger a maintainer.
> 
> So bugzilla wouldn't have helped even for a _trivial_ bug. 
> 
> I personally suspect that bugzilla helps more if a bug actually has a real 
> history, and people want to actually save that history because they are 
> stumped. But I think it should come in *then*, not immediately. Because it 
> is actually too intrustive for the simple stuff - and most bugs actually 
> are simple, it's just that they also get resolved so simply that people 
> don't even think of them as bugs (ie we have a lot of "duh" kind of fixes 
> too).

I completely agree.

My personal experience with bugzilla is that it's very unfriendly to
reporters.  IMHO it's suitable for tracking unresolved problems along with
debug patches, system information etc., but not for _reporting_ new ones.

Greetings,
Rafael

Re: Linux 2.6.21

[Stefan Richter]

Linus Torvalds wrote:
> On Sun, 29 Apr 2007, Andi Kleen wrote:
>> - Ask more people to just categorize and reassign bugs (anybody interested?)
>> - Give more people in bugzilla the power to reassign arbitary bugs
>>   (bugzilla maintainers would need to do that)
> 
> I think both of these are good ideas, but I don't think it's enough. There 
> must be some better form of bug tracking than bugzilla. Some relly 
> distributed way of letting people work together, *without* having to 
> congregate on some central web-site kind of thing. A "git for bugs", where 
> you can track bugs locally and without a web interface.
> 
> And I think it would be something much closer to "distributed email mbox 
> with status tracking facilities", _not_ a web clicky interface.

It has to be able to suck in reports from people who don't know much
about how the Linux guys handle bugs, and has to keep the reporters
involved up until a bug can be closed.  IOW it has to be compatible with
integrators, developers, and reporters/testers.  Luckily though, not all
of them need all of that system's features.  ('system' == the right
people with the right tools)
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> Linus Torvalds <torvalds@linux-foundation.org> writes:
> > 
> > The thing is, bugzilla is totally broken because it's designed to help 
> > track bugs, but it's *not* designed to actually handle the much harder 
> > problem, which is to actually get the *right* developers to be aware 
> > of the *right* bugs!
> 
> This means we need people who figure out who to assign bugs too.
> Aka bugmasters.

Yes. But not using bugzilla.

I don't know if you've noticed already, but I'm not the only one that 
doesn't have a very high opinon of bugzilla ;)

What works is somebody who is a bugmaster, and it doesn't really matter 
*what* bug tracker he points to (bugzilla being one of the possibilities, 
although not necessarily the best, and absolutely NOT the only choice), 
and turn them into emails. Once they are emails, bugzilla can track them.

Andrew does this, and yes, Adrian did it.

> In theory it could be nearly automated. Figure out what files related
> to the bug and assign to the last 5 people who submitted patches
> for them and/or signed off. 

I do think you're pretty optimistic. I think it's true for trivial driver 
bugs, but even for trivial driver bugs the initial report is often not 
enough to pinpoint the driver. 

Let's take the sis900 bug as an example (not because I want to rag on that 
being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's 
a good case of something really really easy - and if that easy case isn't 
handled trivially and obviously, then the bug-reporting doesn't work).

In that case, the initial report was (condensed version, but fairly 
accurate): "system hangs on boot at random points. 2.6.21-rc7 worked 
well".

Now, realistically, if that entry had been in bugzilla, what would you do?

Equally realistically, let's ignore bugzilla for a moment, and ask what 
the best method for handling something like this would be? Have an open 
mind, no rules on "have to use bug tracker XYZ". 

You know what? The report went to me and the kernel mailing list as email. 
And that was the *right* thing in that case. There was no good sign of who 
it should go to, and while there wasn't a whole lot of information there, 
there *was* a very tight timeframe (ie it could pinpoint to within about a 
week when it started). But the only thing I could really ask for was for 
the person to bisect it.

Would bugzilla have helped? HELL NO. It would have been a disaster. It 
would have wasted reporter time, it would have wasted developer time, and 
it would likely have been ignored because the bug report wasn't specific 
enough to really trigger any good queries or trigger a maintainer.

So bugzilla wouldn't have helped even for a _trivial_ bug. 

I personally suspect that bugzilla helps more if a bug actually has a real 
history, and people want to actually save that history because they are 
stumped. But I think it should come in *then*, not immediately. Because it 
is actually too intrustive for the simple stuff - and most bugs actually 
are simple, it's just that they also get resolved so simply that people 
don't even think of them as bugs (ie we have a lot of "duh" kind of fixes 
too).

> BTW one big problem in our current bugzilla is that a lot of people
> cannot reassign bugs they don't own. I sometimes see bugs that I don't
> own bug I know who is responsible, but bugzilla doesn't allow me to do it.

I think that's a problem, but I think the thing that I react to most (and 
I know some other people do too) is that I just don't want a broken mousy 
interface that I have to explicitly go and look at. I'd much rather get 
involved by email once it's clear that I need to be involved. Having a 
link to "more information at xyz" (where xyz _can_ be bugzilla too, but 
doesn't have to be!) is fine, but but it really cannot be the first 
interface, not in the current form, at least.

> So I think what would help:
> 
> - Ask more people to just categorize and reassign bugs (anybody interested?)
> - Give more people in bugzilla the power to reassign arbitary bugs
>   (bugzilla maintainers would need to do that)

I think both of these are good ideas, but I don't think it's enough. There 
must be some better form of bug tracking than bugzilla. Some relly 
distributed way of letting people work together, *without* having to 
congregate on some central web-site kind of thing. A "git for bugs", where 
you can track bugs locally and without a web interface.

And I think it would be something much closer to "distributed email mbox 
with status tracking facilities", _not_ a web clicky interface.

		Linus

Re: Linux 2.6.21

[Andi Kleen]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > 
> > That's exactly where Linus' "drop any bug reports that are more than a 
> > week old" suggestion is completely flawed - no matter what the submitter 
> > does, how often he tests latest kernels, noone will help him.
> 
> You talk, but what do you actually *suggest*?
> 
> Talk is cheap. You use to do the walk too, but you've already said that 
> you're not interested in that any more. So excuse me if I'm not impressed. 
> 
> The thing is, bugzilla is totally broken because it's designed to help 
> track bugs, but it's *not* designed to actually handle the much harder 
> problem, which is to actually get the *right* developers to be aware of 
> the *right* bugs!

This means we need people who figure out who to assign bugs too.
Aka bugmasters.

In theory it could be nearly automated. Figure out what files related
to the bug and assign to the last 5 people who submitted patches
for them and/or signed off. 

Ok I suppose it's not that easy -- you would need some human judgement.

BTW one big problem in our current bugzilla is that a lot of people
cannot reassign bugs they don't own. I sometimes see bugs that I don't
own bug I know who is responsible, but bugzilla doesn't allow me to do it.

So I think what would help:

- Ask more people to just categorize and reassign bugs (anybody interested?)
- Give more people in bugzilla the power to reassign arbitary bugs
(bugzilla maintainers would need to do that)

-Andi

Re: Linux 2.6.21

[Stefan Richter]

Willy Tarreau wrote:
> On Sun, Apr 29, 2007 at 12:58:09AM +0200, Markus Rechberger wrote:
>> I totally disagree here, bugzilla is a very good tool. If someone is
>> too lazy to look at it it's his problem.
> 
> I'm glad we finally found _the_ person using it !
> 
> More seriously, it's so much a complicated interface ! It's hard to
> bring more people into a discussion, it's hard to comment on code or
> suggested patches, etc... Mail is by far more adapted to the job !

To continue on the sarcastic tangent:  This flaw of bugzilla is
irrelevant for subsystems where there are less than three or two persons
who steadily hunt bugs anyway.  At the field I work on, I wouldn't have
anybody else to bring in in the first place, except that I sometimes
suggest to reporters to subscribe to a bug ticket.
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Stefan Richter]

I wrote:
> Joining duplicate reports at a mailinglist involves responding to
> multiple threads and send links into web archives of the list, which
> happens to be redundant to and disparate from your local e-mail storage.
> I can't see how this aspect of bug-handling works easier on mailinglists.

PS:  Of course what _does_ work better on mailinglists than on bugzilla
is to recognize duplicates as such in the first place, when the symptoms
seem only loosely related.  (I.e. seeing the big picture and recognize
patterns.)
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Stefan Richter]

David Lang wrote:
> I'll say that as a user I hate having to deal with bugzilla.
> 
> there's nothing more frustrating then spending a good chunk of time
> trying to find a similar bug, then jumping through all the bugzilla
> hoops to file a report to eventually (days/weeks later) get a message
> 'closed becouse it's a duplicate report), then have to go and track down
> what it's a duplicate of, read through that bug report, only to find
> that it's not solved there either, and to top it off, the people working
> on that bug won't see my report or that I'm available to troubleshoot it.

Ideally, joining duplicate reports should be a low-cost, lossless operation.

That said, when bug B is marked as duplicate of bug A, people at bug A
at least get a link to bug B, aren't they?  If they are too lazy to read
the report B, they obviously are not very interested in A either.  Tough
luck.  Vice versa, people at bug B get notified that the matter is now
continued at bug A and can add their Cc there.  Of course that addition
is one of the very few things that could probably be automated.

Joining duplicate reports at a mailinglist involves responding to
multiple threads and send links into web archives of the list, which
happens to be redundant to and disparate from your local e-mail storage.
 I can't see how this aspect of bug-handling works easier on mailinglists.

> from a user poit of view, e-mailing the kernel list (retrying a few days
> later of there is no response) tends to work _much_ better.

What I from a maintainer's POV agree with is that a report to the
appropriate mailinglist is often easier to triage than a report at
bugzilla, because the reporter often needs initial help to properly
define the problem.  Bugzilla becomes useful after a report reached a
minimum level of quality (after minimum initial triage) and if the bug
can be clearly associated with a maintained subsystem of the kernel (as
e.g. Linus already pointed out in this thread).
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Thomas Gleixner]

On Sat, 2007-04-28 at 20:41 -0700, David Miller wrote:
> From: Adrian Bunk <bunk@stusta.de>
> Date: Sun, 29 Apr 2007 01:04:16 +0200
> 
> > Bugzilla has an email interface.
> > Andrew forwards bugs from Bugzilla to developers.
> 
> Therefore, bugzilla only works at all when Andrew forwards things
> around by-hand.

That's not entirely true. There are people watching the bugs which might
be relevant for them on their own.

It does not make bugzilla better though. The user interface sucks and
getting things correlated is simply not possible.

	tglx

Re: Linux 2.6.21

[Russell King]

On Sun, Apr 29, 2007 at 12:58:09AM +0200, Markus Rechberger wrote:
> I totally disagree here, bugzilla is a very good tool. If someone is
> too lazy to look at it it's his problem.

If you think so, try reading my email and responding constructively
on how the issues there can be resolved.

That email contains good examples where bugzilla fails, and bugs end
up sitting around for ages untouched.  And no, it's not because I'm
"lazy".

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Linux 2.6.21

[Russell King]

On Sun, Apr 29, 2007 at 12:49:04AM +0200, Adrian Bunk wrote:
> On Sat, Apr 28, 2007 at 09:27:01PM +0100, Russell King wrote:
> > On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> > > We are already quite good at ignoring bug reports that come through 
> > > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> > > than 1600 open bugs because this tells how bad we are at handling bugs.
> > > How many thousand bug reports have been ignored during the same time on 
> > > linux-kernel?
> > 
> > However, look at this bug:
> > 
> >   http://bugme.osdl.org/show_bug.cgi?id=7760
> > 
> > It's outside my knowledge to be able to fix for various reasons:
> >...
> > I'm personally very tempted to close it as "won't fix" (I wish there was
> > a "can't fix" category.)
> >...
> 
> So this is a completely debugged bug in a well-maintained subsystem
> (no matter what the status in Bugzilla is).

You're being very optimistic.

I'm not sure where you get the idea that it's "completely debugged".
It isn't - I've no real idea what the problem is, let alone what the
solution might be.  I've only one guess based upon what is sane in
the kernel, and that isn't even based on the data provided in the
bug report.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Linux 2.6.21

[David Lang]

On Sat, 28 Apr 2007, David Miller wrote:

> 
> From: "Markus Rechberger" <mrechberger@gmail.com>
> Date: Sun, 29 Apr 2007 00:58:09 +0200
>
>> On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>>>
>>>
>>> On Sat, 28 Apr 2007, Adrian Bunk wrote:
>>>>
>>>> We are already quite good at ignoring bug reports that come through
>>>> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
>>>> than 1600 open bugs because this tells how bad we are at handling bugs.
>>>
>>> No, it just shows that bugzilla doesn't matter for most of the kernel.
>>>
>>> Don't say that "bugzilla tells how bad we are at handling bugs". It tells
>>> how bad *bugzilla* is for handling bugs, nothing more.
>>>
>>
>> I totally disagree here, bugzilla is a very good tool.
>
> No, Bugzilla really does suck, and I personally refuse to use it when
> I have a choice.  And guess what?  You better be concerned about that
> because I maintain all of the networking code :-)
>
> It puts the onus FAR too much on the developer and not enough on the
> reporter and other minions.  We have a small resource of developers,
> yet lots of users, bug reporters, and minions, so something that
> doesn't take advantage of the larger resource we have is going to
> not function efficiently at all.  Yet that is what bugzilla does.

I'll say that as a user I hate having to deal with bugzilla.

there's nothing more frustrating then spending a good chunk of time trying to 
find a similar bug, then jumping through all the bugzilla hoops to file a report 
to eventually (days/weeks later) get a message 'closed becouse it's a duplicate 
report), then have to go and track down what it's a duplicate of, read through 
that bug report, only to find that it's not solved there either, and to top it 
off, the people working on that bug won't see my report or that I'm available to 
troubleshoot it.

from a user poit of view, e-mailing the kernel list (retrying a few days later 
of there is no response) tends to work _much_ better.

David Lang

Re: Linux 2.6.21

[Willy Tarreau]

On Sun, Apr 29, 2007 at 12:58:09AM +0200, Markus Rechberger wrote:
> On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >
> >
> >On Sat, 28 Apr 2007, Adrian Bunk wrote:
> >>
> >> We are already quite good at ignoring bug reports that come through
> >> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
> >> than 1600 open bugs because this tells how bad we are at handling bugs.
> >
> >No, it just shows that bugzilla doesn't matter for most of the kernel.
> >
> >Don't say that "bugzilla tells how bad we are at handling bugs". It tells
> >how bad *bugzilla* is for handling bugs, nothing more.
> >
> 
> I totally disagree here, bugzilla is a very good tool. If someone is
> too lazy to look at it it's his problem.

I'm glad we finally found _the_ person using it !

More seriously, it's so much a complicated interface ! It's hard to
bring more people into a discussion, it's hard to comment on code or
suggested patches, etc... Mail is by far more adapted to the job !

See how many times people do public patch reviews here. You get one
comment every 5-10 lines. I have yet to see how this could be done
in bugzilla.

And maintainers have to _think_ about going there. Mail comes in
without deliberate action. This is especially important because
only your eye is involved in noticing bugs affecting areas where
you can help.

What _may_ be useful would be to send digests or batches of recently
open bugs to LKML. But not all of them. Maybe doing this once a week
in the same way we post patches lists for review. We could have a batch
of "[BUG 1/23] quick subject". At least more people will notice them
and will be able to comment on them. And given the number of people
reading LKML, the bugs should only be posted once. Because if a bug
posted to LKML in a noticeable manner does not get assigned in a week,
it will never be.

Personally, I got used to review Greg & Chris' announces of stable
patches to find if some of them could affect 2.4. It's far easier
to ask for precisions in a mail you weren't expecting than it is to
go somewhere search for something you don't know exists.

Willy

Re: Linux 2.6.21

[David Miller]

From: Adrian Bunk <bunk@stusta.de>
Date: Sun, 29 Apr 2007 01:04:16 +0200

> Bugzilla has an email interface.
> Andrew forwards bugs from Bugzilla to developers.

Therefore, bugzilla only works at all when Andrew forwards things
around by-hand.

Re: Linux 2.6.21

[David Miller]

From: "Markus Rechberger" <mrechberger@gmail.com>
Date: Sun, 29 Apr 2007 00:58:09 +0200

> On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >
> >
> > On Sat, 28 Apr 2007, Adrian Bunk wrote:
> > >
> > > We are already quite good at ignoring bug reports that come through
> > > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
> > > than 1600 open bugs because this tells how bad we are at handling bugs.
> >
> > No, it just shows that bugzilla doesn't matter for most of the kernel.
> >
> > Don't say that "bugzilla tells how bad we are at handling bugs". It tells
> > how bad *bugzilla* is for handling bugs, nothing more.
> >
> 
> I totally disagree here, bugzilla is a very good tool.

No, Bugzilla really does suck, and I personally refuse to use it when
I have a choice.  And guess what?  You better be concerned about that
because I maintain all of the networking code :-)

It puts the onus FAR too much on the developer and not enough on the
reporter and other minions.  We have a small resource of developers,
yet lots of users, bug reporters, and minions, so something that
doesn't take advantage of the larger resource we have is going to
not function efficiently at all.  Yet that is what bugzilla does.

It's made way too much work for me every time I'm come in contact with
it, it wastes my time instead of making good use of it.

As a developer I do not want to get pounded with emails containing
state changes and other bullshit that typically comes with being
assigned to or on the CC of a bugzilla entry.  I don't want to be
reminded that a bug hasn't been touched in weeks, if the reporter
doesn't care I don't care and I'll work on things that people do care
about.  It makes me delete all the bugzilla email, even the ones with
important information in them, because it's rediculious to have to
sift through all of that crap.

People only use bugzilla because it is well understood and nothing
better has reached critical mass yet.

Re: Linux 2.6.21

[Andrew Morton]

On Sat, 28 Apr 2007 17:07:30 -0700 (PDT) Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Sat, 28 Apr 2007, Martin J. Bligh wrote:
> > 
> > Go to http://bugzilla.kernel.org. Hit query. Find the box that says
> > "Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.
> > 
> > 74 bugs found.
> > 
> > Not hard to do.
> 
> And what part of the "directed" did you miss?
> 
> Do you really expect me to go there every day to look at all bugs? That's 
> nbot a bug tracker. That's just a noise-maker.
> 
> It needs to be email, not some "mouse around for 30 seconds and type 
> thing", and it needs to be *directed*. Preferably with somebody who 
> actually did some manual scanning over it and spent a few minutes just 
> looking at whether it looks like a worthy bug.
> 
> In other words: we shouldn't have all developers wasting time doing this. 

yup.

> It would be much better to have _one_ person (or a group of people) doing 
> it,

I am doing that.  It's only 5-10 a day - routing them to the relevant
culprit is very little work.  It's also very little work for said culprits
to totally ignore said routing, which is a tougher problem.

Re: Linux 2.6.21

[Andrew Morton]

On Sun, 29 Apr 2007 08:11:30 +1000 Neil Brown <neilb@suse.de> wrote:

> On Saturday April 28, mbligh@mbligh.org wrote:
> > 
> > Yes, human involvement from someone with half a brain would be better.
> > Andrew does a lot of that. Not a particularly good use of talent really.
> > but still.
> 
> I think more than half a brain is needed to do this well.  You need a
> reasonable understanding of how all the bits of the kernel work
> together so that you have a good chance of sending the bug in the right
> direction.  You need a good understanding of the kernel community and
> various sub communities so that you know who might be both able and
> willing to deal with the bug.  And it wouldn't hurt to have a good
> over-view of the current 'hot' areas of the kernel so you know if it
> is really worth suggesting "try with the latest -mm" or not.
> And you need good people skills.
> 
> So I think you really need a lot of up-to-date knowledge to do this
> well.  Because of Andrew's position as a funnel, he has a lot of that
> knowledge.

yup

>  It would be really nice if he had some help though.

Amen, Brother Neil.

>  And I
> really think that would mean finding someone in the community who
> would rather be coding (and currently are) and convincing them that
> there is a higher calling for them.  Finding someone out side or on
> the edge of the community is less likely to be effective.

	http://www.google.com/support/jobs/bin/answer.py?answer=53317

This is a fully funded regular full-time position @google's Mountain View
HQ, sitting within nagging range of myself, doing precisely what you
describe.

Unfortunately the recruiting has been a bit tricky - this is not a typical
job and it's a funny mixture of bureaucracy/politics/social engineering
and programming.  People who are skilled in both areas, are, ah, uncommon.
But it will happen eventually.

Meanwhile I

- ensure that all bugzilla reports are routed to the relevant maintainer

- ensure that all those who reported bugs via email are later asked to
  raise bugzilla reports if it didn't get fixed (but I only monitor one list!)

- continue to file away all the real-looking bugzillas with the intention
  of generating aggregated reports, but you know how it is.

Re: Linux 2.6.21

[Markus Rechberger]

On 4/29/07, Bob Tracy <rct@gherkin.frus.com> wrote:
> Linus Torvalds wrote:
> > On Sun, 29 Apr 2007, Markus Rechberger wrote:
> > > How else should bugs get handled, sending them to the lkml?
> >
> > Actually, looking at Adrian's regression lists, yes. lkml worked better
> > than bugzilla did. By at _least_ a factor of two.
>
> Since 1992, lkml (with "Cc:" to the appropriate subsystem mailing list
> if applicable) and the presumed responsible parties are the only channels
> I've used to report the bugs I encounter.
>

since there are subsystems out there which are managed separatly this
doesn't work out.
I wasn't happy when I noticed that patches got applied to the
sourcecode I contributed without notifying me while I still worked on
that code separatly
It was moreover the fault of the subsystem maintainer to not notify me
back then but a centralized bugreporting (as bugzilla) tool would at
least have notified me, or I would have been able to see the suggested
changes there.

But I agree with that if you're only 1 level far away from the linux kernel.

> Other methods come and go, but old habits die hard, particularly when
> they have a knack for producing the desired result.  Historically,
> requiring a developer to fire up a GUI to read a bug report decreases
> the chance that bug report will be noticed.  The development community
> can do whatever flips its collective switch as far as tracking bugs,
> but the bugs have to be reported and noticed before tracking becomes a
> meaningful activity.
>
> One more thought and I'll get off your screens...  We've steadfastly
> resisted making lkml and friends subscriber-only mailing lists precisely
> because we don't want to miss a potential bug report because a would-be
> submitter isn't subscribed.  If people aren't looking for bug reports
> here, what's the point?
>

it's just easy to miss something here, if an ext3 bug comes in and all
people who're involved in the ext3 filesytem are on vacation I'm sure
they won't read all the mails which came in during a week, now take a
part of the kernel which is smaller than the ext3 filesystem (eg. usb
gadgets, smaller drivers)

Markus

Re: Linux 2.6.21

[Markus Rechberger]

On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Sun, 29 Apr 2007, Markus Rechberger wrote:
> >
> > I totally disagree here, bugzilla is a very good tool. If someone is
> > too lazy to look at it it's his problem.
>
> You must be doing things very differently from a lot of other people if
> you think that's the case.
>

Well I'm behind the stuff I'm doing because I'm interested in it. And
if some bugs are introduced by my work or derived by my work I'd like
to get them cleaned up in the end.
If I see that someone reports bugs which doesn't really address my
work at all I just forward them to the subsystem/maintainer who's "in
charge" (if someone can say it that way for an open source project)

> > Kernel Janitors can pick out some bugs which aren't addressed by
> > anyone or got left behind.
>
> IF that happened, it would actually be great. That's what I'm arguing for.
> And it was basically what Adrian was doing!
>

I'm very sure that happens maybe it's just not visible to everyone
because there are so many open issues. (I just take myself as an
example here, I didn't do too much with other bugs but at least some
of my work closed 5 other bugs this year beside the bugreports I'm
getting directly)

> > How else should bugs get handled, sending them to the lkml?
>
> Actually, looking at Adrian's regression lists, yes. lkml worked better
> than bugzilla did. By at _least_ a factor of two.
>

Yes Adrian did a very good job with collecting every bugreport and
sending the mails to all corresponding subsystems.

> > I'm 100% sure some bugreports will also get lost then, but on the lkml
> > they'll very likely remain lost whereas in the bugzilla they'll remain
> > as open.
>
> What's the difference between bugzilla and lkml.org? Both have search
> buttons. Both archive the old stuff. Both can be pointed to.
>

Both have search buttons yes, but the lkml doesn't leave an unread
mail open ontop of the lkml as bugzilla does if you look for open bugs
in a subsystem.

> > what are your suggestions to improve a bugreporting tool, I'm very
> > sure that many people, especially people who want to get into existing
> > projects here, would love to contribute.
>
> I don't know what the perfect setup is, but I do know that bugzilla is
> very close to be totally useless for the top-level maintainers.
>
> Try to think like a person who doesn't maintain *one* specific file in the
> kernel, but who can actually make a good judgement about a lot of things,
> or at least funnel a problem report to the right person?
>
> And now, imagine that that person is also fairly busy (exactly *because*
> he's not looking at a single file, he may be maintaining a huge subsystem
> that has multiple submaintainers etc).
>
> And ask yourself whether bugzilla really helps.
>

bugzilla keeps the bugs open at least, at the lkml I use to skip days sometimes.
Many people who consider themself as maintainer of a subsystem are
assigned to a subsection on bugzilla, if it really doesn't work out we
have to change the corresponding maintainer.
If that maintainer doesn't know where to go with that bugreport he can
easily send it to the lkml and some people will recognize the
sender/email and pay extra attention to it (that's just how I think
about it)

> > I'd say this is a personal opinion, some people will get along with it
> > and some of them will not...
>
> I think bugzilla really only works for very "directed" issues. If you
> already know exactly which driver is affected (which is often wrong
> anyway: some of the bugs that were due timer breakage got blamed as disk
> hangs!) it's almost totally useless.
>
> And yes, maybe that's why you have a much higher opinion of bugzilla than
> I do. To _me_ bugzilla is a total mess. There's absolutely _zero_ useful
> information there. And I'm pretty certain that is true of a *lot* of other
> people too. But if you have a small project, or you maintain a very
> specific (and clearly delineated) part of a big project, bugzilla probably
> looks a lot more palatable.

well are there any bugs that cannot be forwarded/directed to a
corresponding maintainer?
Maybe I don't see something here, can you point me out to a bugreport
which cannot be handled at all?
As a reference I'll take following bugreport:
http://thread.gmane.org/gmane.linux.kernel/521185

the bug doesn't even mention what device is affected, asking for
further detailed information (dmesg) shows up what's left at least..
(in the meanwhile the bug even got solved)

Markus

Re: Linux 2.6.21

[Bob Tracy]

Linus Torvalds wrote:
> On Sun, 29 Apr 2007, Markus Rechberger wrote:
> > How else should bugs get handled, sending them to the lkml?
> 
> Actually, looking at Adrian's regression lists, yes. lkml worked better 
> than bugzilla did. By at _least_ a factor of two.

Since 1992, lkml (with "Cc:" to the appropriate subsystem mailing list
if applicable) and the presumed responsible parties are the only channels
I've used to report the bugs I encounter.

Other methods come and go, but old habits die hard, particularly when
they have a knack for producing the desired result.  Historically,
requiring a developer to fire up a GUI to read a bug report decreases
the chance that bug report will be noticed.  The development community
can do whatever flips its collective switch as far as tracking bugs,
but the bugs have to be reported and noticed before tracking becomes a
meaningful activity.

One more thought and I'll get off your screens...  We've steadfastly
resisted making lkml and friends subscriber-only mailing lists precisely
because we don't want to miss a potential bug report because a would-be
submitter isn't subscribed.  If people aren't looking for bug reports
here, what's the point?

--Bob Tracy
rct@frus.com

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Neil Brown wrote:
> 
> I think more than half a brain is needed to do this well.  You need a
> reasonable understanding of how all the bits of the kernel work
> together so that you have a good chance of sending the bug in the right
> direction.

Yes. However, even if you just end up summarizing the current outstanding 
issues (and are not able to necessarily point to the right person), I 
suspect that all the top-level maintainers would be interested in it. As 
long as it's a "summary report twice a week" kind of thing.

In fact, I suspect a lot of non-maintainers would be interested too!

And if you then have some way for people to add commentary (and directing) 
to entries, you might start out not knowing where some bugreport should 
go, but you'll have people able to forward them.

I end up doing that fairly regularly - adding people to the Cc when I'm 
involved in a bug, and we suddenly notice that "hey, it looks like it's a 
timer" issue or something, and we end up adding Ingo and Thomas Gleixner 
to the cc.

This, btw, is why email is *so* much nicer than a web interface. A web 
interface is always "a single person looking at it". An email, on the 
other hand, is always "you can bounce it to make _another_ person look at 
it".

But I agree that the more involved in the kernel the initiating person has 
been (at least to *some* degree), the better. No question. I just don't 
think it necessarily needs to be a hugely core person. Anybody who has 
been reading lkml for the last few months will already know a lot of the 
people involved in different subsystems!

			Linus

Re: Linux 2.6.21

[Linus Torvalds]

On Sat, 28 Apr 2007, Martin J. Bligh wrote:
> 
> Go to http://bugzilla.kernel.org. Hit query. Find the box that says
> "Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.
> 
> 74 bugs found.
> 
> Not hard to do.

And what part of the "directed" did you miss?

Do you really expect me to go there every day to look at all bugs? That's 
nbot a bug tracker. That's just a noise-maker.

It needs to be email, not some "mouse around for 30 seconds and type 
thing", and it needs to be *directed*. Preferably with somebody who 
actually did some manual scanning over it and spent a few minutes just 
looking at whether it looks like a worthy bug.

In other words: we shouldn't have all developers wasting time doing this. 
It would be much better to have _one_ person (or a group of people) doing 
it, and actually turnign your "Not hard to do" into real information, 
rather than just random data. Adrian did.

The good news is that it looks like now that people are aware of it, we 
hopefully have others who will help do this kind of thing.

		Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
>...
> What's the difference between bugzilla and lkml.org? Both have search 
> buttons. Both archive the old stuff. Both can be pointed to.

Mailing lists don't track bugs.

The _only_ reason why I originally started regression lists was because 
several kernel developers were spreading the fairy tale "noone tests -rc 
kernels".

This was only possible because so many bug reports to linux-kernel never 
get any reply, and are therefore lost.

After I started the regression lists, it suddenly turned out how many 
people test -rc kernels, and that even with regression lists at least 
weekly, it still often took weeks ontil some developer even started 
debugging a regression.

So what's the difference between bugzilla and lkml.org?
In Bugzilla we are able to see how bad our bug handling is...

> 			Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> 
> Bugzilla has an email interface.
> Andrew forwards bugs from Bugzilla to developers.

Yes. And if you search around, you'll even see that I occasionally use it. 

But it's useful only once the bug has been assigned and somebody has 
actually *looked* at it. The fact that some people do this is a big credit 
(Andrew, Dave J etc)

> There might be small room for improvements, but I don't see how 
> man-power or technology could make a big difference in this area.

I'm talking about getting the developers to _look_ at the bugs in the 
first place. Bugzilla is not very good at that, because it has no useful 
interfaces for doing so (unless you can specify your area of interest so 
exactly that you can actually set yourself up as a maintainer of one 
particular area).

Almost none of the subtle (and thus harder) bugs tend to fall into that 
kind of nice category. 

For example, look at suspend/resume bugs. Do you realize that 99% of them 
are device driver issues, but how the heck do you connect a "my laptop 
does't resume" to the _right_ device driver maintainer?

And do you realize (and acknowledge) that it would be total madness to 
send all suspend/resume bugs to _everybody_ who maintains any device 
driver at all?

See? THAT is the problem with bugzilla. It only works for the "easy" 
cases. It works for the case where a reporter can say with certainty that 
the reason his machine doesn't boot is a particular network device driver 
(like the sis900 regression we had in 2.6.21). But once you know the 
subarea that precisely, bugzilla doesn't even help you that much, and it's 
likely easier and more productive to just send email directly to the right 
person and cc Jeff and netdev or something!

> "*once* you've found and gotten the right developer involved" is the 
> real problem, not how to track bugs.

And I agree 100%.

And bugzilla does *nothing* here.

> *This* is *the* problem.
> And no change in bug tracking will help with this problem.

So why are you pushing bugzilla? 

There are actually better bug trackers around. One of them is "google". 
For oopses, one of the thngs I do is to put in the most relevant 
information (backtrace etc) into google, and ask google to try to find the 
pattern. That sometimes actually does pretty well - you can get a real 
feel for "oh, there's a pattern here - they're all AMD machines with the 
NVidia chipset" kind of thing.

Bugzilla doesn't offer anything even remotely as useful. 

It's the "big picture" that tends to be hard to find.

And that's what *you* gave with the regression report. A summary. You even 
noted some of the patterns, so people actually had them already somewhat 
pre-sorted.

THAT is what we want.

But another way to get the "high-level picture" is to actually just say 
"what's active now". And that's why I think the whole "1600 open 
bug-reports" kind of thing is useless. Bugzilla does *not* have mail 
interfaces for that kind of "these 50 bugs were actively reported on and 
unresolved" in the last week summaries! 

(And if it did, it would still almost certainly need some human care and 
understanding)

		Linus

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Markus Rechberger wrote:
> 
> I totally disagree here, bugzilla is a very good tool. If someone is
> too lazy to look at it it's his problem.

You must be doing things very differently from a lot of other people if 
you think that's the case.

> Kernel Janitors can pick out some bugs which aren't addressed by
> anyone or got left behind.

IF that happened, it would actually be great. That's what I'm arguing for. 
And it was basically what Adrian was doing!

> How else should bugs get handled, sending them to the lkml?

Actually, looking at Adrian's regression lists, yes. lkml worked better 
than bugzilla did. By at _least_ a factor of two.

> I'm 100% sure some bugreports will also get lost then, but on the lkml
> they'll very likely remain lost whereas in the bugzilla they'll remain
> as open.

What's the difference between bugzilla and lkml.org? Both have search 
buttons. Both archive the old stuff. Both can be pointed to.

> what are your suggestions to improve a bugreporting tool, I'm very
> sure that many people, especially people who want to get into existing
> projects here, would love to contribute.

I don't know what the perfect setup is, but I do know that bugzilla is 
very close to be totally useless for the top-level maintainers.

Try to think like a person who doesn't maintain *one* specific file in the 
kernel, but who can actually make a good judgement about a lot of things, 
or at least funnel a problem report to the right person?

And now, imagine that that person is also fairly busy (exactly *because* 
he's not looking at a single file, he may be maintaining a huge subsystem 
that has multiple submaintainers etc).

And ask yourself whether bugzilla really helps.

> I'd say this is a personal opinion, some people will get along with it
> and some of them will not...

I think bugzilla really only works for very "directed" issues. If you 
already know exactly which driver is affected (which is often wrong 
anyway: some of the bugs that were due timer breakage got blamed as disk 
hangs!) it's almost totally useless.

And yes, maybe that's why you have a much higher opinion of bugzilla than 
I do. To _me_ bugzilla is a total mess. There's absolutely _zero_ useful 
information there. And I'm pretty certain that is true of a *lot* of other 
people too. But if you have a small project, or you maintain a very 
specific (and clearly delineated) part of a big project, bugzilla probably 
looks a lot more palatable.

			Linus

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> 
> That's exactly where Linus' "drop any bug reports that are more than a 
> week old" suggestion is completely flawed - no matter what the submitter 
> does, how often he tests latest kernels, noone will help him.

You talk, but what do you actually *suggest*?

Talk is cheap. You use to do the walk too, but you've already said that 
you're not interested in that any more. So excuse me if I'm not impressed. 

The thing is, bugzilla is totally broken because it's designed to help 
track bugs, but it's *not* designed to actually handle the much harder 
problem, which is to actually get the *right* developers to be aware of 
the *right* bugs!

And both of those "right"s are important. Spamming everybody will just 
mean that everybody tunes them out. And spamming even the right developers 
with useless bugreports will also just cause them to tune out the good 
ones.

The thing is, the "tracking bugs" part that bugzilla _can_ do is also 
totally _useless_ without that much more important phase, namely the 
"connect the parties". That's what you really used to do. You made 
developers connect to the reports, because your reports were _useful_ and 
not overly noisy.

But go back and look - did you notice that once you connect the dots, it 
turns out that bugzilla itself isn't all that wonderful. Quite a lot of 
your regression reports had other ways of pointing to the problems, 
including very much mailing list archive pointers etc!

So bugzilla isn't actually all it's touted to be even _once_ the 
connection between reporter and developer has been established. I really 
don't see why you are so hung up about bugzilla, when your own regression 
reports didn't generally point to it all that often!

(I just went back and double-checked: you had more than twice as many 
pointers to kernel mailing list archives than you had pointers to bugzilla 
in the one series I looked at. And I'm _not_ saying that's wrong at all: I 
think the mailing list is actually likely to be at LEAST as useful as 
bugzilla is as a bug-tracker!)

And bugzilla actually falls down even more than the mailing list does for 
the whole (and MUCH MORE IMPORTANT!) phase of connecting developers to bug 
reports. And THAT is really the problem here.

And no, I don't actually think that automatically closing entries that 
haven't gotten any attention in the last two weeks would "fix" bugzilla. 
But I think that it might actually make people 
 (a) more likely to look into bugzilla (and thus maybe improve the 
     "connecting" developers/reporters thing)
 (b) act as a trivial noise-removal thing, because it would give 
     preferential treatment to people who are diligent about their 
     bug-reports and are willing to follow up on them and it would also 
     remove the noise that comes from obvious things that broke and got 
    fixed.

But I think the real fix is to have real humans that help de-noisify the 
bug reports. Let's call them the "bunk" group, just to pick a random 
four-letter combination. The kinds of people who help turn the mindless 
noise that is bugzilla (and the kernel mailing list) into directed 
_information_. 

No, nothing is ever going to be perfect. And "filtering" the noise will 
inevitably also end up dropping real information. But not filtering it 
will guarantee that even more is dropped.

		Linus

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 00:33, Adrian Bunk wrote:
> On Sun, Apr 29, 2007 at 08:11:30AM +1000, Neil Brown wrote:
> > On Saturday April 28, mbligh@mbligh.org wrote:
> >...
> > > As Andrew has pointed out before though - even though he forwards
> > > the bugs, nobody does anything with it. The sad truth seems to be
> > > that people have very little interest in fixing bugs when they are
> > > reported - it's not sexy, I guess.
> > 
> > Not sexy, and also not at all easy.   A lot of the interesting bugs
> > seem to be subtle interactions between separate parts of the kernel -
> > one part making an assumption or exhibiting a behaviour that the other
> > part didn't expect.  And we all know that writing bug^Wcode is easier
> > than removing bugs.  I can spend hour and hours reading through code
> > trying to get the big picture, and end up finding a one-line change
> > that then needs documenting, testing and external review.  It's not
> > easy.
> > 
> > > I'm still unconvinced the users or the tool are the problem, but if it
> > > makes you happier, we can do that.
> > 
> > No, they aren't the problem.  Bugs are the problem.  But they might be
> > a more effective part of the solution.
> > 
> > My perception of the kernel bugzilla is that visibility is very low.
> > 
> > I think there is value in weekly reminders, and I wouldn't mind seeing
> > a weekly Email on linux-kernel with something like a list of open bugs
> > that have not seen any activity in between 1 and 2 weeks.  It might
> > get someone out-of-area interested, and might be noticed by someone
> > who thinks they are in-area and get them wondering why they didn't
> > find out when the bug was first reported.
> 
> The 100 kB email limit has to be lifted for this... 
> 
> More seriously, there are > 1000 open bugs in the kernel Bugzilla 
> without any activity during the last 2 weeks.
> 
> The problem is usually either "Not sexy, and also not at all easy." or
> "no maintainer".

Please add "limited human resources" to this list.

Greetings,
Rafael

Re: Linux 2.6.21

[Adrian Bunk]

On Sat, Apr 28, 2007 at 03:33:39PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 28 Apr 2007, Adrian Bunk wrote:
> > 
> > We are already quite good at ignoring bug reports that come through 
> > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> > than 1600 open bugs because this tells how bad we are at handling bugs.
> 
> No, it just shows that bugzilla doesn't matter for most of the kernel.
> 
> Don't say that "bugzilla tells how bad we are at handling bugs". It tells 
> how bad *bugzilla* is for handling bugs, nothing more.
> 
> Trying to play politics by pointing to bugzilla is pointless. Bugzilla is 
> used for a few subsystems (ACPI seems to use it actively, for example), 
> but I doubt most developers use it.
> 
> Would be be good to have a better bug-tracking setup? Yes. But I think it 
> takes man-power, and it would take something *fundamentally* better than 
> bugzilla.

Bugzilla has an email interface.
Andrew forwards bugs from Bugzilla to developers.

There might be small room for improvements, but I don't see how 
man-power or technology could make a big difference in this area.

> Maybe the new "http://kernelnewbies.org/known_regressions" thing will 
> evolve to something worth tracking. Right now, bugzilla isn't it (although 
> it can be a useful tracking place for individual bugs, *once* you've found 
> and gotten the right developer involved - but that's a huge step that 
> bugzilla generally does *not* do for us).

"*once* you've found and gotten the right developer involved" is the 
real problem, not how to track bugs.

And not only a developer active in this area, more important a 
developer who knows the subsystem/driver involved *and is willing
to work on bug reports*.

*This* is *the* problem.
And no change in bug tracking will help with this problem.

> 			Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Markus Rechberger]

On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Sat, 28 Apr 2007, Adrian Bunk wrote:
> >
> > We are already quite good at ignoring bug reports that come through
> > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
> > than 1600 open bugs because this tells how bad we are at handling bugs.
>
> No, it just shows that bugzilla doesn't matter for most of the kernel.
>
> Don't say that "bugzilla tells how bad we are at handling bugs". It tells
> how bad *bugzilla* is for handling bugs, nothing more.
>

I totally disagree here, bugzilla is a very good tool. If someone is
too lazy to look at it it's his problem.
Kernel Janitors can pick out some bugs which aren't addressed by
anyone or got left behind. I also found some bugs there which could
have been solved by anyone here, the matter is just that many people
aren't interested in mainly bug fixing and many also work on different
other topics here.

How else should bugs get handled, sending them to the lkml?
I'm 100% sure some bugreports will also get lost then, but on the lkml
they'll very likely remain lost whereas in the bugzilla they'll remain
as open.

> Trying to play politics by pointing to bugzilla is pointless. Bugzilla is
> used for a few subsystems (ACPI seems to use it actively, for example),
> but I doubt most developers use it.
>

as for the em28xx I actively use it, but I also set up a mailinglist
etc. and there are many supporters already...

> Would be be good to have a better bug-tracking setup? Yes. But I think it
> takes man-power, and it would take something *fundamentally* better than
> bugzilla.
>

what are your suggestions to improve a bugreporting tool, I'm very
sure that many people, especially people who want to get into existing
projects here, would love to contribute.

> Maybe the new "http://kernelnewbies.org/known_regressions" thing will
> evolve to something worth tracking. Right now, bugzilla isn't it (although
> it can be a useful tracking place for individual bugs, *once* you've found
> and gotten the right developer involved - but that's a huge step that
> bugzilla generally does *not* do for us).
>

I'd say this is a personal opinion, some people will get along with it
and some of them will not...

Markus

Re: Linux 2.6.21

[Adrian Bunk]

On Sat, Apr 28, 2007 at 09:27:01PM +0100, Russell King wrote:
> On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> > We are already quite good at ignoring bug reports that come through 
> > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> > than 1600 open bugs because this tells how bad we are at handling bugs.
> > How many thousand bug reports have been ignored during the same time on 
> > linux-kernel?
> 
> However, look at this bug:
> 
>   http://bugme.osdl.org/show_bug.cgi?id=7760
> 
> It's outside my knowledge to be able to fix for various reasons:
>...
> I'm personally very tempted to close it as "won't fix" (I wish there was
> a "can't fix" category.)
>...

So this is a completely debugged bug in a well-maintained subsystem
(no matter what the status in Bugzilla is).

The problem is that the other 1600 open bugs aren't in this state.

> I'm no longer serial maintainer.  Bug IDs after about 7000 reflect bugs
> submitted since I've resigned my serial maintainership, and therefore
> I've ignored them.
>...

That's one of the problems: Unmaintained subsystems.

Since you stepped down as serial maintainer (and it's your right as 
maintainer to do so), the serial subsystem is unmaintained.

That's exactly where Linus' "drop any bug reports that are more than a 
week old" suggestion is completely flawed - no matter what the submitter 
does, how often he tests latest kernels, noone will help him.

> It's far easier to ignore bug reports in bugzilla
> than it is to get categories reassigned (to whom? - dunno) or even
> deleted (if no one steps up presumably that's what needs to happen?)
>...

New subsystems always get default owners like   	 
drivers_ieee1394@kernel-bugs.osdl.org, and people interested in such 
bugs can edit their preferences to watching all (pseudo) users in whose 
bugs they are interested. drivers_serial@kernel-bugs.osdl.org [1] would 
be the logical defult owner.

> Russell King

cu
Adrian

[1] or @linux-foundation.org

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Neil Brown]

On Sunday April 29, bunk@stusta.de wrote:
> On Sun, Apr 29, 2007 at 08:11:30AM +1000, Neil Brown wrote:
> > 
> > I think there is value in weekly reminders, and I wouldn't mind seeing
> > a weekly Email on linux-kernel with something like a list of open bugs
> > that have not seen any activity in between 1 and 2 weeks.  It might
> > get someone out-of-area interested, and might be noticed by someone
> > who thinks they are in-area and get them wondering why they didn't
> > find out when the bug was first reported.
> 
> The 100 kB email limit has to be lifted for this... 
> 
> More seriously, there are > 1000 open bugs in the kernel Bugzilla 
> without any activity during the last 2 weeks.

I meant "between 1 and 2 weeks"  - so each bug hits linux-kernel once,
any only if no-one seem to care.
If I ask for bugs changed in the last 7 days I get 80
If I ask for bugs changed in the last 14 days I get 114.
So today, that list would have (114-80) = 34 bug, which I think it is
manageable to glance through and think "Would I like to care".

NeilBrown

Re: Linux 2.6.21

[Linus Torvalds]

On Sat, 28 Apr 2007, Adrian Bunk wrote:
> 
> We are already quite good at ignoring bug reports that come through 
> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> than 1600 open bugs because this tells how bad we are at handling bugs.

No, it just shows that bugzilla doesn't matter for most of the kernel.

Don't say that "bugzilla tells how bad we are at handling bugs". It tells 
how bad *bugzilla* is for handling bugs, nothing more.

Trying to play politics by pointing to bugzilla is pointless. Bugzilla is 
used for a few subsystems (ACPI seems to use it actively, for example), 
but I doubt most developers use it.

Would be be good to have a better bug-tracking setup? Yes. But I think it 
takes man-power, and it would take something *fundamentally* better than 
bugzilla.

Maybe the new "http://kernelnewbies.org/known_regressions" thing will 
evolve to something worth tracking. Right now, bugzilla isn't it (although 
it can be a useful tracking place for individual bugs, *once* you've found 
and gotten the right developer involved - but that's a huge step that 
bugzilla generally does *not* do for us).

			Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 08:11:30AM +1000, Neil Brown wrote:
> On Saturday April 28, mbligh@mbligh.org wrote:
>...
> > As Andrew has pointed out before though - even though he forwards
> > the bugs, nobody does anything with it. The sad truth seems to be
> > that people have very little interest in fixing bugs when they are
> > reported - it's not sexy, I guess.
> 
> Not sexy, and also not at all easy.   A lot of the interesting bugs
> seem to be subtle interactions between separate parts of the kernel -
> one part making an assumption or exhibiting a behaviour that the other
> part didn't expect.  And we all know that writing bug^Wcode is easier
> than removing bugs.  I can spend hour and hours reading through code
> trying to get the big picture, and end up finding a one-line change
> that then needs documenting, testing and external review.  It's not
> easy.
> 
> > I'm still unconvinced the users or the tool are the problem, but if it
> > makes you happier, we can do that.
> 
> No, they aren't the problem.  Bugs are the problem.  But they might be
> a more effective part of the solution.
> 
> My perception of the kernel bugzilla is that visibility is very low.
> 
> I think there is value in weekly reminders, and I wouldn't mind seeing
> a weekly Email on linux-kernel with something like a list of open bugs
> that have not seen any activity in between 1 and 2 weeks.  It might
> get someone out-of-area interested, and might be noticed by someone
> who thinks they are in-area and get them wondering why they didn't
> find out when the bug was first reported.

The 100 kB email limit has to be lifted for this... 

More seriously, there are > 1000 open bugs in the kernel Bugzilla 
without any activity during the last 2 weeks.

The problem is usually either "Not sexy, and also not at all easy." or
"no maintainer".

Technology can assist, but there are non-technical problems you can't 
solve through technology.

> NeilBrown

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Neil Brown]

On Saturday April 28, mbligh@mbligh.org wrote:
> 
> Yes, human involvement from someone with half a brain would be better.
> Andrew does a lot of that. Not a particularly good use of talent really.
> but still.

I think more than half a brain is needed to do this well.  You need a
reasonable understanding of how all the bits of the kernel work
together so that you have a good chance of sending the bug in the right
direction.  You need a good understanding of the kernel community and
various sub communities so that you know who might be both able and
willing to deal with the bug.  And it wouldn't hurt to have a good
over-view of the current 'hot' areas of the kernel so you know if it
is really worth suggesting "try with the latest -mm" or not.
And you need good people skills.

So I think you really need a lot of up-to-date knowledge to do this
well.  Because of Andrew's position as a funnel, he has a lot of that
knowledge.  It would be really nice if he had some help though.  And I
really think that would mean finding someone in the community who
would rather be coding (and currently are) and convincing them that
there is a higher calling for them.  Finding someone out side or on
the edge of the community is less likely to be effective.

> 
> As Andrew has pointed out before though - even though he forwards
> the bugs, nobody does anything with it. The sad truth seems to be
> that people have very little interest in fixing bugs when they are
> reported - it's not sexy, I guess.

Not sexy, and also not at all easy.   A lot of the interesting bugs
seem to be subtle interactions between separate parts of the kernel -
one part making an assumption or exhibiting a behaviour that the other
part didn't expect.  And we all know that writing bug^Wcode is easier
than removing bugs.  I can spend hour and hours reading through code
trying to get the big picture, and end up finding a one-line change
that then needs documenting, testing and external review.  It's not
easy.

> I'm still unconvinced the users or the tool are the problem, but if it
> makes you happier, we can do that.

No, they aren't the problem.  Bugs are the problem.  But they might be
a more effective part of the solution.

My perception of the kernel bugzilla is that visibility is very low.

I think there is value in weekly reminders, and I wouldn't mind seeing
a weekly Email on linux-kernel with something like a list of open bugs
that have not seen any activity in between 1 and 2 weeks.  It might
get someone out-of-area interested, and might be noticed by someone
who thinks they are in-area and get them wondering why they didn't
find out when the bug was first reported.

NeilBrown

Re: Linux 2.6.21

[Krzysztof Halasa]

Michael Tokarev <mjt@tls.msk.ru> writes:

> For how long you plan to maintain 2.6.x.y -stable series for each
> 2.6.x release?  The thing is that tehere will probably be NO
> .123 "revision"

Actually I meant .1, .2 and maybe even .3 :-)
-- 
Krzysztof Halasa

Re: Linux 2.6.21

[Tomasz Chmielewski]

Adrian Bunk wrote:

> I do hereby promise you to manually ask the submitters of all 1600 open 
> bugs in the kernel Bugzilla within one month whether their problem is 
> still present with 2.6.21 and forwarding all bugs if the answer was 
> "yes" to whoever is the right recipient if you promise me that all bugs 
> where the submitter said "yes" will be debugged by a kernel developer 
> who knows the corresponding subsystem. [4]

Hmm.
I reported a bug on 2006-08-07 ("ISDN/hisax doesn't work on ARM 
architecture"):

http://bugzilla.kernel.org/show_bug.cgi?id=6970


There was not much activity there... and the bug is still opened in 
kernel's bugzilla.


However, I received two emails from Andrew Morton on 02.10.2006 - one 
with a patch, and the second (automated?) which said:

The patch titled

      isdn: work around excessive udelay()

has been removed from the -mm tree.  Its filename is

      isdn-work-around-excessive-udelay.patch

This patch was dropped because it was merged into mainline or a 
subsystem tree


I'm very happy that the bug was fixed, but why wasn't it automatically 
closed in bugzilla (I closed it only today as I looked up into bugzilla)?

Do we have some wrong communication here? How many bugs are there that 
are falsely opened, when in reality they were resolved long ago?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[Russell King]

On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> We are already quite good at ignoring bug reports that come through 
> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> than 1600 open bugs because this tells how bad we are at handling bugs.
> How many thousand bug reports have been ignored during the same time on 
> linux-kernel?

However, look at this bug:

  http://bugme.osdl.org/show_bug.cgi?id=7760

It's outside my knowledge to be able to fix for various reasons:

1. I don't know _anything_ about IXP4xx hardware, so I'm not the right
   person to own this bug.

2. I've no idea who's looking after IXP4xx stuff now.  (When it was
   posted to the ARM kernel list by Rob, it was ignored so I guess the
   IXP4xx maintainer isn't watching for bugs, if such a person even
   exists.)

3. I've little idea about why the MM page allocation is failing early.

4. The ARM DMA bounce code is a hack, and I'm pretty sure that this
   failure is a result of trying to use this contorted code instead
   of the relevant subsystems having a correct DMA mask.

Can you make any suggestion what should be done about this bug?

I'm personally very tempted to close it as "won't fix" (I wish there was
a "can't fix" category.)

As for my other bugs:

  http://bugme.osdl.org/show_bug.cgi?id=8149

Again, EP93xx is not my thing, but I've recently merged a patch to allow
AMBA PL010 uarts (which are present on this platform) to use the clock
control API.

The EP93xx people (provided they're willing) now need to do whatever's
required to resolve that bug.  (Hopefully they've taken ownership of
that bug now.)

  http://bugme.osdl.org/show_bug.cgi?id=4270
  http://bugme.osdl.org/show_bug.cgi?id=5875
  http://bugme.osdl.org/show_bug.cgi?id=7750

I'm no longer serial maintainer.  Bug IDs after about 7000 reflect bugs
submitted since I've resigned my serial maintainership, and therefore
I've ignored them.  It's far easier to ignore bug reports in bugzilla
than it is to get categories reassigned (to whom? - dunno) or even
deleted (if no one steps up presumably that's what needs to happen?)

  http://bugme.osdl.org/show_bug.cgi?id=7389

This one isn't a regression, or even a bug IMHO, but could be viewed as
undesirable behaviour.  Fixing it is utterly non-trivial though.

A serial port which happens to have an IrDA device on it is still a serial
port at the end of the day, and as long as we have legacy probing, a serial
port at a standard COM port address will be detected as such.  There's
never been any facility in the kernel to get around serial ports being
at standard COM port addresses.

Moreover, there's a whole class of applications which want IrDA devices
to be presented to userspace as a serial port rather than a special
network device, so unilaterally ignoring IrDA devices which look like
serial is not really an option.

Basically, IrDA and serial need to develop a way to work in harmony
with each other.  That's a long-term thing though.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 02:13:08PM -0700, Linus Torvalds wrote:
>...
> (I've said this before, but I'll say it again: one thing that would 
> already make bugzilla better is to just always drop any bug reports that 
> are more than a week old and haven't been touched. It wouldn't need *much* 
> touching, but if a reporter cannot be bothered to say "still true with 
> current snapshot" once a week, then it shouldn't be seen as being somehow 
> up to those scare resources we call "developers" to have to go through 
> it).
>...

And if it's a bug in an unmaintained subsystem, a user could do this for 
100 weeks without any effect.
There's no value in keeping reporters busy with useless tasks. [1]

Don't forget:
A good bug report is an important contribution.

We are already quite good at ignoring bug reports that come through 
linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
than 1600 open bugs because this tells how bad we are at handling bugs.
How many thousand bug reports have been ignored during the same time on 
linux-kernel?

If a developer asked for further information and the submitter didn't
answer within 1 months, I will close this bug. [2] And "I will" is not
talking about the future, I'm doing this in the kernel Bugzilla for
three years or so.

The problem is we have tree states of subsystems and drivers:
- unmaintained
- maintained [3]
- maintained and maintainer looks after bug reports

I do hereby promise you to manually ask the submitters of all 1600 open 
bugs in the kernel Bugzilla within one month whether their problem is 
still present with 2.6.21 and forwarding all bugs if the answer was 
"yes" to whoever is the right recipient if you promise me that all bugs 
where the submitter said "yes" will be debugged by a kernel developer 
who knows the corresponding subsystem. [4]

> 		Limis

cu
Adrian

[1] "Still present with the latest kernel?" is a valid question
    if a developer intends to debug further, but otherwise it's silly.
[2] sometimes a bit later, but I'll do it
[3] I'll not do public maintainer bashing, but we have very active 
    maintainers with nearly zero activity in debugging user bug reports
[4] I don't think you'll do - but if you do that's a serious offer

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Martin J. Bligh]

> The thing is, these reports MUST NOT go to "everybody". If they do, that 
> is actually *worse* than nothing, because people will just ignore them 
> entirely, since they aren't "directed".
> 
> The emails need to be directed to the appropriate parties, not go to 
> everybody. There is nobody who is interested in seeing all regressions, 
> except perhaps me and Andrew. Most *real* developers (as opposed to people 
> like me, who are integrators, not "real developers") want to be notified 
> about problems in *their* area, and if it's just automation that sends out 
> everything, it just dilutes the value of the thing, to the point where 
> people will ignore it even for the cases when they happen to be related to 
> what they do.

It's easy to send the different categories to different mailing lists,
if that's what we want to do. Apart from some aggressive filtering on
the SCSI lists etc stops me from bouncing messages to it, but that's
fixable.

Yes, human involvement from someone with half a brain would be better.
Andrew does a lot of that. Not a particularly good use of talent really.
but still.

As Andrew has pointed out before though - even though he forwards
the bugs, nobody does anything with it. The sad truth seems to be
that people have very little interest in fixing bugs when they are
reported - it's not sexy, I guess.

> Let me put it another way: I would never use a source control system that 
> forces me to look at my 22,000 files one at a time. I think such a system 
> is fundamentally broken, because it makes it impossible to get the big 
> picture ("what changed in the last week" kind of thing). The same is true 
> of bugzilla: if you *know* which bug you're looking at, it's good. For 
> anythign else, it's almost worse than useless, exactly because there is no 
> way to get an overview

Go to http://bugzilla.kernel.org. Hit query. Find the box that says
"Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.

74 bugs found.

Not hard to do.

> (I've said this before, but I'll say it again: one thing that would 
> already make bugzilla better is to just always drop any bug reports that 
> are more than a week old and haven't been touched. It wouldn't need *much* 
> touching, but if a reporter cannot be bothered to say "still true with 
> current snapshot" once a week, then it shouldn't be seen as being somehow 
> up to those scare resources we call "developers" to have to go through 
> it).

I'm reluctant to drop / close them. We could fairly easily move them to
a "STALE" state if you want, and have that ping the user. Not sure what
we'd ping them with apart from "Nobody seems to give a toss about your
bug. Life's a bitch. Try sending chocolates, flowers, or fireworks".
I'm still unconvinced the users or the tool are the problem, but if it
makes you happier, we can do that.

> So there are probably things that bugzilla could do to become more useful, 
> but I don't see that happening. We'd need either a smarter/better 
> bugzilla, or somebody who actually turns noise into real information. 
> Adrian did that (although in fairness to others, other people definitely 
> do it too. Dave Jones, for example. Very useful).

What would you want from a smarter / better bugzilla or other bug
tracking tool? A list of requirements / suggestions would be nice. The
main complaint we had before was lack of an email interface, and that
was fixed a long time ago. I admit development has not exactly been
active since, but the only person I got real feedback from was Dave J,
and we've been fixing his UI issues.

M.

Re: Linux 2.6.21

[Markus Rechberger]

On 4/27/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Fri, Apr 27, 2007 at 12:31:43PM -0400, Theodore Tso wrote:
> > On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
> > > "no regressions" is definitely not feasible.
> > >
> > > 14 known regressions, some of them not yet debugged at all, are
> > > different from your "some small regression".
> >
> > Yes, but when were some of these regressions reported?  Past a certain
> > point, I think it's reasonable to look at the regression, decide how
> > many people would be affected by it, and why it hadn't been noticed
> > earlier, and in some cases, decide that it's better to get this
> > debugged and fixed in the stable and development trees in parallel.
>
> 8 of them have been reported in March or earlier. [1]
>
> Patches for 2 of these 8 were available at the time of the release. [2]
> While the question whether to merge one of them into 2.6.21 was
> controversial, the other one was not controversial.
>

I can imagine that the dvb oops bugfix got held back to avoid some
noise with dvb developers who claimed that they didn't get notified
about how that patch got into the v4l-dvb repository (it didn't get
reviewed by these people for weeks because it simply got ignored and
some of them were aware of that)

On the other side if I read bugreports like the following one:
http://www.mail-archive.com/linux-dvb%40linuxtv.org/msg23028.html

(My Nova-T 500 crashes quite regularly. My machine has been running for
about a week and in that time has had 5 oopses.)

It doesn't solve the Nova-T disconnects, but it at least solves that
the machine doesn't go down when this happens till the driver gets
fixed.
I think it would have been nice to get that patch into it

Markus

> For one of the bugs, it became obvious when someone looked at it after
> the release of 2.6.21 that between the bug report on March 31th and the
> release of 2.6.21 on April 21th, noone started debugging this bug. [3] [4]
>
> > > And look e.g. at the many (and non-trivial) changes between -rc7 and
> > > -final, resulting in more than one report from people who were running
> > > -rc7 without problems - and 2.6.21 doesn't work for them.
> >
> > I agree that's unfortunate.
> >
> > > It's not a choice between "regressions don't matter" and "no
> regressions",
> > > it's about the place in the area between these two extremes. I have my
> > > opinions on what I want to expect from a stable Linux kernel, and other
> > > people have different opinins.
> >
> > Everyone is going to disagree to some extent; and their own comfort
> > zone.  So a certain amount compromise is always going to be necessary.
> > Of course, it's up to you decide whether this has gone beyond the zone
> > where you aren't comfortable working with other people's development
> > style.
> >
> > Regards,
> >
> > 					- Ted
>
> cu
> Adrian
>
> [1] http://lkml.org/lkml/2007/4/26/2
> [2] http://lkml.org/lkml/2007/4/25/496
> [3] http://lkml.org/lkml/2007/4/26/496
> [4] and although it turned out this specific regression was already
>     fixed in 2.6.21, I hope you get my point
>
> --
>
>        "Is there not promise of rain?" Ling Tan asked suddenly out
>         of the darkness. There had been need of rain for many days.
>        "Only a promise," Lao Er said.
>                                        Pearl S. Buck - Dragon Seed
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


-- 
Markus Rechberger

Re: Linux 2.6.21

[Daniel Barkalow]

On Thu, 26 Apr 2007, Adrian Bunk wrote:

> Linus said 2.6.20 was a stable kernel. My impression was that at least 
> two of the regressions from my 2.6.20 regressions list should have been 
> fixed before 2.6.20.
> 
> They have both been fixed through -stable, but I also remember a quite 
> experienced kernel maintainer running into one of them after 2.6.20 was 
> released and spending half a day tracking it down - and my answer was
> "known unfixed regression, first reported more than a month ago".

I think there is an issue with two different things being conflated, and 
this causes real stability problems. 2.6.x is both the first kernel in a 
series that is judged to be "stable" and the kernel that is the split 
between 2.6.x.y and 2.6.x+1. This is a fundamental problem, because it 
means that 2.6.x must have all of the problems that are being debugged by 
the people who understand the areas they are in, because 2.6.x+1 has to 
start so that people who are clueless about all of the areas with 
remaining bugs don't spend their time putting more regressions into their 
submissions for 2.6.x+1.

It is also a problem because it is easily possible for a problem to exist 
in 2.6.x-rcN which can only be correctly fixed by doing intrusive things, 
but can be papered over in an obviously-safe way. (E.g., the issue with 
legacy interrupt delivery when MSI is enabled). The intrusive patch could 
easily break a bunch of unrelated stuff, so that's no good for 2.6.x-rcN, 
but papering over bugs is no good for mainline. These bugs have to be 
fixed after the split, which means that the version at the fork must 
contain the bug.

Furthermore, everybody (people reporting bugs, people fixing them, and 
people merging fixes) seem to doze off late in -rc kernels. Having an 
announcement of something with a qualitatively different version wakes 
them up.

I say have a target of no known regressions in 2.6.21.1, with 2.6.21 being 
pretty good, and don't count too much on the stability of 3-number kernel 
versions.

> And a serious delay of the next regression-merge window due to unfixed 
> regressions might even have the positive side effect of more developers 
> becoming interested in fixing the current regressions for getting their 
> shiny new regressions^Wfeatures faster into Linus' tree.

I think the "stick" can't be delaying the window, because that's too 
broad. I think it has to be making people who are needed for fixing stuff 
miss the window. People aren't going to go learn a new area of the kernel 
to resolve regressions in it, but they're more likely to keep their own 
area clean so that they get to merge every 2 months instead of every 4.

> These are just my personal opinions, and other people consider the 
> resulting 2.6.20 and 2.6.21 kernels OK.

I don't think 2.6.x can be OK, by policy. I think 2.6.20.y got to an OK 
state eventually, which is to say that there's no need now to use a 
2.6.19.y kernel. I think that 2.6.21 isn't OK yet, but I think looking for 
an OK 2.6.21-derived kernel is premature still. Ignoring the version 
scheme entirely, I think the success condition should be that the "latest 
stable version of the Linux kernel" link on www.kernel.org is 
always strictly better than all previous links in that spot, and new 
features get there eventually (ideally, within 4 months of hitting 
mainline). I think this is actually possible, although it would require 
changing the policy for this link. And I don't think it requires a change 
in what goes into Linus's git repository when.

Furthermore, I think we're a lot closer to an OK kernel derived from 
Linus's Apr 25 version than we would be if "2.6.21" had not been released 
at that point. It sounds like more items were resolved in the past few 
days than in the preceding week.

Incidentally, will you continue to track 2.6.21 regressions against 
2.6.20? You said there was at least one that you haven't sent out, and 
there's been movement on several others since your last report.

	-Daniel
*This .sig left intentionally blank*

Re: Linux 2.6.21

[Bill Davidsen]

Stephen Clark wrote:

> If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?
> 
Failure of that assumption is the heart of the whole "regression" 
discussion. It's not limited to hardware, kernel security might be an 
issue, some network protocols might work faster and less reliably, etc.

Kernel behavior changes sometimes totally break user software which 
makes unwarranted assumptions. That's not a regression, although users 
may see it that way. When a change in fork() changed the 
child-runs-first behavior, many programs broke, as was true with 
threading changes. Bad reliability is the reward for bad code, but if a 
kernel change makes that obvious some people think it's a regression.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Bill Davidsen]

Adrian Bunk wrote:

> Life has taught me that sometimes I'm right, sometimes I'm wrong, and 
> sometimes both sides have a possible solution. We might agree to 
> disagree, and you are the one who's opinion counts. I can only say that 
> I am not happy with the result, and that I do therefore not spend my 
> time on maintaining regression lists for 2.6.22 - and maintaining such 
> lists is not something special noone else could do equally well.
>
And the next kernel will go out with no list to warn users, and no to-do 
list for -stable.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Bill Davidsen]

Linus Torvalds wrote:
> 
> On Thu, 26 Apr 2007, Bill Davidsen wrote:
>> If the result is fixing things which then don't get fixed in mainline, as
>> Adrian notes
> 
> That whole premise is flawed. The *rule* for the stable tree is that 
> things don't get merged into the stable tree unless they are fixed in 
> mainline already. 
> 
If Adrian cares to note which two regressions he had in mind in his 
previous post <20070426125802.GL3468@stusta.de> or what the exact timing 
was I'll let him continue this.

> We had that problem in the 2.4.x / 2.5.x split. I think we learnt our 
> lesson.
>
I'd love to think that's the case.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Stephen Clark]

Adrian Bunk wrote:

>On Fri, Apr 27, 2007 at 12:31:43PM -0400, Theodore Tso wrote:
>  
>
>>On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
>>    
>>
>>>"no regressions" is definitely not feasible.
>>>
>>>14 known regressions, some of them not yet debugged at all, are 
>>>different from your "some small regression".
>>>      
>>>
>>Yes, but when were some of these regressions reported?  Past a certain
>>point, I think it's reasonable to look at the regression, decide how
>>many people would be affected by it, and why it hadn't been noticed
>>earlier, and in some cases, decide that it's better to get this
>>debugged and fixed in the stable and development trees in parallel.
>>    
>>
>
>8 of them have been reported in March or earlier. [1]
>
>Patches for 2 of these 8 were available at the time of the release. [2]
>While the question whether to merge one of them into 2.6.21 was 
>controversial, the other one was not controversial.
>
>For one of the bugs, it became obvious when someone looked at it after 
>the release of 2.6.21 that between the bug report on March 31th and the 
>release of 2.6.21 on April 21th, noone started debugging this bug. [3] [4]
>
>  
>
>>>And look e.g. at the many (and non-trivial) changes between -rc7 and 
>>>-final, resulting in more than one report from people who were running 
>>>-rc7 without problems - and 2.6.21 doesn't work for them.
>>>      
>>>
>>I agree that's unfortunate.
>>
>>    
>>
>>>It's not a choice between "regressions don't matter" and "no regressions",
>>>it's about the place in the area between these two extremes. I have my 
>>>opinions on what I want to expect from a stable Linux kernel, and other 
>>>people have different opinins.
>>>      
>>>
>>Everyone is going to disagree to some extent; and their own comfort
>>zone.  So a certain amount compromise is always going to be necessary.
>>Of course, it's up to you decide whether this has gone beyond the zone
>>where you aren't comfortable working with other people's development
>>style.
>>
>>Regards,
>>
>>					- Ted
>>    
>>
>
>cu
>Adrian
>
>[1] http://lkml.org/lkml/2007/4/26/2
>[2] http://lkml.org/lkml/2007/4/25/496
>[3] http://lkml.org/lkml/2007/4/26/496
>[4] and although it turned out this specific regression was already 
>    fixed in 2.6.21, I hope you get my point
>
>  
>
What Adrian was doing, or anybody in the future, is not going to be 
productive unless
Linus holds the people who are responsible for the bug to get it fixed 
or report why they
can't.

My $.02
Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Adrian Bunk]

On Fri, Apr 27, 2007 at 12:31:43PM -0400, Theodore Tso wrote:
> On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
> > "no regressions" is definitely not feasible.
> > 
> > 14 known regressions, some of them not yet debugged at all, are 
> > different from your "some small regression".
> 
> Yes, but when were some of these regressions reported?  Past a certain
> point, I think it's reasonable to look at the regression, decide how
> many people would be affected by it, and why it hadn't been noticed
> earlier, and in some cases, decide that it's better to get this
> debugged and fixed in the stable and development trees in parallel.

8 of them have been reported in March or earlier. [1]

Patches for 2 of these 8 were available at the time of the release. [2]
While the question whether to merge one of them into 2.6.21 was 
controversial, the other one was not controversial.

For one of the bugs, it became obvious when someone looked at it after 
the release of 2.6.21 that between the bug report on March 31th and the 
release of 2.6.21 on April 21th, noone started debugging this bug. [3] [4]

> > And look e.g. at the many (and non-trivial) changes between -rc7 and 
> > -final, resulting in more than one report from people who were running 
> > -rc7 without problems - and 2.6.21 doesn't work for them.
> 
> I agree that's unfortunate.
> 
> > It's not a choice between "regressions don't matter" and "no regressions",
> > it's about the place in the area between these two extremes. I have my 
> > opinions on what I want to expect from a stable Linux kernel, and other 
> > people have different opinins.
> 
> Everyone is going to disagree to some extent; and their own comfort
> zone.  So a certain amount compromise is always going to be necessary.
> Of course, it's up to you decide whether this has gone beyond the zone
> where you aren't comfortable working with other people's development
> style.
> 
> Regards,
> 
> 					- Ted

cu
Adrian

[1] http://lkml.org/lkml/2007/4/26/2
[2] http://lkml.org/lkml/2007/4/25/496
[3] http://lkml.org/lkml/2007/4/26/496
[4] and although it turned out this specific regression was already 
    fixed in 2.6.21, I hope you get my point

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Stefan Richter]

Michael Tokarev wrote:
[...]
> there will be just no stable
> kernels *at all*. because when the next 2.6.x will start looking
> more or less useful due to 2.6.x.y series, there will be new 2.6.x+1,
> and work with 2.6.x stops...
> 
> It's not the case currently, but this way ("let's fix the bugs
> in 2.6.x.y -stable series, don't bother releasing 2.6.x in a good
> shape"), we can finally come to the above situation...

Actually, bugs are fixed in --> 2.6.(x+1)-rc <--, and shortly thereafter
a small selection of the fixes are "backported" to 2.6.x.y and to
various distributor kernels.
-- 
Stefan Richter
-=====-=-=== -=-- ==-==
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Michael Tokarev]

Krzysztof Halasa wrote:
[]
> We've got stable series.
> With KNOWN_PROBLEMS information, sysadmins can decide if they can
> safely upgrade to .0 or if they have to wait for .123. Pressing
> the responsible people to fix the problems in .123 (would) help
> it greatly.

For how long you plan to maintain 2.6.x.y -stable series for each
2.6.x release?  The thing is that tehere will probably be NO
.123 "revision" (with maybe the exception of 2.6.16, thanks to
Adrian again).  The end result is that there will be just no stable
kernels *at all*. because when the next 2.6.x will start looking
more or less useful due to 2.6.x.y series, there will be new 2.6.x+1,
and work with 2.6.x stops...

It's not the case currently, but this way ("let's fix the bugs
in 2.6.x.y -stable series, don't bother releasing 2.6.x in a good
shape"), we can finally come to the above situation...

/mjt

Re: Linux 2.6.21

[Theodore Tso]

On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
> "no regressions" is definitely not feasible.
> 
> 14 known regressions, some of them not yet debugged at all, are 
> different from your "some small regression".

Yes, but when were some of these regressions reported?  Past a certain
point, I think it's reasonable to look at the regression, decide how
many people would be affected by it, and why it hadn't been noticed
earlier, and in some cases, decide that it's better to get this
debugged and fixed in the stable and development trees in parallel.

> And look e.g. at the many (and non-trivial) changes between -rc7 and 
> -final, resulting in more than one report from people who were running 
> -rc7 without problems - and 2.6.21 doesn't work for them.

I agree that's unfortunate.

> It's not a choice between "regressions don't matter" and "no regressions",
> it's about the place in the area between these two extremes. I have my 
> opinions on what I want to expect from a stable Linux kernel, and other 
> people have different opinins.

Everyone is going to disagree to some extent; and their own comfort
zone.  So a certain amount compromise is always going to be necessary.
Of course, it's up to you decide whether this has gone beyond the zone
where you aren't comfortable working with other people's development
style.

Regards,

					- Ted

Re: Linux 2.6.21

[Stephen Clark]

Stephen Clark wrote:

>Jeff Garzik wrote:
>
>  
>
>>Stephen Clark wrote:
>> 
>>
>>    
>>
>>>It is laptop that does not have a serial port and I could not couldn't 
>>>get  the
>>>kernel to boot using a usb serial port so I couldn't get a screen 
>>>capture of
>>>the intermittant panic.
>>>   
>>>
>>>      
>>>
>>If you can write it down with a pen and paper, or manually copy the 
>>panic onto another computer (w/ hand and eyes), that would be helpful.
>>
>>	Jeff
>>
>>
>>
>> 
>>
>>    
>>
>Hi Jeff,
>
>It was with FC7 livecd 6.92 - thought I would just try 6.93 which I see 
>has appeared at the mirrors.
>If it still happens I will copy the exception and send it in.
>
>Steve
>
>
>  
>
I downloaded and compile 2.6.21 and so far it seem to be running fine on 
my asus based
S96F whitebook laptop - no panic from the rtl8169

Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 09:50:15PM +0100, Alan Cox wrote:
> > They get frustrated because they focussed on developing new features 
> > instead of fixing regressions, and now it takes longer until their new 
> > features get merged because noone fixed the regressions...
> 
> I would disagree: They get frustrated because they are blocked on some
> small regression which is stopping a ton of other fixed including
> features people need (like new hardware support) from being released.
> 
> The "no regressions" model doesn't really work when you ask about the
> greater good of the userbase.
> 
> The goal of no regressions is great and the regression lists for ATA were
> certainly very helpful but the greater good comes first.

"no regressions" is definitely not feasible.

14 known regressions, some of them not yet debugged at all, are 
different from your "some small regression".

And look e.g. at the many (and non-trivial) changes between -rc7 and 
-final, resulting in more than one report from people who were running 
-rc7 without problems - and 2.6.21 doesn't work for them.

It's not a choice between "regressions don't matter" and "no regressions",
it's about the place in the area between these two extremes. I have my 
opinions on what I want to expect from a stable Linux kernel, and other 
people have different opinins.

> Alan

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Marek Wawrzyczny]

On Fri, 27 Apr 2007 07:13:08 Linus Torvalds wrote:
> I think it could be more interesting if part of the job was doing the
> tools. Tools *are* important. Most of my actual _development_ for the last
> couple of years has been on "git", not the kernel, but I think I was more
> productive that way, so I don't think that's wasted time at all.
>
> So yes, automation would be a good idea, but I don't think bugzilla is it.

<...>

> So I really don't think you can compare to "other projects". They simply
> don't have these issues.

Seems like the bug tracking system needs to be re-evaluated. Perhaps Bugzilla 
can be modified to better serve the needs of kernel developers.

There might be alternatives, JIRA (http://www.atlassian.com/software/jira/) 
for example. I am sure there are others.

Finally, I have over 7 years experience writing various web based systems 
(using WebObjects, J2EE and others). I would be quite willing to volunteer my 
spare time and experience if this community decides that a custom written 
solution (one that would handle email bug submissions/resolutions :) is in 
order.

Kind regards,

Marek Wawrzyczny

Re: Linux 2.6.21

[Jan Engelhardt]

On Apr 26 2007 12:23, Marat Buharov wrote:
>
> [Offtopic]
> Today, April, 26, 21 year has been passed since Chernobyl Nuclear
> Power Plant disaster, and Linus announced .... *drum roll* .... 2.6.21
> !!! What a mysterious coincidence...

And 2.6.26 will be released on April 01 2008.


Jan
-- 

Re: Linux 2.6.21

[Mike Galbraith]

On Thu, 2007-04-26 at 21:02 +0200, Willy Tarreau wrote:
> On Thu, Apr 26, 2007 at 09:40:26AM -0700, Linus Torvalds wrote:
> 
> > So we should have somebody like Christoph running -mm, and when things 
> > break, we'll just sic Christoph on whoever broke it, and teach people 
> > proper fear and respect!
> 
> And with Al Viro doing random code review and fill in the commits for
> regression fixes, even long established developers will check their code
> twice before submitting ;-)

Yeah!  We can call them The Black And Blues Brothers :)

Re: Linux 2.6.21

[Linus Torvalds]

On Fri, 27 Apr 2007, Thomas Gleixner wrote:
> 
> Maybe we need to coordinate changes better. 2.6.21 got three big updates
> which affected suspend/resume - one of them is my fault. But fiddling
> out which one of those - we had nested problems as well - makes it quite
> hard to grok them in time, especially if they happen only on one
> reporters system.

Yes. _If_ we had known how painful the timer changes would end up being, 
we'd probably have done them separately from everything else.

That is the kind of thing that looks obvious in hindsight: merge stuff 
that is questionable and scary alone, and don't do anything else that 
release cycle.

But while the timer code is obviously pretty core, I think everybody 
expected it to be a lot easier to merge (and it had existed as patches in 
various forms for some time).

So we simply didn't know beforehand that it was going to cause the kinds 
of regressions it did cause (and in fact, some of the regressions were 
initially blamed on other things entirely - some of them looked like IO 
regressions).

Water under the bridge. It's also easy to say in hindsight that something 
should have been merged separately and been given a release cycle all its 
own.

			Linus

Re: Linux 2.6.21

[Thomas Gleixner]

Adrian,

On Thu, 2007-04-26 at 14:58 +0200, Adrian Bunk wrote:
> I am aware that my work had some effect, and I am aware that my work 
> gets appreciated - there's no need for everyone to repeat this.

Nevertheless, thanks for your efforts and time spent. You did a great
job and I hope you can convince yourself to carry on.

> The point is: I'm not satisfied with the result.

Nobody is satisfied with pending regressions. I can completely
understand your frustration, but you need to adjust your expectations on
that as well.

Your regression lists are extremly useful, as they point folks like me
to the burning points. I try to follow LKML as far as I can, but I have
to admit that I occasionally go the easy way of marking 10000 mails as
read in one go after a week of travelling. I don't do this to my
personal inbox, so your mails get my attention.

> They have both been fixed through -stable, but I also remember a quite 
> experienced kernel maintainer running into one of them after 2.6.20 was 
> released and spending half a day tracking it down - and my answer was
> "known unfixed regression, first reported more than a month ago".

That happens all the time. I have a dozen of boxen around and I can't do
tests on all of them continously. So trapping into some known regression
is nothing which surprises me.

> There is a conflict between Linus trying to release kernels every
> 2 months and releasing with few regressions.

Yes, it's a conflict, but one that is unresolvable except we want to go
back to the 2.4 model which sucked way more than the current one.

> And a serious delay of the next regression-merge window due to unfixed 
> regressions might even have the positive side effect of more developers 
> becoming interested in fixing the current regressions for getting their 
> shiny new regressions^Wfeatures faster into Linus' tree.

Maybe we need to coordinate changes better. 2.6.21 got three big updates
which affected suspend/resume - one of them is my fault. But fiddling
out which one of those - we had nested problems as well - makes it quite
hard to grok them in time, especially if they happen only on one
reporters system.

Your reports are not invalid, when Linus releases a final. They are
still there and worked on.

I believe we are getting better at that, and one reason for this is your
relentless effort to poke the experts^culprits to actually solve the
problems.

> I'm not satisfied with the result, and the world won't stop turning when 
> I'm not tracking 2.6.22-rc regressions.

Please take a couple of days to reconsider. I personally would welcome
if you carry on.

Thanks,

	tglx

Re: Linux 2.6.21

[Mel Gorman]

On (26/04/07 09:40), Linus Torvalds didst pronounce:
> 
> 
> On Thu, 26 Apr 2007, Jan Engelhardt wrote:
> >
> > I really appreciate the lot of -rcs, especially if there are so many 
> > intrusive changes/regressions. Like Andrew, I have a feeling that it 
> > gets buggier, but at least, it seems to be made up every ... two 
> > releases.
> 
> I wouldn't say that, but yes, there is at least *some* tendency to not 
> merge scary stuff after a painful release.
> 
> For example, I can certainly say that after 2.6.21, I'm likely to be very 
> unhappy merging something that isn't "obviously safe". I knew the timer 
> changes were potentially painful, I just hadn't realized just *how* 
> painful they would be (we had some SATA/IDE changes too, of course, it's 
> not all just about the timers, those just ended up being more noticeable 
> to me than some of the other things were).
> 
> > About 2.6.21 - will see, rc has been to my liking.
> 
> I actually hope that 2.6.21 isn't even all that bad, despite all the 
> worries about it. And I may be complaining about the problems the timers 
> caused, but it was definitely something that was not only worth it, it was 
> overdue - and those NO_HZ issues had been brewing literally for years. So 
> considering issues like that, I think we're actually doing fairly well.
> 
> One of the bigger issues is that I think -mm (and I'm pretty sure Andrew 
> will agree with me on this) has really had a rather spotty history. It's 
> been unstable enough at times that I suspect people have largely stopped 
> testing it, with just the most die-hard testers running -mm.
> 

test.kernel.org also picks up -mm and additional machines run -mm
kernels. It doesn't catch everything but a few bugs get rattled out. That
said, it's automated with Andy Whitcroft and Steve Fox kicking it along
periodically. After spending the day tracking just two issues in -mm and
simple ones at that, the lack of testing may be simply because it's really
bloody boring even with the access to an automated system. There is no
avoiding this really, fixing regressions will never be entertaining.

> So -mm is still very useful just because *Andrew* tests it, and finds all 
> kinds of issues with it, but I literally suspect that Andrew himself is 
> personally a big part of that, which is kind of wasteful - we should be 
> able to spread out the pain more. Andrew is also too damn polite when 
> something goes wrong ;)
> 

A few more "you're a spanner" mails probably would not hurt even though
I'm pretty sure I'll receive a fair number of them :/

> So we should have somebody like Christoph running -mm, and when things 
> break, we'll just sic Christoph on whoever broke it, and teach people 
> proper fear and respect! As it is, I think people tend to send things to 
> -mm a bit *too* eagerly, because there is no downside - Andrew is a "cheap 
> date" testing-wise, and always puts out ;)
> 
> 			Linus

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: Linux 2.6.21

[Mel Gorman]

On (26/04/07 20:45), Adrian Bunk didst pronounce:
> On Thu, Apr 26, 2007 at 08:37:14PM +0200, Krzysztof Halasa wrote:
> >...
> > With KNOWN_PROBLEMS information, sysadmins can decide if they can
> > safely upgrade to .0 or if they have to wait for .123.
> >...
> 
> Listing regressions like the following will most likely be zero help for 
> them when deciding whether to upgrade now or later (and BTW, the latter 
> might imply running a kernel with known security issues):
> 
> Subject    : acpi_pm clocksource loses time on x86-64
> References : http://lkml.org/lkml/2007/4/17/143
> Submitter  : Mikael Pettersson <mikpe@it.uu.se>
> Handled-By : John Stultz <johnstul@us.ibm.com>
>              Len Brown <lenb@kernel.org>
> Status     : problem is being debugged
> 

On the other hand when I was responsible for a bug, it was a great help
to see this sort of mail. Not only as it reminded I was responsible for a
screw-up but when I sent a fix out, these mails let me know it was being
picked up and ensured it made it to mainline.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Diego Calleja wrote:
> 
> Bugzilla sucks quite a lot at email, but you can answer emails and they get
> into the bugzilla database; and there're two mailing lists (listed in
> Documentation/HOWTO) that send notifications about every new bug
> added/modified- I know it's not the perfect email interface every hacker
> wants, but it's better than nothing.

No, it's *not* better than nothing.

The thing is, these reports MUST NOT go to "everybody". If they do, that 
is actually *worse* than nothing, because people will just ignore them 
entirely, since they aren't "directed".

The emails need to be directed to the appropriate parties, not go to 
everybody. There is nobody who is interested in seeing all regressions, 
except perhaps me and Andrew. Most *real* developers (as opposed to people 
like me, who are integrators, not "real developers") want to be notified 
about problems in *their* area, and if it's just automation that sends out 
everything, it just dilutes the value of the thing, to the point where 
people will ignore it even for the cases when they happen to be related to 
what they do.

> I suggested some time ago that it'd be useful to send every new bug
> notification from bugme-new to the LKML (and/or other lists).

I don't know a lot of developers who actually read LKML. I know a lot of 
people who look for interesting subject lines and interesting people, but 
read LKML in the sense of reading everything? Not likely.

That's why I think Adrian did a great job: he took the "noise" and made it 
somethng worth looking at! And part of that is very much to make it 
directred to only relevant parties (yes, they *also* got cc'd to 
linux-kernel, but people would get them in their personal mailboxes and 
*not* feel like it was just noise that didn't matter to them!)

> I can understand Adrian's resign. Bugzilla is crap, but there're users
> reporting bugs there and willing to cooperate to fix them, and they're
> not getting listened.

I personally refuse to have anything at all with bugzilla. The interface 
is so horrible that it's just not worth my time. I know there are a few 
people who use it productively, but I'm always amazed that they can do 
that.

The *big* problem with bugzilla is that it's such a "detail-oriented" 
thing. It's fine if you have *one* bug that you're tracking. But whenever 
that's not the case, it's almost totally useless.

Let me put it another way: I would never use a source control system that 
forces me to look at my 22,000 files one at a time. I think such a system 
is fundamentally broken, because it makes it impossible to get the big 
picture ("what changed in the last week" kind of thing). The same is true 
of bugzilla: if you *know* which bug you're looking at, it's good. For 
anythign else, it's almost worse than useless, exactly because there is no 
way to get an overview.

> There're even a few description of patches (ie: "line
> 6 in foo.c is wrong and it breaks our testing, it should read like this:")
> that have been sitting there for *years* and not getting merged.

.. and you claim that this shows that developers don't listen. I'd say it 
shows the exact *opposite*: the users don't listen. There's a lot mroe 
users than developers, and bugzilla is pretty much designed to let the 
users "report and forget", which is exactly the *wrong* thing to do, 
because it puts the onus on the developer.

(I've said this before, but I'll say it again: one thing that would 
already make bugzilla better is to just always drop any bug reports that 
are more than a week old and haven't been touched. It wouldn't need *much* 
touching, but if a reporter cannot be bothered to say "still true with 
current snapshot" once a week, then it shouldn't be seen as being somehow 
up to those scare resources we call "developers" to have to go through 
it).

So there are probably things that bugzilla could do to become more useful, 
but I don't see that happening. We'd need either a smarter/better 
bugzilla, or somebody who actually turns noise into real information. 
Adrian did that (although in fairness to others, other people definitely 
do it too. Dave Jones, for example. Very useful).

> So I, or anyone else, could try to do Adrian's job. But if Adrian (a guy
> that sends patches to make global functions static 8) got tired
> of doing that job, I suspect that I, or anyone else would also got 
> tired of it even sooner.

I do agree - one of the problems with the job is not that it's thankless 
(I think we've had at least ten kernel developers, very much including me, 
talking about how _useful_ it is), but there is definitely a lack of 
glamour and probably interest.

I think it could be more interesting if part of the job was doing the 
tools. Tools *are* important. Most of my actual _development_ for the last 
couple of years has been on "git", not the kernel, but I think I was more 
productive that way, so I don't think that's wasted time at all.

So yes, automation would be a good idea, but I don't think bugzilla is it. 

> There're other big projects with probably more bug reports than linux, 
> they don't work this way, and they look more succesful in their bug 
> handling.

Well, one thing to keep in mind is that the kernel really does have a 
*lot* more development going on that most other projects.

I don't think you'll find another project that has about six megabytes of 
diffs every release (every two months). That's really one of the 
fundamental issues - things really *happen* in the kernel. A *lot* of 
things. You can't take a breather - I can do "stabilization releases" 
every once in a while, and Andrew can kick out patches he decides aren't 
ready to be merged rather than maintain them in his tree (and he does do 
that), but the kernel simply tends to have a different *scale* than other 
projects.

And almost all hard bugs are about hardware interactions. Drivers. Big 
iron. Things like that - ie unlike something like a compiler, you can 
seldom say "this test-case crashes". Yes, that does happen for the kernel 
too, but those are the *easy* bugs. Those generally get fixed in a day or 
two.

So I really don't think you can compare to "other projects". They simply 
don't have these issues.

		Limis

Re: Linux 2.6.21

[Gene Heskett]

On Thursday 26 April 2007, Stephen Clark wrote:
>Jeff Garzik wrote:
>>IMO, the closer you look, the more warts you find.  Before you starting
>>doing your work with kernel regressions, no one was really tracking it.
>>  I bet you have helped cut down on the regressions, but I have no good
>>way to quantify my gut feeling.
>>
>>Additional comments on developers and fixing regressions:
>>
>>* Sometimes seeing a long list, peoples' eyes glaze over.  Its just
>>human nature.  A long list also gives us no idea of scale, or severity.
>>  I bet a weekly "top 10 bugs and regressions" email would help focus
>>developer attention.
>>
>>* To be effective, lists, either long or top-10, must be pruned if you
>>get a sense that only one user is affected.  [With oopses and BUGs as a
>>clear exception,] many problems benefit from at least two users
>>reporting a bug.
>>
>>* It gets a bit tiresome to field the large number of driver bug reports
>>that eventually turn out to be related to broken interrupt handling
>>somehow.  I think we developers need to get better at showing users how
>>to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start
>>introducing interrupt delivery tests into their probe code.  Overall,
>>broken interrupt handling manifests in several ways, most of which
>>initially appear symptomatic of a broken driver.
>>
>>	Jeff
>>
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>
>Jeff,
>
>If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?
>
>Steve

I think that is indeed a reasonable expectation.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
It is exactly because a man cannot do a thing that he is a proper judge of it.
		-- Oscar Wilde

Re: Linux 2.6.21

[Gene Heskett]

On Thursday 26 April 2007, Stephen Clark wrote:
>Jeff Garzik wrote:
>>IMO, the closer you look, the more warts you find.  Before you starting
>>doing your work with kernel regressions, no one was really tracking it.
>>  I bet you have helped cut down on the regressions, but I have no good
>>way to quantify my gut feeling.
>>
>>Additional comments on developers and fixing regressions:
>>
>>* Sometimes seeing a long list, peoples' eyes glaze over.  Its just
>>human nature.  A long list also gives us no idea of scale, or severity.
>>  I bet a weekly "top 10 bugs and regressions" email would help focus
>>developer attention.
>>
>>* To be effective, lists, either long or top-10, must be pruned if you
>>get a sense that only one user is affected.  [With oopses and BUGs as a
>>clear exception,] many problems benefit from at least two users
>>reporting a bug.
>>
>>* It gets a bit tiresome to field the large number of driver bug reports
>>that eventually turn out to be related to broken interrupt handling
>>somehow.  I think we developers need to get better at showing users how
>>to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start
>>introducing interrupt delivery tests into their probe code.  Overall,
>>broken interrupt handling manifests in several ways, most of which
>>initially appear symptomatic of a broken driver.
>>
>>	Jeff
>>
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>
>Jeff,
>
>If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?
>
>Steve

I think that is indeed a reasonable expectation.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
It is exactly because a man cannot do a thing that he is a proper judge of it.
		-- Oscar Wilde

Re: Linux 2.6.21

[Alan Cox]

> They get frustrated because they focussed on developing new features 
> instead of fixing regressions, and now it takes longer until their new 
> features get merged because noone fixed the regressions...

I would disagree: They get frustrated because they are blocked on some
small regression which is stopping a ton of other fixed including
features people need (like new hardware support) from being released.

The "no regressions" model doesn't really work when you ask about the
greater good of the userbase.

The goal of no regressions is great and the regression lists for ATA were
certainly very helpful but the greater good comes first.

Alan

Re: Linux 2.6.21

[Diego Calleja]

El Thu, 26 Apr 2007 11:42:22 -0700 (PDT), Linus Torvalds <torvalds@linux-foundation.org> escribió:

> I bet that's true even of a lot of people who are more "web oriented" than 
> I am. They may look at webpages, but getting notified by email is still 
> the wakeup call. There's a difference between "active and directed pushing 

Bugzilla sucks quite a lot at email, but you can answer emails and they get
into the bugzilla database; and there're two mailing lists (listed in
Documentation/HOWTO) that send notifications about every new bug
added/modified- I know it's not the perfect email interface every hacker
wants, but it's better than nothing.

I suggested some time ago that it'd be useful to send every new bug
notification from bugme-new to the LKML (and/or other lists). The
volume should not be so high to make it so annoying that it makes it
unuseful, and at least it makes the bugzilla-haters aware of the bugs
reported, and since bugzilla tracks the answers to emails and the
reporter email address is in the email, it makes easier for bugzilla-haters
to ask for more data and try to fix the problem, without starting
any browser.

I can understand Adrian's resign. Bugzilla is crap, but there're users
reporting bugs there and willing to cooperate to fix them, and they're
not getting listened. There're even a few description of patches (ie: "line
6 in foo.c is wrong and it breaks our testing, it should read like this:")
that have been sitting there for *years* and not getting merged. I guess
that Adrian tried to canalize the important regressions to the hackers,
and he got tired of apparently being the only one that cares about getting
them fixed.

So I, or anyone else, could try to do Adrian's job. But if Adrian (a guy
that sends patches to make global functions static 8) got tired
of doing that job, I suspect that I, or anyone else would also got 
tired of it even sooner. There're other big projects with probably
more bug reports than linux, they don't work this way, and they
look more succesful in their bug handling.

So in my humble opinion there's a problem, about how the whole
bug reporting/fixing process works. With the current linux
development model, a good bug reporting/fixing process doesn't
looks optional, since it's important to fix bugs ASAP to get the
fixes into -stable. The fix may go as further as "writing our own
bug tracking software" in the same way git fixed other
development issues, or it may be a human issue, or a mix of the
two.

Re: Linux 2.6.21

[Jan Engelhardt]

On Apr 26 2007 09:40, Linus Torvalds wrote:
>On Thu, 26 Apr 2007, Jan Engelhardt wrote:
>>
>For example, I can certainly say that after 2.6.21, I'm likely to be very 
>unhappy merging something that isn't "obviously safe". I knew the timer 
>changes were potentially painful, I just hadn't realized just *how* 
>painful they would be (we had some SATA/IDE changes too, of course, it's 
>not all just about the timers, those just ended up being more noticeable 
>to me than some of the other things were).

Perhaps do one at a time [ at the cost of queueing other stuff, yeah :( ]
Like: 2.6.21 - only NO_HZ & hrtimers, and the SATA code in .22. Probably does
not work out in reality, so perhaps just live with long rc cycles.
(Let rc8 come.)

>So we should have somebody like Christoph running -mm, and when things 
>break, we'll just sic Christoph on whoever broke it, and teach people 
>proper fear and respect! As it is, I think people tend to send things to 
>-mm a bit *too* eagerly, because there is no downside - Andrew is a "cheap 
>date" testing-wise, and always puts out ;)

Yes, perhaps we need a weakchanges-mm ("weak" is inteded, not to be confused
with week) that can carry stuff like doc updates, Kconfig updates, etc. -
patches that are a little more than -trivial.


Jan
-- 

Re: Linux 2.6.21

[Krzysztof Halasa]

Adrian Bunk <bunk@stusta.de> writes:

> Listing regressions like the following will most likely be zero help for 
> them when deciding whether to upgrade now or later (and BTW, the latter 
> might imply running a kernel with known security issues):
>
> Subject    : acpi_pm clocksource loses time on x86-64
> References : http://lkml.org/lkml/2007/4/17/143
> Submitter  : Mikael Pettersson <mikpe@it.uu.se>
> Handled-By : John Stultz <johnstul@us.ibm.com>
>              Len Brown <lenb@kernel.org>
> Status     : problem is being debugged

I don't think so, any info is better than no info.
-- 
Krzysztof Halasa

Re: Linux 2.6.21

[Stephen Clark]

Francois Romieu wrote:

>Jeff Garzik <jeff@garzik.org> :
>  
>
>>Francois Romieu wrote:
>>    
>>
>>>Pointer for the rtl8139 regression please ?
>>>      
>>>
>>I'm guessing it's this one:
>>
>>    
>>
>>>	Subject    : boot failure: rtl8139: exception in interrupt routine
>>>	References : http://lkml.org/lkml/2007/3/31/160
>>>	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
>>>	Status     : unknown
>>>      
>>>
>>The poster says rtl8139, but doesn't provide more info.  His lspci says 
>>"RTL8169SC", which sounds more like r8169 to me.
>>    
>>
>
>Yes, thanks.
>
>The r8169 driver has been added a few bugfixes between 2.6.21-rc5 and now.
>Some are related to latent, timing changes induced bugs.
>
>Stephen, would you mind testing 2.6.21 and open a PR at bugzilla.kernel.org
>if the bug does not go away ?
>
>Bugzilla e-mails end directly in my mailbox. l-k traffic can be temporarily
>unnoticed (especially on saturday night).
>
>  
>
Sure I'll give it a try in the next couple of days.

Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Stephen Clark]

Jeff Garzik wrote:

>Stephen Clark wrote:
>  
>
>>It is laptop that does not have a serial port and I could not couldn't 
>>get  the
>>kernel to boot using a usb serial port so I couldn't get a screen 
>>capture of
>>the intermittant panic.
>>    
>>
>
>
>If you can write it down with a pen and paper, or manually copy the 
>panic onto another computer (w/ hand and eyes), that would be helpful.
>
>	Jeff
>
>
>
>  
>
Hi Jeff,

It was with FC7 livecd 6.92 - thought I would just try 6.93 which I see 
has appeared at the mirrors.
If it still happens I will copy the exception and send it in.

Steve


-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Francois Romieu]

Jeff Garzik <jeff@garzik.org> :
> Francois Romieu wrote:
> >Pointer for the rtl8139 regression please ?
> 
> I'm guessing it's this one:
> 
> >	Subject    : boot failure: rtl8139: exception in interrupt routine
> >	References : http://lkml.org/lkml/2007/3/31/160
> >	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
> >	Status     : unknown
> 
> 
> The poster says rtl8139, but doesn't provide more info.  His lspci says 
> "RTL8169SC", which sounds more like r8169 to me.

Yes, thanks.

The r8169 driver has been added a few bugfixes between 2.6.21-rc5 and now.
Some are related to latent, timing changes induced bugs.

Stephen, would you mind testing 2.6.21 and open a PR at bugzilla.kernel.org
if the bug does not go away ?

Bugzilla e-mails end directly in my mailbox. l-k traffic can be temporarily
unnoticed (especially on saturday night).

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: Linux 2.6.21

[Stephen Clark]

Jeff Garzik wrote:

>Francois Romieu wrote:
>  
>
>>Pointer for the rtl8139 regression please ?
>>    
>>
>
>I'm guessing it's this one:
>
>  
>
>>	Subject    : boot failure: rtl8139: exception in interrupt routine
>>	References : http://lkml.org/lkml/2007/3/31/160
>>	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
>>	Status     : unknown
>>    
>>
>
>
>The poster says rtl8139, but doesn't provide more info.  His lspci says 
>"RTL8169SC", which sounds more like r8169 to me.
>
>	Jeff
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>
Yeah, that was my bad it is a RTL8169SC, and the 
problem was intermittent
sometimes is cause a panic othertimes it didn't.

It is laptop that does not have a serial port and 
I could not couldn't
get  the
kernel to boot using a usb serial port so I 
couldn't get a screen capture of
the intermittant panic.

Steve

-- 

"They that give up essential liberty to obtain 
temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government 
grows, liberty
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Jeff Garzik]

Stephen Clark wrote:
> If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?

In general, yes.

	Jeff

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 03:13:15PM -0400, Jeff Garzik wrote:
> Francois Romieu wrote:
>> Pointer for the rtl8139 regression please ?
>
> I'm guessing it's this one:
>
>> 	Subject    : boot failure: rtl8139: exception in interrupt routine
>> 	References : http://lkml.org/lkml/2007/3/31/160
>> 	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
>> 	Status     : unknown
>
>
> The poster says rtl8139, but doesn't provide more info.  His lspci says 
> "RTL8169SC", which sounds more like r8169 to me.

Interesting.

I didn't notice it, Andrew didn't notice it, and it seems that although 
it was in three posted versions of my regression lists, noone looked at 
this bug (or it would have been noticed earlier).

> 	Jeff

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Stephen Clark]

Jeff Garzik wrote:

>IMO, the closer you look, the more warts you find.  Before you starting 
>doing your work with kernel regressions, no one was really tracking it. 
>  I bet you have helped cut down on the regressions, but I have no good 
>way to quantify my gut feeling.
>
>Additional comments on developers and fixing regressions:
>
>* Sometimes seeing a long list, peoples' eyes glaze over.  Its just 
>human nature.  A long list also gives us no idea of scale, or severity. 
>  I bet a weekly "top 10 bugs and regressions" email would help focus 
>developer attention.
>
>* To be effective, lists, either long or top-10, must be pruned if you 
>get a sense that only one user is affected.  [With oopses and BUGs as a 
>clear exception,] many problems benefit from at least two users 
>reporting a bug.
>
>* It gets a bit tiresome to field the large number of driver bug reports 
>that eventually turn out to be related to broken interrupt handling 
>somehow.  I think we developers need to get better at showing users how 
>to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start 
>introducing interrupt delivery tests into their probe code.  Overall, 
>broken interrupt handling manifests in several ways, most of which 
>initially appear symptomatic of a broken driver.
>
>	Jeff
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>
Jeff,

If hardware worked in the previous version of the kernel can't users expect
 the same hardware to work in this kernel?

Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 08:58:51PM +0200, Francois Romieu wrote:
> Adrian Bunk <bunk@stusta.de> :
> [...]
> > And how to order the rtl8139 netdriver regression reported one month ago 
> > against the snd_hda_intel regression reported one month ago?
> 
> Pointer for the rtl8139 regression please ?

Subject    : boot failure: rtl8139: exception in interrupt routine
References : http://lkml.org/lkml/2007/3/31/160
Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
Status     : unknown


cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Jeff Garzik]

Francois Romieu wrote:
> Pointer for the rtl8139 regression please ?

I'm guessing it's this one:

> 	Subject    : boot failure: rtl8139: exception in interrupt routine
> 	References : http://lkml.org/lkml/2007/3/31/160
> 	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
> 	Status     : unknown


The poster says rtl8139, but doesn't provide more info.  His lspci says 
"RTL8169SC", which sounds more like r8169 to me.

	Jeff

Re: Linux 2.6.21

[Stephen Clark]

Krzysztof Halasa wrote:

>Adrian Bunk <bunk@stusta.de> writes:
>
>  
>
>>Look at the facts:
>>8 out of 14 regressions in my current list were reported in March or earlier.
>>And for many regressions fixed it took several weeks until debugging 
>>by a kernel developer was started.
>>
>>We do not lack testers for getting bug reports quickly.
>>We lack developer manpower for debugging the many regression reports.
>>    
>>
>
>Quite possible, given the (very) limited range of the bugs. Most people
>just can't debug them.
>
>This isn't IMHO fundamentally wrong, and releasing a ".0" kernel with
>known problems isn't fundamentally wrong either.
>
>What is missing is easily accessible KNOWN_PROBLEMS information for
>released kernels. While I think your work documenting etc. known
>regressions is a very good thing, publishing it with the released
>kernels (certainly .0 and next stable releases, perhaps "quite stable"
>rc versions as well) would be ideal.
>
>A pressure for fixing the bugs is, obviously, the other very good thing.
>
>  
>
>>>2.6.20 was actually really good. Yes, it had some regressions, but I do 
>>>believe that it was one of the least buggy releases we've had. The process 
>>>_worked_.
>>>      
>>>
>
>I the process worked with 2.6.21 as well. Obviously no two releases
>are equal, one has to be better than the other.
>
>  
>
>>>>I'm not satisfied with the result, and the world won't stop turning when 
>>>>I'm not tracking 2.6.22-rc regressions.
>>>>        
>>>>
>
>Anyway, I and many others are satisfied with the result. I think it's
>one of the few "quite recent" things which are a great improvements.
>Other such things are using that weird git thing :-) and perhaps the
>most important - the length of devel cycle under control (I mean the
>lack of "2.5 series" thing).
>
>  
>
>>But I am not happy with the current state of released kernels.
>>    
>>
>
>We've got stable series.
>With KNOWN_PROBLEMS information, sysadmins can decide if they can
>safely upgrade to .0 or if they have to wait for .123. Pressing
>the responsible people to fix the problems in .123 (would) help
>it greatly.
>  
>
The biggest problem I see is that developers want to make
 improvements in an area, like ide, but they don't seem to
look at the old code and make it  sure the new code supports
everything the old code did. This causes hardware that used
to work to not work, or work in a degraded fashion.

My $.02
Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Willy Tarreau]

On Thu, Apr 26, 2007 at 09:40:26AM -0700, Linus Torvalds wrote:

> So we should have somebody like Christoph running -mm, and when things 
> break, we'll just sic Christoph on whoever broke it, and teach people 
> proper fear and respect!

And with Al Viro doing random code review and fill in the commits for
regression fixes, even long established developers will check their code
twice before submitting ;-)

Willy

Re: Linux 2.6.21

[Francois Romieu]

Adrian Bunk <bunk@stusta.de> :
[...]
> And how to order the rtl8139 netdriver regression reported one month ago 
> against the snd_hda_intel regression reported one month ago?

Pointer for the rtl8139 regression please ?

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 08:37:14PM +0200, Krzysztof Halasa wrote:
>...
> With KNOWN_PROBLEMS information, sysadmins can decide if they can
> safely upgrade to .0 or if they have to wait for .123.
>...

Listing regressions like the following will most likely be zero help for 
them when deciding whether to upgrade now or later (and BTW, the latter 
might imply running a kernel with known security issues):

Subject    : acpi_pm clocksource loses time on x86-64
References : http://lkml.org/lkml/2007/4/17/143
Submitter  : Mikael Pettersson <mikpe@it.uu.se>
Handled-By : John Stultz <johnstul@us.ibm.com>
             Len Brown <lenb@kernel.org>
Status     : problem is being debugged

> Krzysztof Halasa

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Diego Calleja wrote:
> 
> From my humble POV, it's a problem that all this discussion was generated
> on what Adrian does or stop doing. Apparently, unless Adrian posts his
> list of know regressions, most of the people doesn't look at the bugzilla
> at all. Maybe it'd be useful to create a per-release bug tracker in the
> bugzilla or collect them into one of the a kernel.org's wiki, to make easier
> to follow the current state of all the "important" regressions. 

Any web-based interface is a no-no. It's one reason I don't use bugzilla a 
lot. If I can't get it by email, it doesn't exist, as far as I'm 
concerned.

I bet that's true even of a lot of people who are more "web oriented" than 
I am. They may look at webpages, but getting notified by email is still 
the wakeup call. There's a difference between "active and directed pushing 
to the involved people" and "the resource exists, that people could look 
at".

So it would have to be more than just a wiki or a bugzilla entry. It would 
have to have that weekly email status thing, and I think that it needs to 
have some human who tries to find messages on the kernel mailing list too, 
and make a first-level judgement on the bugs. Adrian was doing a good job.

But it doesn't necessarily need somebody with intimate knowledge of the 
kernel. In fact, almost everybody who *does* have intimate knowledge tends 
to have so in a very specific area (nobody knows everything - and that 
very much includes people like me and Andrew too) and maybe be skewed in 
other ways too, so a "generalist" is probably more useful than somebody 
who is a "deep coder" in some subsystem.

And it almost certainly doesn't have/prefer to be _one_ person. I suspect 
that this is something where it actually might be better to have some 
collection of people interested in it, and yes, perhaps editing a wiki is 
part of the process, but with at least that "automated email" thing going 
on in additin (and it needs to go to the people involved, not just the 
kernel mailing list - so part of it is not just gathering the reports 
themselves, but also gathering target addresses from MAINTAINERS files and 
perhaps git logs etc).

And yes, it's quite possibly a good way to get into kernel development - 
it definitely helps to know about programming, but as mentioned, I don't 
think it is something where you even need to know specifically about 
*kernel* programming per se.

For example, I don't think it was an accident that Adrian (who has been 
involved in kernelnewbies, janitors and the trivial tree) was the one who 
picked it up. That's exactly the kind of involvement that the regression 
tracking is all about!

(In fact, I think regression tracking might be "easier" to get into than 
actually getting into some of the janitorial projects, exactly because 
it's less about coding. The fact that a person who tracks regressions 
might then *also* indirectly get into the code itself would just be a big 
additional bonus!)

So yes, some automation can almost certainly help (especially if there are 
multiple people involved), but I think a lot of it is that "human 
judgement" and ability to group things and communicate. Are there any 
kernel janitors/newbies/mentors that can think of people who would want to 
do something like this?

			Linus

Re: Linux 2.6.21

[Krzysztof Halasa]

Adrian Bunk <bunk@stusta.de> writes:

> Look at the facts:
> 8 out of 14 regressions in my current list were reported in March or earlier.
> And for many regressions fixed it took several weeks until debugging 
> by a kernel developer was started.
>
> We do not lack testers for getting bug reports quickly.
> We lack developer manpower for debugging the many regression reports.

Quite possible, given the (very) limited range of the bugs. Most people
just can't debug them.

This isn't IMHO fundamentally wrong, and releasing a ".0" kernel with
known problems isn't fundamentally wrong either.

What is missing is easily accessible KNOWN_PROBLEMS information for
released kernels. While I think your work documenting etc. known
regressions is a very good thing, publishing it with the released
kernels (certainly .0 and next stable releases, perhaps "quite stable"
rc versions as well) would be ideal.

A pressure for fixing the bugs is, obviously, the other very good thing.

>> 2.6.20 was actually really good. Yes, it had some regressions, but I do 
>> believe that it was one of the least buggy releases we've had. The process 
>> _worked_.

I the process worked with 2.6.21 as well. Obviously no two releases
are equal, one has to be better than the other.

>> > I'm not satisfied with the result, and the world won't stop turning when 
>> > I'm not tracking 2.6.22-rc regressions.

Anyway, I and many others are satisfied with the result. I think it's
one of the few "quite recent" things which are a great improvements.
Other such things are using that weird git thing :-) and perhaps the
most important - the length of devel cycle under control (I mean the
lack of "2.5 series" thing).

> But I am not happy with the current state of released kernels.

We've got stable series.
With KNOWN_PROBLEMS information, sysadmins can decide if they can
safely upgrade to .0 or if they have to wait for .123. Pressing
the responsible people to fix the problems in .123 (would) help
it greatly.
-- 
Krzysztof Halasa

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 02:04:56PM -0400, Jeff Garzik wrote:
> IMO, the closer you look, the more warts you find.  Before you starting 
> doing your work with kernel regressions, no one was really tracking it.  I 
> bet you have helped cut down on the regressions, but I have no good way to 
> quantify my gut feeling.
>
> Additional comments on developers and fixing regressions:
>
> * Sometimes seeing a long list, peoples' eyes glaze over.  Its just human 
> nature.  A long list also gives us no idea of scale, or severity.  I bet a 
> weekly "top 10 bugs and regressions" email would help focus developer 
> attention.

Top 10 ordered by what?

I always tried to put the most mysterious ones like "kwin dies silently" 
or "gammu no longer works" at the top of my lists hoping some developer 
might become interested in getting clues about them.

But when there were 15 outstanding suspend regressions, these were also 
important.

And how to order the rtl8139 netdriver regression reported one month ago 
against the snd_hda_intel regression reported one month ago?

Both are "only" drivers no longer working for some users, but there 
might be many more users for whom they don't work in 2.6.21 because the 
maintainers didn't bother to debug and fix them.

Ideally, maintainers should debug and fix regressions (and other bugs) 
without requiring weekly regression lists.

If bug reports and weekly reminders aren't enough, I don't think 
increasing the level of babysitting would make sense.

> * To be effective, lists, either long or top-10, must be pruned if you get 
> a sense that only one user is affected.  [With oopses and BUGs as a clear 
> exception,] many problems benefit from at least two users reporting a bug.

First of all we had several cases where one report by one user resulted 
in a serious bug being fixed. There might be only one -rc tester with a 
strange workload or some unusual hardware.

A bigger point is that "at least two users reporting a bug" assumes you 
always know whether two bug reports are for the same bug.
Some examples for problems:
- during early 2.6.21-rc, it was not unusual for users to run into
  three unrelated suspend related regressions on one computer
- during 2.6.20-rc, we had 4 similar looking reports, 3 for one
  regression, and the forth for a completely different regression
- during 2.6.21-rc, it turned out the following reports were caused
  by the same regression:
    kwin dies silently
    KDE problem if a sound is played while the display is suspended

> * It gets a bit tiresome to field the large number of driver bug reports 
> that eventually turn out to be related to broken interrupt handling 
> somehow.  I think we developers need to get better at showing users how to 
> isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start 
> introducing interrupt delivery tests into their probe code.  Overall, 
> broken interrupt handling manifests in several ways, most of which 
> initially appear symptomatic of a broken driver.

Regressions have the advantage of being able to compare with a 
known-good kernel.

If a driver works with 2.6.20 but doesn't work with 2.6.21-rc, ask the 
submitter for dmesg's from both and diff them. In my experience that 
usually gives a good indication whether or not it's an interrupt related 
ACPI/PCI issue.

> 	Jeff

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Diego Calleja]

El Thu, 26 Apr 2007 13:02:28 -0400, Chuck Ebbert <cebbert@redhat.com> escribió:

> Problem is, not enough developers pay attention to the -stable
> series. Adrian, maybe you could shift your attention there and
> stop trying to track the bleeding edge?

>From my humble POV, it's a problem that all this discussion was generated
on what Adrian does or stop doing. Apparently, unless Adrian posts his
list of know regressions, most of the people doesn't look at the bugzilla
at all. Maybe it'd be useful to create a per-release bug tracker in the
bugzilla or collect them into one of the a kernel.org's wiki, to make easier
to follow the current state of all the "important" regressions. 

Re: Linux 2.6.21

[Jeff Garzik]

IMO, the closer you look, the more warts you find.  Before you starting 
doing your work with kernel regressions, no one was really tracking it. 
  I bet you have helped cut down on the regressions, but I have no good 
way to quantify my gut feeling.

Additional comments on developers and fixing regressions:

* Sometimes seeing a long list, peoples' eyes glaze over.  Its just 
human nature.  A long list also gives us no idea of scale, or severity. 
  I bet a weekly "top 10 bugs and regressions" email would help focus 
developer attention.

* To be effective, lists, either long or top-10, must be pruned if you 
get a sense that only one user is affected.  [With oopses and BUGs as a 
clear exception,] many problems benefit from at least two users 
reporting a bug.

* It gets a bit tiresome to field the large number of driver bug reports 
that eventually turn out to be related to broken interrupt handling 
somehow.  I think we developers need to get better at showing users how 
to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start 
introducing interrupt delivery tests into their probe code.  Overall, 
broken interrupt handling manifests in several ways, most of which 
initially appear symptomatic of a broken driver.

	Jeff

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 10:20:46AM -0700, Linus Torvalds wrote:
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
>...
> > But I am not happy with the current state of released kernels.
> 
> So you're going to help exactly how? By stopping to help? Or kvetching 
> about developers that can't figure out why something regressed.
> 
> Sure, that makes tons of sense sense, Adrian.
> 
> NOT.

It is my time, and it's therefore my decision what I consider to make 
sense spending it for.

Instead of continuing our discussion it makes more sense that we simply 
accept that we disagree regarding when a kernel is ready for being 
released instead of repeating the same arguments in a lengthy 
discussion.

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Bill Davidsen wrote:
>
> If the result is fixing things which then don't get fixed in mainline, as
> Adrian notes

That whole premise is flawed. The *rule* for the stable tree is that 
things don't get merged into the stable tree unless they are fixed in 
mainline already. 

We had that problem in the 2.4.x / 2.5.x split. I think we learnt our 
lesson.

			Linus

Re: Linux 2.6.21

[Bill Davidsen]

Linus Torvalds wrote:

> In other words, there's a _reason_ we have staggered development. We have 
> the "crazy development trees" (aka -mm and various other trees), we have 
> the "development tree" (aka Linus' tree), and we have the -stable tree. If 
> the stable tree has a dozen known issues that they'll have to sort out 
> over the next two months, that's *fine*. That's kind of the point of the 
> stable tree.
> 
If the result is fixing things which then don't get fixed in mainline, 
as Adrian notes, then there is something wrong with the process, and why 
will people bother to work on stable if they have doubts that there will 
be long term benefit.

With all the effort the regressions list takes and the stable group puts 
into fixes, someone in charge should insist that regressions fixed in 
stable be fixed in mainline. Since there's only one "someone in charge" 
of policy, I think that's a reasonable commitment to the people doing 
the work.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Bill Davidsen]

Adrian Bunk wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
>> ...
>> So it's been over two and a half months, and while it's certainly not the 
>> longest release cycle ever, it still dragged out a bit longer than I'd 
>> have hoped for and it should have. As usual, I'd like to thank Adrian (and 
>> the people who jumped on the entries Adrian had) for keeping everybody on 
>> their toes with the regression list - there's a few entries there still, 
>> but it got to the point where we didn't even know if they were real 
>> regressions, and delaying things further just wasn't going to help.
>> ...
> 
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release that were first reported in March or earlier:
> 8
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21 
> release [1]:
> 3
> 
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
> 
> 
> We have an astonishing amount of -rc testers, but obviously not the 
> developer manpower for handling them.
> 
> If we would take "no regressions" seriously, it might take 4 or 5 months 
> between releases due to the lack of developer manpower for handling 
> regressions. But that should be considered OK if avoiding regressions 
> was considered more important than getting as quick as possible to the 
> next two week regression-merge window.
> 
> But releasing with so many known regressions is insulting for the many 
> people who spent their time testing -rc kernels.
> 
Without someone holding Linus feet to the fire the next release may be a 
real POS. I think you have done the perfect job, identifying the show 
stoppers, quantifying the obscure and minor regressions, and serving to 
give testing targets as purported fixes are applied.

I don't think you should judge your work by leaving some targets for 
-stable and 2.6.22, but rather from the number of problems you detected, 
documented, and caused to be addressed.

If it were my week to be God, I would insist that the rcN to final step 
was regressions-only, and that all regressions be classified as (a) 
acceptable results of changes to fix other problems, (b) must be fixed 
before release, or (c) obscure enough to tolerate for a short time, must 
be fixed in stable and mainline before N+1 release.

Measuring releases or your own value against perfection is thankless!

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> They get frustrated because they focussed on developing new features 
> instead of fixing regressions, and now it takes longer until their new 
> features get merged because noone fixed the regressions...

I agree. That's part of it. But part of it is not just the "it's 2 months 
until the next release", part of it is also very much a "nothing has 
happened in the normal kernel for the last 8 weeks, this is boring, so 
I'll do my own exciting stuff".

So one _fundmanetal_ issue is that all the people who aren't directly 
involved with a particular regression are simply bored. And bored is not 
good. You want people productive - and that meas that you want a 
active development kernel that they can work with, since they aren't going 
to help with the regressions anyway.

This is why the -stable tree is so useful. It's not only that users want a 
stable tree - it allows people who do *not* have regressions on their 
plate to not be stuck twiddling their thumbs - they can be on the regular 
kernel.

> I'm not saying it always have to be 4 months.

I'm saying that four months wouldn't even have *helped* in the case of 
2.6.21.

Do you really think bugs get fixed faster just because there wasn't a 
release? Quite the reverse. Bugs get _found_ faster thanks to a release 
(simply because you tend to get more information thanks to more users), 
giving the stable people more information, causing the bugs to be able to 
be found and fixed _more_quickly_ in the stable release than if we had 
waited for four months to release 2.6.21.

The two last weeks of 2.6.21-rc were almost entirely "wasted", apart from 
getting the e1000 issue at least resolved (which was the reason for that 
delay, so I'm not complaining - I'm just saying that not a lot of people 
actually were able to _help_ with regressions during that time, and for 
some of them, we might well be better off with more information about the 
issue).

Did we fix other bugs? Yes. There was one long-time bug (since 2.6.15 or 
something) that happened to come in during that time, and we had some 
cleanups, we had MIPS bugs, we found some networking issues etc etc. But 
the amount of combined effort people put on it was pretty weak. 

> "wait for reports to trickle in from testers" is exactly the opposite of 
> our problem.

I disagree. Quite often, having 5 people report the same thing is actually 
more useful (because you see a pattern) than having one known regression 
that you don't know _why_ that regression happened. And that's the case we 
had for most of them. 

You have things like the maintainer (see Oliver's reply, for example) 
simply unable to reproduce it, and needing more information. It 
*does*not*matter* that the original report may be old. If you need more 
information, you need more information, and a two-month-old report isn't 
any better just because it's two months old.

At some point, you need to say: we're not making progress, need to release 
it, that might get us *off* this stuck situation.

That's the part you seem unable to accept. You think that "we have a 
listed regression" means that you should be able to fix it. Not so. We 
*often* need more information. 

> But I am not happy with the current state of released kernels.

So you're going to help exactly how? By stopping to help? Or kvetching 
about developers that can't figure out why something regressed.

Sure, that makes tons of sense sense, Adrian.

NOT.

		Linus

Re: Linux 2.6.21

[Chuck Ebbert]

Linus Torvalds wrote:
> 
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
>> There is a conflict between Linus trying to release kernels every
>> 2 months and releasing with few regressions.
> 
> No. 
> 
> Regressions _increase_ with longer release cycles. They don't get fewer.
> 
> The fact is, we have a -stable series for a reason.

Problem is, not enough developers pay attention to the -stable
series. Adrian, maybe you could shift your attention there and
stop trying to track the bleeding edge?

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 08:47:26AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> > 
> > There is a conflict between Linus trying to release kernels every
> > 2 months and releasing with few regressions.
> 
> No. 
> 
> Regressions _increase_ with longer release cycles. They don't get fewer.
> 
> The fact is, we have a -stable series for a reason. The reason is that the 
> normal development kernel can work in three ways:
> 
>  (a) long release cycles, with two subcases:
> 	(a1) huge changes (ie a "long development series". This is what we 
> 	     used to have. There's no way to even track the regressions, 
> 	     because things just change too much.
> 	(a2) keep the development limited, just stretch out the 
> 	     "stabilization phase". This simply *does*not*work*. You might 
> 	     want it to work, but it's against human psychology. People 
> 	     get bored, and start wasting their time discussing esoteric 
> 	     scheduler issues which weren't regressions at all.
>  (b) Short and staggered release cycle: keep changes limited (like a2), 
>      but recognize when it gets counter-productive, and cut a release so 
>      that the stable team can continue with it, while most developers (who 
>      wouldn't have worked on the stable kernel _anyway_) don't get 
>      frustrated.

<SCNR>

They get frustrated because they focussed on developing new features 
instead of fixing regressions, and now it takes longer until their new 
features get merged because noone fixed the regressions...

</SCNR>

> And yes, we've gone for (b). With occasional "I'm not taking any half-way 
> scary things at _all_" releases, like 2.6.20 was.
> 
> > Trying to avoid regressions might in the worst case result in an -rc12 
> > and 4 months between releases. If the focus is on avoiding regressions 
> > this has to be accepted.
> 
> No. You are ignoring the reality of development. The reality is that you 
> have to balance things. If you have a four-month release cycle, where 

I'm not saying it always have to be 4 months.

> three and a half months are just "wait for reports to trickle in from 
> testers", you simply won't get _anything_ done. People will throw their 
> hands up in frustration and go somewhere else.

"wait for reports to trickle in from testers" is exactly the opposite of 
our problem.

I started the regression lists originally to prove the fairy tale
"noone tests -rc kernels" some kernel developers spread as wrong.

Look at the facts:
8 out of 14 regressions in my current list were reported in March or earlier.
And for many regressions fixed it took several weeks until debugging 
by a kernel developer was started.

We do not lack testers for getting bug reports quickly.
We lack developer manpower for debugging the many regression reports.

>...
> > 0 regressions is never realistic (especially since many regressions 
> > might not be reported during -rc), but IMHO we could do much better than 
> > what happened in 2.6.20 and 2.6.21.
> 
> 2.6.20 was actually really good. Yes, it had some regressions, but I do 
> believe that it was one of the least buggy releases we've had. The process 
> _worked_. 

In the country of the blind the one-eyed man is king...

> 2.6.21 was much less pleasant, but the timer thing really was 
> 
> > I'm not satisfied with the result, and the world won't stop turning when 
> > I'm not tracking 2.6.22-rc regressions.
> 
> True. However, it's sad that you feel like you can't bother to track them. 
> They were _very_ useful. The fact that you felt they weren't is just 
> becasue I think you had unrealistic expectations, and you think that the 
> stable people shouldn't have to have anything to do.
> 
> You're maintaining 2.6.16 yourself - do you not see what happens when you 
> decide that "zero regressions" is the target? You have to stop 
> development. And while that may sound like a good thing at any particular 
> time, it's a total *disaster* in the long run (not even very long, 
> actually: in the two-to-three release cycle kind of run), because while 
> you are in a "regression fix" mode, people still go on developing, and 
> you're just causing problems for the _next_ release by holding things up 
> too long.
> 
> That's the *real* reality: 5 to 7 _million_ lines of diffs in a release 
> every two to three months. Do you really think those changes stop just 
> because of a release process? No. If you drag out the releases to be 4+ 
> months, you'll just have 10-15 million lines of changes instead (or, more 
> likely, you'll have developers who can't be bothered any more, and you may 
> have just 2 million lines, and three years later you have a kernel that 
> isn't relevant any more. Look at any of the other Unixes).

There's not a realistic chance for 0 regressions, and 4 months was 
a worst case, not the average case.

But I am not happy with the current state of released kernels.

> In other words, there's a _reason_ we have staggered development. We have 
> the "crazy development trees" (aka -mm and various other trees), we have 
> the "development tree" (aka Linus' tree), and we have the -stable tree. If 
> the stable tree has a dozen known issues that they'll have to sort out 
> over the next two months, that's *fine*. That's kind of the point of the 
> stable tree.

And all the people who have to upgrade to 2.6.21 for getting an 
important security fix run into a dozen known (and many unknown) 
regressions.

I don't think that's fine.

> And you would helpe them with the 2.6.22-stable releases if you'd maintain 
> that list. Even if it is _designed_ not to go down to zero.
> 
> I suspect that you got overly optimistic from the fact that 2.6.20 really 
> _was_ an easy release. It was designed that way. You feel that it was bad 
> or average, but that's actually because of _your_ unrealistic 
> expectations, not becasue there was anything wrong with 2.6.20.

If we had the developer manpower to get each reported regression 
debugged and fixed [1] within three weeks, 2.6.21 might be in the shape 
I would have liked it to be today.

But there are the three interdependent variables time, developer 
manpower and quality. And few developer manpower and few time results in 
a lower quality of the release I'm not happy with.

Life has taught me that sometimes I'm right, sometimes I'm wrong, and 
sometimes both sides have a possible solution. We might agree to 
disagree, and you are the one who's opinion counts. I can only say that 
I am not happy with the result, and that I do therefore not spend my 
time on maintaining regression lists for 2.6.22 - and maintaining such 
lists is not something special noone else could do equally well.

> 		Linus

cu
Adrian

[1] "fixed" can also be e.g. "patch reverted" or "not a bug"

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Jan Engelhardt wrote:
>
> I really appreciate the lot of -rcs, especially if there are so many 
> intrusive changes/regressions. Like Andrew, I have a feeling that it 
> gets buggier, but at least, it seems to be made up every ... two 
> releases.

I wouldn't say that, but yes, there is at least *some* tendency to not 
merge scary stuff after a painful release.

For example, I can certainly say that after 2.6.21, I'm likely to be very 
unhappy merging something that isn't "obviously safe". I knew the timer 
changes were potentially painful, I just hadn't realized just *how* 
painful they would be (we had some SATA/IDE changes too, of course, it's 
not all just about the timers, those just ended up being more noticeable 
to me than some of the other things were).

> About 2.6.21 - will see, rc has been to my liking.

I actually hope that 2.6.21 isn't even all that bad, despite all the 
worries about it. And I may be complaining about the problems the timers 
caused, but it was definitely something that was not only worth it, it was 
overdue - and those NO_HZ issues had been brewing literally for years. So 
considering issues like that, I think we're actually doing fairly well.

One of the bigger issues is that I think -mm (and I'm pretty sure Andrew 
will agree with me on this) has really had a rather spotty history. It's 
been unstable enough at times that I suspect people have largely stopped 
testing it, with just the most die-hard testers running -mm.

So -mm is still very useful just because *Andrew* tests it, and finds all 
kinds of issues with it, but I literally suspect that Andrew himself is 
personally a big part of that, which is kind of wasteful - we should be 
able to spread out the pain more. Andrew is also too damn polite when 
something goes wrong ;)

So we should have somebody like Christoph running -mm, and when things 
break, we'll just sic Christoph on whoever broke it, and teach people 
proper fear and respect! As it is, I think people tend to send things to 
-mm a bit *too* eagerly, because there is no downside - Andrew is a "cheap 
date" testing-wise, and always puts out ;)

			Linus

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> There is a conflict between Linus trying to release kernels every
> 2 months and releasing with few regressions.

No. 

Regressions _increase_ with longer release cycles. They don't get fewer.

The fact is, we have a -stable series for a reason. The reason is that the 
normal development kernel can work in three ways:

 (a) long release cycles, with two subcases:
	(a1) huge changes (ie a "long development series". This is what we 
	     used to have. There's no way to even track the regressions, 
	     because things just change too much.
	(a2) keep the development limited, just stretch out the 
	     "stabilization phase". This simply *does*not*work*. You might 
	     want it to work, but it's against human psychology. People 
	     get bored, and start wasting their time discussing esoteric 
	     scheduler issues which weren't regressions at all.
 (b) Short and staggered release cycle: keep changes limited (like a2), 
     but recognize when it gets counter-productive, and cut a release so 
     that the stable team can continue with it, while most developers (who 
     wouldn't have worked on the stable kernel _anyway_) don't get 
     frustrated.

And yes, we've gone for (b). With occasional "I'm not taking any half-way 
scary things at _all_" releases, like 2.6.20 was.

> Trying to avoid regressions might in the worst case result in an -rc12 
> and 4 months between releases. If the focus is on avoiding regressions 
> this has to be accepted.

No. You are ignoring the reality of development. The reality is that you 
have to balance things. If you have a four-month release cycle, where 
three and a half months are just "wait for reports to trickle in from 
testers", you simply won't get _anything_ done. People will throw their 
hands up in frustration and go somewhere else.

> And a serious delay of the next regression-merge window due to unfixed 
> regressions might even have the positive side effect of more developers 
> becoming interested in fixing the current regressions for getting their 
> shiny new regressions^Wfeatures faster into Linus' tree.

No. Quite the reverse. 

If we have a problem right now

> 0 regressions is never realistic (especially since many regressions 
> might not be reported during -rc), but IMHO we could do much better than 
> what happened in 2.6.20 and 2.6.21.

2.6.20 was actually really good. Yes, it had some regressions, but I do 
believe that it was one of the least buggy releases we've had. The process 
_worked_. 

2.6.21 was much less pleasant, but the timer thing really was 

> I'm not satisfied with the result, and the world won't stop turning when 
> I'm not tracking 2.6.22-rc regressions.

True. However, it's sad that you feel like you can't bother to track them. 
They were _very_ useful. The fact that you felt they weren't is just 
becasue I think you had unrealistic expectations, and you think that the 
stable people shouldn't have to have anything to do.

You're maintaining 2.6.16 yourself - do you not see what happens when you 
decide that "zero regressions" is the target? You have to stop 
development. And while that may sound like a good thing at any particular 
time, it's a total *disaster* in the long run (not even very long, 
actually: in the two-to-three release cycle kind of run), because while 
you are in a "regression fix" mode, people still go on developing, and 
you're just causing problems for the _next_ release by holding things up 
too long.

That's the *real* reality: 5 to 7 _million_ lines of diffs in a release 
every two to three months. Do you really think those changes stop just 
because of a release process? No. If you drag out the releases to be 4+ 
months, you'll just have 10-15 million lines of changes instead (or, more 
likely, you'll have developers who can't be bothered any more, and you may 
have just 2 million lines, and three years later you have a kernel that 
isn't relevant any more. Look at any of the other Unixes).

In other words, there's a _reason_ we have staggered development. We have 
the "crazy development trees" (aka -mm and various other trees), we have 
the "development tree" (aka Linus' tree), and we have the -stable tree. If 
the stable tree has a dozen known issues that they'll have to sort out 
over the next two months, that's *fine*. That's kind of the point of the 
stable tree.

And you would helpe them with the 2.6.22-stable releases if you'd maintain 
that list. Even if it is _designed_ not to go down to zero.

I suspect that you got overly optimistic from the fact that 2.6.20 really 
_was_ an easy release. It was designed that way. You feel that it was bad 
or average, but that's actually because of _your_ unrealistic 
expectations, not becasue there was anything wrong with 2.6.20.

		Linus

Re: Linux 2.6.21

[Adrian Bunk]

A clarification:

I am aware that my work had some effect, and I am aware that my work 
gets appreciated - there's no need for everyone to repeat this.

The point is: I'm not satisfied with the result.

Linus said 2.6.20 was a stable kernel. My impression was that at least 
two of the regressions from my 2.6.20 regressions list should have been 
fixed before 2.6.20.

They have both been fixed through -stable, but I also remember a quite 
experienced kernel maintainer running into one of them after 2.6.20 was 
released and spending half a day tracking it down - and my answer was
"known unfixed regression, first reported more than a month ago".

There is a conflict between Linus trying to release kernels every
2 months and releasing with few regressions.

Trying to avoid regressions might in the worst case result in an -rc12 
and 4 months between releases. If the focus is on avoiding regressions 
this has to be accepted.

And a serious delay of the next regression-merge window due to unfixed 
regressions might even have the positive side effect of more developers 
becoming interested in fixing the current regressions for getting their 
shiny new regressions^Wfeatures faster into Linus' tree.

0 regressions is never realistic (especially since many regressions 
might not be reported during -rc), but IMHO we could do much better than 
what happened in 2.6.20 and 2.6.21.

These are just my personal opinions, and other people consider the 
resulting 2.6.20 and 2.6.21 kernels OK.

I'm not satisfied with the result, and the world won't stop turning when 
I'm not tracking 2.6.22-rc regressions.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 02:46:08AM -0400, Daniel Barkalow wrote:
> 
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release:
> > 14
> 
> I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found 
> to be inapplicable.

Plus one that has been reported since my last list.

> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release with patches available at the time of the 2.6.21 
> > release [1]:
> > 3
> 
> The -stable team can presumably take care of these in 2.6.21.1, right? 

Two of them are heavily discussed patches, and I'm therefore not sure 
they will ever reach the 2.6.21 branch now that the attention has been 
shifted away from 2.6.21 regressions.

> That leaves 10 that need developer attention.
> 
>  John Stultz seems to be taking care of 3 of them.
> 
>  Oliver Neukum has 1.
> 
>  2 are particular drivers (ali_pata and rtl8139, according to the 
>  reports).
> 
>  2 seem to be ACPI-related; at least one has a candidate patch now.
> 
>  1 seems to be an ALSA problem.
> 
>  1 is STD and being debugged.

You are overinterpreting the Handled-By field in my reports.

It does not imply that this person promised to fix this issue, it only 
says that this person is or was working on this issue.

And more than 50% of the issues were reported first last month or 
earlier and are still unfixed despite repeated reminder emails - if
they weren't fixed until now they might as well become similar to
"foo seems to be broken since at about 2.6.9" issues.

It sounds highly unrealistic that all issues unfixed for a month 
suddenly become fixed even though the main focus of everyone shifts to 
2.6.21.

> It looks like all of the known regressions are being worked on, and 
> getting fixes in for them is -stable material at this point. Furthermore, 
> it doesn't look to me like anyone who is needed for dealing with these 
> regressions is trying to get stuff into the 2.6.22 merge window.
>...

That's a wrong impression, nearly every active kernel developer has at 
least one patch pending for 2.6.22.

> 	-Daniel

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Jesper Juhl]

On 26/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
> >...
> > So it's been over two and a half months, and while it's certainly not the
> > longest release cycle ever, it still dragged out a bit longer than I'd
> > have hoped for and it should have. As usual, I'd like to thank Adrian (and
> > the people who jumped on the entries Adrian had) for keeping everybody on
> > their toes with the regression list - there's a few entries there still,
> > but it got to the point where we didn't even know if they were real
> > regressions, and delaying things further just wasn't going to help.
> >...
>
>
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14
>
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release that were first reported in March or earlier:
> 8
>
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21
> release [1]:
> 3
>
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
>
>
> We have an astonishing amount of -rc testers, but obviously not the
> developer manpower for handling them.
>
> If we would take "no regressions" seriously, it might take 4 or 5 months
> between releases due to the lack of developer manpower for handling
> regressions. But that should be considered OK if avoiding regressions
> was considered more important than getting as quick as possible to the
> next two week regression-merge window.
>
> But releasing with so many known regressions is insulting for the many
> people who spent their time testing -rc kernels.
>

Many people have already said this, but it needs saying again.
You are doing a great job that really helps, both with the regression
lists and the trivial patch monkey stuff. Please keep up the good
work, you are really helping the kernel.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Linux 2.6.21

[Jens Axboe]

On Thu, Apr 26 2007, Adrian Bunk wrote:
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.

Adding my voice to the chorus, I too hope you'll reconsider and continue
doing a great job with the regressions lists. It's really useful!

-- 
Jens Axboe

Re: Linux 2.6.21

[Soeren Sonnenburg]

On Thu, 2007-04-26 at 06:08 +0200, Adrian Bunk wrote:
[...]
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.

Adrian, please reconsider. Without you the issues I've reported (most
likely to the wrong people) would have been missed too. And also keep in
mind that it takes 2 to tango. If reporters can nail down issues to
single config options/patches or go further by adding kprint's chances
that things get fixed increase a lot.

Soeren.
-- 
Sometimes, there's a moment as you're waking, when you become aware of
the real world around you, but you're still dreaming.

Re: Linux 2.6.21

[Jan Engelhardt]

On Apr 25 2007 20:29, Linus Torvalds wrote:
>
>If the goal for 2.6.20 was to be a stable release (and it was), the goal
>for 2.6.21 is to have just survived the big timer-related changes and
>some of the other surprises [...] So it's been over two and a half
>months, and while it's certainly not the longest release cycle ever, it
>still dragged out a bit longer than I'd have hoped for and it should
>have.

I really appreciate the lot of -rcs, especially if there are so many 
intrusive changes/regressions. Like Andrew, I have a feeling that it 
gets buggier, but at least, it seems to be made up every ... two 
releases. 2.6.16 was a good one, .18, and .20. I suppose the 
bug/regression distribution between [2.6.16-2.6.17, 2.6.17-2.6.18] was 
biased like [70, 50]. About 2.6.21 - will see, rc has been to my liking.
Since a picture says more than a thousand words, have
http://jengelh.hopto.org/GFX0/kernel-rating-2.6.21.png
(The last kernel to have only 5 -rcs was 2.6.14 - interesting)


Happy hacking,
Jan
-- 

Re: Linux 2.6.21

[Marat Buharov]

[Offtopic]
Today, April, 26, 21 year has been passed since Chernobyl Nuclear
Power Plant disaster, and Linus announced .... *drum roll* .... 2.6.21
!!! What a mysterious coincidence...

Re: Linux 2.6.21

[Oliver Neukum]

Am Donnerstag, 26. April 2007 08:46 schrieb Daniel Barkalow:
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release:
> > 14
> 
> I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found 
> to be inapplicable.
> 
> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release with patches available at the time of the 2.6.21 
> > release [1]:
> > 3
> 
> The -stable team can presumably take care of these in 2.6.21.1, right? 
> That leaves 10 that need developer attention.
> 
>  John Stultz seems to be taking care of 3 of them.
> 
>  Oliver Neukum has 1.

Which I haven't succeeded in reproducing and gotten no further info on.

	Regards
		Oliver

Re: Linux 2.6.21

[linux]

Today, 26 april, is the *21*'th anniversary of nuclear 
explosion at Chernobyl's station (ex USSR).
And linux 2.6.*21* is released. Nice!

Re: Linux 2.6.21

[Daniel Barkalow]

On Thu, 26 Apr 2007, Adrian Bunk wrote:

> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14

I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found 
to be inapplicable.

> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21 
> release [1]:
> 3

The -stable team can presumably take care of these in 2.6.21.1, right? 
That leaves 10 that need developer attention.

 John Stultz seems to be taking care of 3 of them.

 Oliver Neukum has 1.

 2 are particular drivers (ali_pata and rtl8139, according to the 
 reports).

 2 seem to be ACPI-related; at least one has a candidate patch now.

 1 seems to be an ALSA problem.

 1 is STD and being debugged.

It looks like all of the known regressions are being worked on, and 
getting fixes in for them is -stable material at this point. Furthermore, 
it doesn't look to me like anyone who is needed for dealing with these 
regressions is trying to get stuff into the 2.6.22 merge window.

I think it's clear that this is the right point for Linus to start the 
2.6.22 cycle and leave the rest of the 2.6.21 work to the -stable team, 
who are the experts of taking care of this sort of stuff. Furthermore, it 
seems like -rc testers at this point have found everything in 2.6.21-rc 
they're going to, so, again, it's time for new regressions. Personally, I'd 
vote for having Linus leave off at 2.6.X-final, and have 2.6.X be the 
first -stable release of the series, where the remaining known regressions 
get fixed, but that's an issue of nomenclature, not development process.

I think you've allowed for a well-tested 2.6.21, and a good chance of a 
2.6.21.1 or .2 with no known regressions against 2.6.20, which seems to me 
like you succeeded as far as everything except making Linus a release 
engineer.

	-Daniel
*This .sig left intentionally blank*

Re: Linux 2.6.21

[Jan De Luyck]

On Thursday 26 April 2007, Linus Torvalds wrote:
> If the goal for 2.6.20 was to be a stable release (and it was), the goal
> for 2.6.21 is to have just survived the big timer-related changes and some
> of the other surprises (just as an example: we were apparently unlucky
> enough to hit what looks like a previously unknown hardware errata in one
> of the ethernet drivers that got updated etc).
>
> So it's been over two and a half months, and while it's certainly not the
> longest release cycle ever, it still dragged out a bit longer than I'd
> have hoped for and it should have. As usual, I'd like to thank Adrian (and
> the people who jumped on the entries Adrian had) for keeping everybody on
> their toes with the regression list - there's a few entries there still,
> but it got to the point where we didn't even know if they were real
> regressions, and delaying things further just wasn't going to help.

I've just compiled 2.6.21, seems to run fine, in tickless mode, on a Dell 
D610.

Is there any way to check how many timer ticks are firing, except for 
monitoring /proc/interrupts?

Thanks for all the hard work!

Jan

-- 
She has an alarm clock and a phone that don't ring -- they applaud.

Re: Linux 2.6.21

[Jeff Chua]

On 4/26/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
> >...

> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
>
>
> We have an astonishing amount of -rc testers, but obviously not the
> developer manpower for handling them.

Hey, if I know enough to help, I would. But since there's so many more
talented developers, I can only contribute by testing. Without your
help, I would not have gotten STD/STR working on my notebook for
2.6.21-rcx.

We need definitely need a more robust system.

Thank you,
Jeff.

Re: Linux 2.6.21

[Nick Piggin]

Greg KH wrote:
> On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
> 
>>What I will NOT do:
>>Waste my time with tracking 2.6.22-rc regressions.
> 
> 
> I sure hope you don't do this.
> 
> Tracking these is tough, and I think you are doing a great job with it.
> 
> No release will have no regressions, there's just too many different
> combinations of hardware and sometimes people don't have the time to
> test to see if their original report is even fixed or not.
> 
> And some of them will get fixed with patches coming in the next kernel
> release, which will then be tracked down and added to the -stable
> releases.
> 
> So if you can, please keep it up, if you think it's a thankless job,
> here's my hearty thanks for doing this work.  It's really needed and I
> really appreciate it.

Fifthed here, Adrian. It could potentially become one of the best things
to happen to the mainline release process (and I believe has already been
worthwhile). Even if it takes a while for people to get on board, or some
regressions slip through. And note, a release with regressions doesn't
make your hard work useless -- you've still got the important who, when,
how, etc. info that can be used in future, and it could serve as a "known
issues for upgraders" document as well.

-- 
SUSE Labs, Novell Inc.

Re: Linux 2.6.21

[Willy Tarreau]

On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
> >...
> > So it's been over two and a half months, and while it's certainly not the 
> > longest release cycle ever, it still dragged out a bit longer than I'd 
> > have hoped for and it should have. As usual, I'd like to thank Adrian (and 
> > the people who jumped on the entries Adrian had) for keeping everybody on 
> > their toes with the regression list - there's a few entries there still, 
> > but it got to the point where we didn't even know if they were real 
> > regressions, and delaying things further just wasn't going to help.
> >...
> 
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release that were first reported in March or earlier:
> 8
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21 
> release [1]:
> 3
> 
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
> 
> 
> We have an astonishing amount of -rc testers, but obviously not the 
> developer manpower for handling them.
> 
> If we would take "no regressions" seriously, it might take 4 or 5 months 
> between releases due to the lack of developer manpower for handling 
> regressions. But that should be considered OK if avoiding regressions 
> was considered more important than getting as quick as possible to the 
> next two week regression-merge window.
> 
> But releasing with so many known regressions is insulting for the many 
> people who spent their time testing -rc kernels.

Adrian,

I understand your concerns, it's more and more common to see developers
considering their work is worthless. But it's not. You should see the
current development model as a pipeline. What you feed at the input can
take some time to reach the output, and if we wait for the whole pipeline
to flush, more crap gets released.

What is needed is a higher priority on fixes for known regressions. I
find your summary above more readable than the large lists of regressions.
I think that you should reply to Linus' announces with something that
short, starting from the known-with-patch, known-for-more-than-1-month,
and all-known-regressions. It may help Linus focus even more on those.
Also, while it will not prevent any release with regressions, at least
it will prevent such a stupid case of known regressions with patch
available.

Also, check how many regressions you have reported and which have been
fixed during the -rc stage. You'll see your work really was useful.

Maybe Linus should accept to dedicate -final to known regressions only,
to force a check in this area ? Whether or not all of them get fixed is
not the real problem, but at least we will not have any regressions with
pending patch unapplied !

Please do continue that task if you have the time to do so !

Thanks,
Willy

Re: Linux 2.6.21

[Greg KH]

On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.

I sure hope you don't do this.

Tracking these is tough, and I think you are doing a great job with it.

No release will have no regressions, there's just too many different
combinations of hardware and sometimes people don't have the time to
test to see if their original report is even fixed or not.

And some of them will get fixed with patches coming in the next kernel
release, which will then be tracked down and added to the -stable
releases.

So if you can, please keep it up, if you think it's a thankless job,
here's my hearty thanks for doing this work.  It's really needed and I
really appreciate it.

thanks,

greg k-h

Re: Linux 2.6.21

[Dave Jones]

On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
 
 > What I will NOT do:
 > Waste my time with tracking 2.6.22-rc regressions.

I seriously hope you'll reconsider.  If you hadn't have done this,
things would have been a *lot* worse imo.

But either way, thanks for doing what remains a really grotty
job that may not get you as many kernel groupies as rewriting the
process scheduler, but is equally as (if not moreso) important.

	Dave

-- 
http://www.codemonkey.org.uk

Re: Linux 2.6.21

[Adrian Bunk]

On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
>...
> So it's been over two and a half months, and while it's certainly not the 
> longest release cycle ever, it still dragged out a bit longer than I'd 
> have hoped for and it should have. As usual, I'd like to thank Adrian (and 
> the people who jumped on the entries Adrian had) for keeping everybody on 
> their toes with the regression list - there's a few entries there still, 
> but it got to the point where we didn't even know if they were real 
> regressions, and delaying things further just wasn't going to help.
>...


Number of different known regressions compared to 2.6.20 at the time
of the 2.6.21 release:
14

Number of different known regressions compared to 2.6.20 at the time
of the 2.6.21 release that were first reported in March or earlier:
8

Number of different known regressions compared to 2.6.20 at the time
of the 2.6.21 release with patches available at the time of the 2.6.21 
release [1]:
3

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.


We have an astonishing amount of -rc testers, but obviously not the 
developer manpower for handling them.

If we would take "no regressions" seriously, it might take 4 or 5 months 
between releases due to the lack of developer manpower for handling 
regressions. But that should be considered OK if avoiding regressions 
was considered more important than getting as quick as possible to the 
next two week regression-merge window.

But releasing with so many known regressions is insulting for the many 
people who spent their time testing -rc kernels.


cu
Adrian

[1] http://lkml.org/lkml/2007/4/25/496

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Linux 2.6.21

[Linus Torvalds]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 15080 bytes --]


If the goal for 2.6.20 was to be a stable release (and it was), the goal 
for 2.6.21 is to have just survived the big timer-related changes and some 
of the other surprises (just as an example: we were apparently unlucky 
enough to hit what looks like a previously unknown hardware errata in one 
of the ethernet drivers that got updated etc).

So it's been over two and a half months, and while it's certainly not the 
longest release cycle ever, it still dragged out a bit longer than I'd 
have hoped for and it should have. As usual, I'd like to thank Adrian (and 
the people who jumped on the entries Adrian had) for keeping everybody on 
their toes with the regression list - there's a few entries there still, 
but it got to the point where we didn't even know if they were real 
regressions, and delaying things further just wasn't going to help.

So the big change during 2.6.21 is all the timer changes to support a 
tickless system (and even with ticks, more varied time sources). Thanks 
(when it no longer broke for lots of people ;) go to Thomas Gleixner and 
Ingo Molnar and a cadre of testers and coders.

Of course, the timer stuff was just the most painful and core part (and 
thus the one that I remember most): there's a lot of changes all over. The 
appended changelog is just for the fixes since -rc7, so that doesn't look 
very impressive, the full changes since 2.6.20 are obviously a *lot* 
bigger (and you're better off reading the individual -rc changelogs).

We now return you to your regular scheduler discussions,

		Linus

---
Akinobu Mita (1):
      fault injection: add entry to MAINTAINERS

Alan Cox (3):
      exec.c: fix coredump to pipe problem and obscure "security hole"
      pata_sis: Fix oops on boot
      [SPARC] openprom: Switch to ref counting PCI API

Alexey Dobriyan (1):
      paride drivers: initialize spinlocks

Alexey Kuznetsov (1):
      [NETLINK]: Infinite recursion in netlink.

Andi Kleen (5):
      x86: Fix gcc 4.2 _proxy_pda workaround
      x86: Fix potential overflow in perfctr reservation
      x86: Remove noreplacement option
      x86-64: Always flush all pages in change_page_attr
      i386: Fix some warnings added by earlier patch

Andrea Righi (1):
      [netdrvr] depca: handle platform_device_add() failure

Andrew Morton (4):
      drivers/macintosh/smu.c: fix locking snafu
      acpi-thermal: fix mod_timer() interval
      drivers/net/hamradio/baycom_ser_fdx build fix
      packet: fix error handling

Atsushi Nemoto (3):
      [MIPS] Disallow CpU exception in kernel again.
      [MIPS] Retry {save,restore}_fp_context if failed in atomic context.
      [MIPS] Fix BUG(), BUG_ON() handling

Aubrey.Li (1):
      [NET]: Fix UDP checksum issue in net poll mode.

Avi Kivity (1):
      KVM: Fix off-by-one when writing to a nonpae guest pde

Badari Pulavarty (1):
      cache_k8_northbridges() overflows beyond allocation

Balbir Singh (1):
      Taskstats fix the structure members alignment issue

Bartlomiej Zolnierkiewicz (2):
      ide/Kconfig: add missing range check for IDE_MAX_HWIFS
      Revert "adjust legacy IDE resource setting (v2)"

Bastian Blank (1):
      Allow reading tainted flag as user

Ben Dooks (2):
      [ARM] 4313/1: S3C24XX: Update s3c2410 defconfig to 2.6.21-rc6
      spi: fix use of set_cs in spi_s3c24xx driver

Benjamin Herrenschmidt (1):
      fix bogon in /dev/mem mmap'ing on nommu

Christoph Lameter (1):
      page migration: fix NR_FILE_PAGES accounting

Dan Williams (1):
      usb-net/pegasus: fix pegasus carrier detection

Dave Jiang (1):
      gianfar needs crc32 lib dependency

Dave Johnson (1):
      [MIPS] Fix wrong checksum for split TCP packets on 64-bit MIPS

Dave Jones (1):
      Longhaul - Revert ACPI C3 on Longhaul ver. 2

David Brownell (1):
      MAINTAINERS: use lists.linux-foundation.org

David Rientjes (1):
      oom: kill all threads that share mm with killed task

David S. Miller (2):
      [IPSEC] af_key: Fix thinko in pfkey_xfrm_policy2msg()
      [PARPORT] SUNBPP: Fix OOPS when debugging is enabled.

Denis Lunev (1):
      [NETLINK]: Don't attach callback to a going-away netlink socket

Divy Le Ray (2):
      cxgb3 - Fix low memory conditions
      cxgb3 - PHY interrupts and GPIO pins.

Don Zickus (1):
      allow vmsplice to work in 32-bit mode on ppc64

Evgeniy Dushistov (1):
      ufs proper handling of zero link case

Evgeny Kravtsunov (1):
      [BRIDGE]: Unaligned access when comparing ethernet addresses

Herbert Xu (1):
      [NET]: Get rid of alloc_skb_from_cache

Hugh Dickins (1):
      fix OOM killing processes wrongly thought MPOL_BIND

Ivan Kokshaysky (3):
      alpha: fixes for specific machine types
      alpha: more fixes for specific machine types
      alpha: build fixes - force architecture

Jan Yenya Kasprzak (1):
      Char: mxser_new, fix recursive locking

Jean Delvare (3):
      hwmon/w83627ehf: Fix the fan5 clock divider write
      i2c-pasemi: Depend on PPC_PASEMI again
      hwmon/w83627ehf: Don't redefine REGION_OFFSET

Jeff Mahoney (1):
      reiserfs: fix xattr root locking/refcount bug

Jens Axboe (2):
      cfq-iosched: fix sequential write regression
      cfq-iosched: fix alias + front merge bug

Jiri Kosina (1):
      8250: fix possible deadlock between serial8250_handle_port() and serial8250_interrupt()

Jiri Slaby (3):
      Char: mxser_new, fix TIOCMIWAIT
      Char: mxser, fix TIOCMIWAIT
      Char: icom, mark __init as __devinit

Joachim Deguara (1):
      x86-64: make GART PTEs uncacheable

Kazunori MIYAZAWA (1):
      [KEY]: Fix conversion between IPSEC_MODE_xxx and XFRM_MODE_xxx.

Latchesar Ionkov (1):
      v9fs: don't use primary fid when removing file

Linas Vepstas (1):
      spidernet: Fix problem sending IP fragments

Linus Torvalds (2):
      Revert "e1000: fix NAPI performance on 4-port adapters"
      Linux 2.6.21

Marcel van Nies (3):
      [SUNQE]: Fix MAC address assignment.
      [SUNLANCE]: Fix module unload.
      [SUNHME]: Fix module unload.

Mark Lord (1):
      ide/pci/delkin_cb.c: add new PCI ID

Mark Mason (1):
      [MIPS] Add missing silicon revisions for BCM112x

Michael Buesch (1):
      Add mbuesch to .mailmap

Michael Chan (1):
      [BNX2]: Fix occasional NETDEV WATCHDOG on 5709.

Michael S. Tsirkin (1):
      IB/mthca: Fix data corruption after FMR unmap on Sinai

Miguel Ojeda (1):
      Fix spelling in drivers/video/Kconfig

Neil Horman (1):
      sis900: Allocate rx replacement buffer before rx operation

NeilBrown (1):
      knfsd: use a spinlock to protect sk_info_authunix

Olaf Hering (1):
      do not truncate irq number for icom adapter

Olaf Kirch (1):
      [IrDA]: Correctly handling socket error

Olof Johansson (1):
      Minor bug fixes to i2c-pasemi

Paolo Galtieri (1):
      [SCTP]: Unmap v4mapped addresses during SCTP_BINDX_REM_ADDR operation.

Patrick McHardy (1):
      [XFRM]: beet: fix pseudo header length value

Paul Mackerras (1):
      [PPP]: Fix skbuff.c:BUG due incorrect logic in process_input_packet()

Pavel Emelianov (1):
      [NET]: Set a separate lockdep class for neighbour table's proxy_queue

Ralf Baechle (1):
      [MIPS] Fix oprofile logic to physical counter remapping

Randy Dunlap (1):
      kernel-doc: fix plist.h comments

Russell King (2):
      [ARM] Update mach-types
      Provide dummy devm_ioport_* if !HAS_IOPORT

S.Çağlar Onur (1):
      Add missing USRobotics Wireless Adapter (Model 5423) id into zd1211rw

Sergei Shtylyov (1):
      hpt366: fix kernel oops with HPT302N

Stefan Richter (1):
      ieee1394: update MAINTAINERS database

Stephen Hemminger (7):
      sky2: disable support for 88E8056
      sky2: handle descriptor errors
      sky2: disable ASF on all chip types
      sky2: EC-U performance and jumbo support
      sky2: no jumbo on Yukon FE
      sky2: version 1.14
      [TCP]: Congestion control initialization.

Taku Izumi (1):
      Fix possible NULL pointer access in 8250 serial driver

Trond Myklebust (5):
      NFS: clean up the unstable write code
      NFS: Don't clear PG_writeback until after we've processed unstable writes
      NFS: Fix the 'desynchronized value of nfs_i.ncommit' error
      NFS: Fix race in nfs_set_page_dirty
      RPC: Fix the TCP resend semantics for NFSv4

Tsutomu Fujii (1):
      [SCTP]: Fix assertion (!atomic_read(&sk->sk_rmem_alloc)) failed message

Vlad Yasevich (1):
      [SCTP]: Do not interleave non-fragments when in partial delivery

YOSHIFUJI Hideaki (2):
      [IPV6]: Disallow RH0 by default.
      IPv6: fix Routing Header Type 0 handling thinko

vignesh babu (1):
      [SBUS] vfc_dev.c: kzalloc

---
 .mailmap                                |    2 +
 Documentation/networking/ip-sysctl.txt  |    9 ++
 Documentation/x86_64/boot-options.txt   |    4 -
 MAINTAINERS                             |   39 +++----
 Makefile                                |    2 +-
 arch/alpha/kernel/core_mcpcia.c         |    2 -
 arch/alpha/kernel/err_titan.c           |    1 +
 arch/alpha/kernel/module.c              |    8 +-
 arch/alpha/kernel/sys_nautilus.c        |    6 +
 arch/alpha/kernel/sys_noritake.c        |    9 ++-
 arch/alpha/kernel/sys_rawhide.c         |   15 +++
 arch/alpha/kernel/sys_sio.c             |   14 ++-
 arch/alpha/kernel/sys_sx164.c           |    2 +-
 arch/alpha/kernel/sys_titan.c           |    3 +-
 arch/arm/configs/s3c2410_defconfig      |   11 +-
 arch/arm/tools/mach-types               |   99 ++++++++++++++++-
 arch/i386/kernel/alternative.c          |   21 +---
 arch/i386/kernel/cpu/cpufreq/longhaul.c |    2 +-
 arch/i386/kernel/nmi.c                  |   17 ++--
 arch/i386/kernel/vmlinux.lds.S          |    2 +-
 arch/mips/kernel/r2300_switch.S         |   10 +-
 arch/mips/kernel/r4k_switch.S           |   10 +-
 arch/mips/kernel/signal-common.h        |    9 ++
 arch/mips/kernel/signal.c               |   52 +++++++--
 arch/mips/kernel/signal32.c             |   52 +++++++--
 arch/mips/kernel/traps.c                |   25 +----
 arch/mips/oprofile/op_model_mipsxx.c    |    2 +-
 arch/mips/sibyte/sb1250/setup.c         |   12 ++
 arch/x86_64/kernel/functionlist         |    1 -
 arch/x86_64/kernel/k8.c                 |    4 +-
 arch/x86_64/kernel/nmi.c                |   10 +-
 arch/x86_64/kernel/pci-gart.c           |    6 +-
 arch/x86_64/kernel/vmlinux.lds.S        |    2 +-
 arch/x86_64/mm/pageattr.c               |    2 +-
 block/cfq-iosched.c                     |   46 ++++----
 drivers/acpi/thermal.c                  |    3 +-
 drivers/ata/pata_sis.c                  |   10 +-
 drivers/block/paride/pcd.c              |    2 +-
 drivers/block/paride/pf.c               |    2 +-
 drivers/block/pktcdvd.c                 |    3 +-
 drivers/char/mem.c                      |    2 +-
 drivers/char/mxser.c                    |   48 +++------
 drivers/char/mxser_new.c                |   45 +++-----
 drivers/hwmon/w83627ehf.c               |   20 ++--
 drivers/i2c/busses/Kconfig              |    3 +-
 drivers/i2c/busses/i2c-pasemi.c         |    6 +-
 drivers/ide/Kconfig                     |    1 +
 drivers/ide/pci/delkin_cb.c             |    1 +
 drivers/ide/pci/hpt366.c                |    5 +-
 drivers/infiniband/hw/mthca/mthca_mr.c  |    1 +
 drivers/kvm/mmu.c                       |    1 +
 drivers/macintosh/smu.c                 |    4 +-
 drivers/net/Kconfig                     |    1 +
 drivers/net/bnx2.c                      |    7 +-
 drivers/net/bnx2.h                      |    1 +
 drivers/net/cxgb3/cxgb3_defs.h          |    5 +-
 drivers/net/cxgb3/cxgb3_offload.c       |   69 +++++++++---
 drivers/net/cxgb3/t3_hw.c               |   18 ++-
 drivers/net/depca.c                     |    3 +-
 drivers/net/e1000/e1000_main.c          |   13 +--
 drivers/net/hamradio/baycom_ser_fdx.c   |    6 +-
 drivers/net/ppp_async.c                 |    4 +-
 drivers/net/sis900.c                    |   44 ++++----
 drivers/net/sky2.c                      |  176 ++++++++++++++++++-----------
 drivers/net/sky2.h                      |   11 ++
 drivers/net/spider_net.c                |    2 +-
 drivers/net/sunhme.c                    |    2 +-
 drivers/net/sunlance.c                  |    4 +-
 drivers/net/sunqe.c                     |    4 +-
 drivers/net/wireless/zd1211rw/zd_usb.c  |    1 +
 drivers/parport/parport_sunbpp.c        |   10 +-
 drivers/pci/probe.c                     |   45 ++------
 drivers/sbus/char/openprom.c            |    3 +-
 drivers/sbus/char/vfc_dev.c             |    3 +-
 drivers/serial/8250.c                   |    8 +-
 drivers/serial/icom.c                   |    9 +-
 drivers/serial/icom.h                   |    1 -
 drivers/spi/spi_s3c24xx.c               |    4 +-
 drivers/usb/net/pegasus.c               |   17 ++-
 drivers/usb/net/pegasus.h               |    3 +-
 drivers/video/Kconfig                   |    2 +-
 fs/9p/vfs_inode.c                       |    2 +-
 fs/exec.c                               |   18 ++-
 fs/nfs/write.c                          |  185 ++++++++++++++++++-------------
 fs/reiserfs/xattr.c                     |   92 ++++-----------
 fs/ufs/inode.c                          |   29 ++++-
 include/asm-alpha/compiler.h            |   47 ++++++--
 include/asm-alpha/core_mcpcia.h         |    2 +
 include/asm-alpha/io.h                  |    1 +
 include/asm-mips/bug.h                  |    3 +-
 include/asm-mips/checksum.h             |    2 +-
 include/asm-mips/fpu.h                  |   25 +---
 include/asm-mips/sibyte/sb1250_scd.h    |    1 +
 include/asm-mips/thread_info.h          |    1 -
 include/asm-powerpc/systbl.h            |    2 +-
 include/linux/io.h                      |   13 ++
 include/linux/ipv6.h                    |    3 +
 include/linux/nfs_page.h                |   30 -----
 include/linux/plist.h                   |   54 ++++-----
 include/linux/skbuff.h                  |   10 +-
 include/linux/sysctl.h                  |    1 +
 include/linux/taskstats.h               |   13 ++-
 kernel/sysctl.c                         |    2 +-
 mm/migrate.c                            |   15 +++-
 mm/oom_kill.c                           |    4 +-
 net/bridge/br_stp_if.c                  |    9 +-
 net/core/neighbour.c                    |    5 +-
 net/core/netpoll.c                      |    7 +
 net/core/skbuff.c                       |   55 ---------
 net/ipv4/fib_frontend.c                 |    8 +-
 net/ipv4/tcp_cong.c                     |   23 ++--
 net/ipv4/xfrm4_mode_beet.c              |    4 +-
 net/ipv6/addrconf.c                     |   11 ++
 net/ipv6/exthdrs.c                      |   40 ++++++-
 net/irda/af_irda.c                      |    3 +-
 net/key/af_key.c                        |   90 +++++++++++++---
 net/netlink/af_netlink.c                |    6 +-
 net/sctp/socket.c                       |   54 ++++++++-
 net/sctp/ulpqueue.c                     |    9 ++-
 net/sunrpc/clnt.c                       |    4 +
 net/sunrpc/svcauth_unix.c               |   21 +++-
 net/sunrpc/xprt.c                       |   10 --
 122 files changed, 1230 insertions(+), 828 deletions(-)