Linux 2.6.21

Linux 2.6.21

[Linus Torvalds]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 15080 bytes --]


If the goal for 2.6.20 was to be a stable release (and it was), the goal 
for 2.6.21 is to have just survived the big timer-related changes and some 
of the other surprises (just as an example: we were apparently unlucky 
enough to hit what looks like a previously unknown hardware errata in one 
of the ethernet drivers that got updated etc).

So it's been over two and a half months, and while it's certainly not the 
longest release cycle ever, it still dragged out a bit longer than I'd 
have hoped for and it should have. As usual, I'd like to thank Adrian (and 
the people who jumped on the entries Adrian had) for keeping everybody on 
their toes with the regression list - there's a few entries there still, 
but it got to the point where we didn't even know if they were real 
regressions, and delaying things further just wasn't going to help.

So the big change during 2.6.21 is all the timer changes to support a 
tickless system (and even with ticks, more varied time sources). Thanks 
(when it no longer broke for lots of people ;) go to Thomas Gleixner and 
Ingo Molnar and a cadre of testers and coders.

Of course, the timer stuff was just the most painful and core part (and 
thus the one that I remember most): there's a lot of changes all over. The 
appended changelog is just for the fixes since -rc7, so that doesn't look 
very impressive, the full changes since 2.6.20 are obviously a *lot* 
bigger (and you're better off reading the individual -rc changelogs).

We now return you to your regular scheduler discussions,

		Linus

---
Akinobu Mita (1):
      fault injection: add entry to MAINTAINERS

Alan Cox (3):
      exec.c: fix coredump to pipe problem and obscure "security hole"
      pata_sis: Fix oops on boot
      [SPARC] openprom: Switch to ref counting PCI API

Alexey Dobriyan (1):
      paride drivers: initialize spinlocks

Alexey Kuznetsov (1):
      [NETLINK]: Infinite recursion in netlink.

Andi Kleen (5):
      x86: Fix gcc 4.2 _proxy_pda workaround
      x86: Fix potential overflow in perfctr reservation
      x86: Remove noreplacement option
      x86-64: Always flush all pages in change_page_attr
      i386: Fix some warnings added by earlier patch

Andrea Righi (1):
      [netdrvr] depca: handle platform_device_add() failure

Andrew Morton (4):
      drivers/macintosh/smu.c: fix locking snafu
      acpi-thermal: fix mod_timer() interval
      drivers/net/hamradio/baycom_ser_fdx build fix
      packet: fix error handling

Atsushi Nemoto (3):
      [MIPS] Disallow CpU exception in kernel again.
      [MIPS] Retry {save,restore}_fp_context if failed in atomic context.
      [MIPS] Fix BUG(), BUG_ON() handling

Aubrey.Li (1):
      [NET]: Fix UDP checksum issue in net poll mode.

Avi Kivity (1):
      KVM: Fix off-by-one when writing to a nonpae guest pde

Badari Pulavarty (1):
      cache_k8_northbridges() overflows beyond allocation

Balbir Singh (1):
      Taskstats fix the structure members alignment issue

Bartlomiej Zolnierkiewicz (2):
      ide/Kconfig: add missing range check for IDE_MAX_HWIFS
      Revert "adjust legacy IDE resource setting (v2)"

Bastian Blank (1):
      Allow reading tainted flag as user

Ben Dooks (2):
      [ARM] 4313/1: S3C24XX: Update s3c2410 defconfig to 2.6.21-rc6
      spi: fix use of set_cs in spi_s3c24xx driver

Benjamin Herrenschmidt (1):
      fix bogon in /dev/mem mmap'ing on nommu

Christoph Lameter (1):
      page migration: fix NR_FILE_PAGES accounting

Dan Williams (1):
      usb-net/pegasus: fix pegasus carrier detection

Dave Jiang (1):
      gianfar needs crc32 lib dependency

Dave Johnson (1):
      [MIPS] Fix wrong checksum for split TCP packets on 64-bit MIPS

Dave Jones (1):
      Longhaul - Revert ACPI C3 on Longhaul ver. 2

David Brownell (1):
      MAINTAINERS: use lists.linux-foundation.org

David Rientjes (1):
      oom: kill all threads that share mm with killed task

David S. Miller (2):
      [IPSEC] af_key: Fix thinko in pfkey_xfrm_policy2msg()
      [PARPORT] SUNBPP: Fix OOPS when debugging is enabled.

Denis Lunev (1):
      [NETLINK]: Don't attach callback to a going-away netlink socket

Divy Le Ray (2):
      cxgb3 - Fix low memory conditions
      cxgb3 - PHY interrupts and GPIO pins.

Don Zickus (1):
      allow vmsplice to work in 32-bit mode on ppc64

Evgeniy Dushistov (1):
      ufs proper handling of zero link case

Evgeny Kravtsunov (1):
      [BRIDGE]: Unaligned access when comparing ethernet addresses

Herbert Xu (1):
      [NET]: Get rid of alloc_skb_from_cache

Hugh Dickins (1):
      fix OOM killing processes wrongly thought MPOL_BIND

Ivan Kokshaysky (3):
      alpha: fixes for specific machine types
      alpha: more fixes for specific machine types
      alpha: build fixes - force architecture

Jan Yenya Kasprzak (1):
      Char: mxser_new, fix recursive locking

Jean Delvare (3):
      hwmon/w83627ehf: Fix the fan5 clock divider write
      i2c-pasemi: Depend on PPC_PASEMI again
      hwmon/w83627ehf: Don't redefine REGION_OFFSET

Jeff Mahoney (1):
      reiserfs: fix xattr root locking/refcount bug

Jens Axboe (2):
      cfq-iosched: fix sequential write regression
      cfq-iosched: fix alias + front merge bug

Jiri Kosina (1):
      8250: fix possible deadlock between serial8250_handle_port() and serial8250_interrupt()

Jiri Slaby (3):
      Char: mxser_new, fix TIOCMIWAIT
      Char: mxser, fix TIOCMIWAIT
      Char: icom, mark __init as __devinit

Joachim Deguara (1):
      x86-64: make GART PTEs uncacheable

Kazunori MIYAZAWA (1):
      [KEY]: Fix conversion between IPSEC_MODE_xxx and XFRM_MODE_xxx.

Latchesar Ionkov (1):
      v9fs: don't use primary fid when removing file

Linas Vepstas (1):
      spidernet: Fix problem sending IP fragments

Linus Torvalds (2):
      Revert "e1000: fix NAPI performance on 4-port adapters"
      Linux 2.6.21

Marcel van Nies (3):
      [SUNQE]: Fix MAC address assignment.
      [SUNLANCE]: Fix module unload.
      [SUNHME]: Fix module unload.

Mark Lord (1):
      ide/pci/delkin_cb.c: add new PCI ID

Mark Mason (1):
      [MIPS] Add missing silicon revisions for BCM112x

Michael Buesch (1):
      Add mbuesch to .mailmap

Michael Chan (1):
      [BNX2]: Fix occasional NETDEV WATCHDOG on 5709.

Michael S. Tsirkin (1):
      IB/mthca: Fix data corruption after FMR unmap on Sinai

Miguel Ojeda (1):
      Fix spelling in drivers/video/Kconfig

Neil Horman (1):
      sis900: Allocate rx replacement buffer before rx operation

NeilBrown (1):
      knfsd: use a spinlock to protect sk_info_authunix

Olaf Hering (1):
      do not truncate irq number for icom adapter

Olaf Kirch (1):
      [IrDA]: Correctly handling socket error

Olof Johansson (1):
      Minor bug fixes to i2c-pasemi

Paolo Galtieri (1):
      [SCTP]: Unmap v4mapped addresses during SCTP_BINDX_REM_ADDR operation.

Patrick McHardy (1):
      [XFRM]: beet: fix pseudo header length value

Paul Mackerras (1):
      [PPP]: Fix skbuff.c:BUG due incorrect logic in process_input_packet()

Pavel Emelianov (1):
      [NET]: Set a separate lockdep class for neighbour table's proxy_queue

Ralf Baechle (1):
      [MIPS] Fix oprofile logic to physical counter remapping

Randy Dunlap (1):
      kernel-doc: fix plist.h comments

Russell King (2):
      [ARM] Update mach-types
      Provide dummy devm_ioport_* if !HAS_IOPORT

S.Çağlar Onur (1):
      Add missing USRobotics Wireless Adapter (Model 5423) id into zd1211rw

Sergei Shtylyov (1):
      hpt366: fix kernel oops with HPT302N

Stefan Richter (1):
      ieee1394: update MAINTAINERS database

Stephen Hemminger (7):
      sky2: disable support for 88E8056
      sky2: handle descriptor errors
      sky2: disable ASF on all chip types
      sky2: EC-U performance and jumbo support
      sky2: no jumbo on Yukon FE
      sky2: version 1.14
      [TCP]: Congestion control initialization.

Taku Izumi (1):
      Fix possible NULL pointer access in 8250 serial driver

Trond Myklebust (5):
      NFS: clean up the unstable write code
      NFS: Don't clear PG_writeback until after we've processed unstable writes
      NFS: Fix the 'desynchronized value of nfs_i.ncommit' error
      NFS: Fix race in nfs_set_page_dirty
      RPC: Fix the TCP resend semantics for NFSv4

Tsutomu Fujii (1):
      [SCTP]: Fix assertion (!atomic_read(&sk->sk_rmem_alloc)) failed message

Vlad Yasevich (1):
      [SCTP]: Do not interleave non-fragments when in partial delivery

YOSHIFUJI Hideaki (2):
      [IPV6]: Disallow RH0 by default.
      IPv6: fix Routing Header Type 0 handling thinko

vignesh babu (1):
      [SBUS] vfc_dev.c: kzalloc

---
 .mailmap                                |    2 +
 Documentation/networking/ip-sysctl.txt  |    9 ++
 Documentation/x86_64/boot-options.txt   |    4 -
 MAINTAINERS                             |   39 +++----
 Makefile                                |    2 +-
 arch/alpha/kernel/core_mcpcia.c         |    2 -
 arch/alpha/kernel/err_titan.c           |    1 +
 arch/alpha/kernel/module.c              |    8 +-
 arch/alpha/kernel/sys_nautilus.c        |    6 +
 arch/alpha/kernel/sys_noritake.c        |    9 ++-
 arch/alpha/kernel/sys_rawhide.c         |   15 +++
 arch/alpha/kernel/sys_sio.c             |   14 ++-
 arch/alpha/kernel/sys_sx164.c           |    2 +-
 arch/alpha/kernel/sys_titan.c           |    3 +-
 arch/arm/configs/s3c2410_defconfig      |   11 +-
 arch/arm/tools/mach-types               |   99 ++++++++++++++++-
 arch/i386/kernel/alternative.c          |   21 +---
 arch/i386/kernel/cpu/cpufreq/longhaul.c |    2 +-
 arch/i386/kernel/nmi.c                  |   17 ++--
 arch/i386/kernel/vmlinux.lds.S          |    2 +-
 arch/mips/kernel/r2300_switch.S         |   10 +-
 arch/mips/kernel/r4k_switch.S           |   10 +-
 arch/mips/kernel/signal-common.h        |    9 ++
 arch/mips/kernel/signal.c               |   52 +++++++--
 arch/mips/kernel/signal32.c             |   52 +++++++--
 arch/mips/kernel/traps.c                |   25 +----
 arch/mips/oprofile/op_model_mipsxx.c    |    2 +-
 arch/mips/sibyte/sb1250/setup.c         |   12 ++
 arch/x86_64/kernel/functionlist         |    1 -
 arch/x86_64/kernel/k8.c                 |    4 +-
 arch/x86_64/kernel/nmi.c                |   10 +-
 arch/x86_64/kernel/pci-gart.c           |    6 +-
 arch/x86_64/kernel/vmlinux.lds.S        |    2 +-
 arch/x86_64/mm/pageattr.c               |    2 +-
 block/cfq-iosched.c                     |   46 ++++----
 drivers/acpi/thermal.c                  |    3 +-
 drivers/ata/pata_sis.c                  |   10 +-
 drivers/block/paride/pcd.c              |    2 +-
 drivers/block/paride/pf.c               |    2 +-
 drivers/block/pktcdvd.c                 |    3 +-
 drivers/char/mem.c                      |    2 +-
 drivers/char/mxser.c                    |   48 +++------
 drivers/char/mxser_new.c                |   45 +++-----
 drivers/hwmon/w83627ehf.c               |   20 ++--
 drivers/i2c/busses/Kconfig              |    3 +-
 drivers/i2c/busses/i2c-pasemi.c         |    6 +-
 drivers/ide/Kconfig                     |    1 +
 drivers/ide/pci/delkin_cb.c             |    1 +
 drivers/ide/pci/hpt366.c                |    5 +-
 drivers/infiniband/hw/mthca/mthca_mr.c  |    1 +
 drivers/kvm/mmu.c                       |    1 +
 drivers/macintosh/smu.c                 |    4 +-
 drivers/net/Kconfig                     |    1 +
 drivers/net/bnx2.c                      |    7 +-
 drivers/net/bnx2.h                      |    1 +
 drivers/net/cxgb3/cxgb3_defs.h          |    5 +-
 drivers/net/cxgb3/cxgb3_offload.c       |   69 +++++++++---
 drivers/net/cxgb3/t3_hw.c               |   18 ++-
 drivers/net/depca.c                     |    3 +-
 drivers/net/e1000/e1000_main.c          |   13 +--
 drivers/net/hamradio/baycom_ser_fdx.c   |    6 +-
 drivers/net/ppp_async.c                 |    4 +-
 drivers/net/sis900.c                    |   44 ++++----
 drivers/net/sky2.c                      |  176 ++++++++++++++++++-----------
 drivers/net/sky2.h                      |   11 ++
 drivers/net/spider_net.c                |    2 +-
 drivers/net/sunhme.c                    |    2 +-
 drivers/net/sunlance.c                  |    4 +-
 drivers/net/sunqe.c                     |    4 +-
 drivers/net/wireless/zd1211rw/zd_usb.c  |    1 +
 drivers/parport/parport_sunbpp.c        |   10 +-
 drivers/pci/probe.c                     |   45 ++------
 drivers/sbus/char/openprom.c            |    3 +-
 drivers/sbus/char/vfc_dev.c             |    3 +-
 drivers/serial/8250.c                   |    8 +-
 drivers/serial/icom.c                   |    9 +-
 drivers/serial/icom.h                   |    1 -
 drivers/spi/spi_s3c24xx.c               |    4 +-
 drivers/usb/net/pegasus.c               |   17 ++-
 drivers/usb/net/pegasus.h               |    3 +-
 drivers/video/Kconfig                   |    2 +-
 fs/9p/vfs_inode.c                       |    2 +-
 fs/exec.c                               |   18 ++-
 fs/nfs/write.c                          |  185 ++++++++++++++++++-------------
 fs/reiserfs/xattr.c                     |   92 ++++-----------
 fs/ufs/inode.c                          |   29 ++++-
 include/asm-alpha/compiler.h            |   47 ++++++--
 include/asm-alpha/core_mcpcia.h         |    2 +
 include/asm-alpha/io.h                  |    1 +
 include/asm-mips/bug.h                  |    3 +-
 include/asm-mips/checksum.h             |    2 +-
 include/asm-mips/fpu.h                  |   25 +---
 include/asm-mips/sibyte/sb1250_scd.h    |    1 +
 include/asm-mips/thread_info.h          |    1 -
 include/asm-powerpc/systbl.h            |    2 +-
 include/linux/io.h                      |   13 ++
 include/linux/ipv6.h                    |    3 +
 include/linux/nfs_page.h                |   30 -----
 include/linux/plist.h                   |   54 ++++-----
 include/linux/skbuff.h                  |   10 +-
 include/linux/sysctl.h                  |    1 +
 include/linux/taskstats.h               |   13 ++-
 kernel/sysctl.c                         |    2 +-
 mm/migrate.c                            |   15 +++-
 mm/oom_kill.c                           |    4 +-
 net/bridge/br_stp_if.c                  |    9 +-
 net/core/neighbour.c                    |    5 +-
 net/core/netpoll.c                      |    7 +
 net/core/skbuff.c                       |   55 ---------
 net/ipv4/fib_frontend.c                 |    8 +-
 net/ipv4/tcp_cong.c                     |   23 ++--
 net/ipv4/xfrm4_mode_beet.c              |    4 +-
 net/ipv6/addrconf.c                     |   11 ++
 net/ipv6/exthdrs.c                      |   40 ++++++-
 net/irda/af_irda.c                      |    3 +-
 net/key/af_key.c                        |   90 +++++++++++++---
 net/netlink/af_netlink.c                |    6 +-
 net/sctp/socket.c                       |   54 ++++++++-
 net/sctp/ulpqueue.c                     |    9 ++-
 net/sunrpc/clnt.c                       |    4 +
 net/sunrpc/svcauth_unix.c               |   21 +++-
 net/sunrpc/xprt.c                       |   10 --
 122 files changed, 1230 insertions(+), 828 deletions(-)

Re: Linux 2.6.21

[Adrian Bunk]

On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
>...
> So it's been over two and a half months, and while it's certainly not the 
> longest release cycle ever, it still dragged out a bit longer than I'd 
> have hoped for and it should have. As usual, I'd like to thank Adrian (and 
> the people who jumped on the entries Adrian had) for keeping everybody on 
> their toes with the regression list - there's a few entries there still, 
> but it got to the point where we didn't even know if they were real 
> regressions, and delaying things further just wasn't going to help.
>...


Number of different known regressions compared to 2.6.20 at the time
of the 2.6.21 release:
14

Number of different known regressions compared to 2.6.20 at the time
of the 2.6.21 release that were first reported in March or earlier:
8

Number of different known regressions compared to 2.6.20 at the time
of the 2.6.21 release with patches available at the time of the 2.6.21 
release [1]:
3

What I will NOT do:
Waste my time with tracking 2.6.22-rc regressions.


We have an astonishing amount of -rc testers, but obviously not the 
developer manpower for handling them.

If we would take "no regressions" seriously, it might take 4 or 5 months 
between releases due to the lack of developer manpower for handling 
regressions. But that should be considered OK if avoiding regressions 
was considered more important than getting as quick as possible to the 
next two week regression-merge window.

But releasing with so many known regressions is insulting for the many 
people who spent their time testing -rc kernels.


cu
Adrian

[1] http://lkml.org/lkml/2007/4/25/496

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Dave Jones]

On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
 
 > What I will NOT do:
 > Waste my time with tracking 2.6.22-rc regressions.

I seriously hope you'll reconsider.  If you hadn't have done this,
things would have been a *lot* worse imo.

But either way, thanks for doing what remains a really grotty
job that may not get you as many kernel groupies as rewriting the
process scheduler, but is equally as (if not moreso) important.

	Dave

-- 
http://www.codemonkey.org.uk

Re: Linux 2.6.21

[Greg KH]

On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.

I sure hope you don't do this.

Tracking these is tough, and I think you are doing a great job with it.

No release will have no regressions, there's just too many different
combinations of hardware and sometimes people don't have the time to
test to see if their original report is even fixed or not.

And some of them will get fixed with patches coming in the next kernel
release, which will then be tracked down and added to the -stable
releases.

So if you can, please keep it up, if you think it's a thankless job,
here's my hearty thanks for doing this work.  It's really needed and I
really appreciate it.

thanks,

greg k-h

Re: Linux 2.6.21

[Nick Piggin]

Greg KH wrote:
> On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
> 
>>What I will NOT do:
>>Waste my time with tracking 2.6.22-rc regressions.
> 
> 
> I sure hope you don't do this.
> 
> Tracking these is tough, and I think you are doing a great job with it.
> 
> No release will have no regressions, there's just too many different
> combinations of hardware and sometimes people don't have the time to
> test to see if their original report is even fixed or not.
> 
> And some of them will get fixed with patches coming in the next kernel
> release, which will then be tracked down and added to the -stable
> releases.
> 
> So if you can, please keep it up, if you think it's a thankless job,
> here's my hearty thanks for doing this work.  It's really needed and I
> really appreciate it.

Fifthed here, Adrian. It could potentially become one of the best things
to happen to the mainline release process (and I believe has already been
worthwhile). Even if it takes a while for people to get on board, or some
regressions slip through. And note, a release with regressions doesn't
make your hard work useless -- you've still got the important who, when,
how, etc. info that can be used in future, and it could serve as a "known
issues for upgraders" document as well.

-- 
SUSE Labs, Novell Inc.

Re: Linux 2.6.21

[Willy Tarreau]

On Thu, Apr 26, 2007 at 06:08:06AM +0200, Adrian Bunk wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
> >...
> > So it's been over two and a half months, and while it's certainly not the 
> > longest release cycle ever, it still dragged out a bit longer than I'd 
> > have hoped for and it should have. As usual, I'd like to thank Adrian (and 
> > the people who jumped on the entries Adrian had) for keeping everybody on 
> > their toes with the regression list - there's a few entries there still, 
> > but it got to the point where we didn't even know if they were real 
> > regressions, and delaying things further just wasn't going to help.
> >...
> 
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release that were first reported in March or earlier:
> 8
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21 
> release [1]:
> 3
> 
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
> 
> 
> We have an astonishing amount of -rc testers, but obviously not the 
> developer manpower for handling them.
> 
> If we would take "no regressions" seriously, it might take 4 or 5 months 
> between releases due to the lack of developer manpower for handling 
> regressions. But that should be considered OK if avoiding regressions 
> was considered more important than getting as quick as possible to the 
> next two week regression-merge window.
> 
> But releasing with so many known regressions is insulting for the many 
> people who spent their time testing -rc kernels.

Adrian,

I understand your concerns, it's more and more common to see developers
considering their work is worthless. But it's not. You should see the
current development model as a pipeline. What you feed at the input can
take some time to reach the output, and if we wait for the whole pipeline
to flush, more crap gets released.

What is needed is a higher priority on fixes for known regressions. I
find your summary above more readable than the large lists of regressions.
I think that you should reply to Linus' announces with something that
short, starting from the known-with-patch, known-for-more-than-1-month,
and all-known-regressions. It may help Linus focus even more on those.
Also, while it will not prevent any release with regressions, at least
it will prevent such a stupid case of known regressions with patch
available.

Also, check how many regressions you have reported and which have been
fixed during the -rc stage. You'll see your work really was useful.

Maybe Linus should accept to dedicate -final to known regressions only,
to force a check in this area ? Whether or not all of them get fixed is
not the real problem, but at least we will not have any regressions with
pending patch unapplied !

Please do continue that task if you have the time to do so !

Thanks,
Willy

Re: Linux 2.6.21

[Jeff Chua]

On 4/26/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
> >...

> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
>
>
> We have an astonishing amount of -rc testers, but obviously not the
> developer manpower for handling them.

Hey, if I know enough to help, I would. But since there's so many more
talented developers, I can only contribute by testing. Without your
help, I would not have gotten STD/STR working on my notebook for
2.6.21-rcx.

We need definitely need a more robust system.

Thank you,
Jeff.

Re: Linux 2.6.21

[Daniel Barkalow]

On Thu, 26 Apr 2007, Adrian Bunk wrote:

> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14

I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found 
to be inapplicable.

> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21 
> release [1]:
> 3

The -stable team can presumably take care of these in 2.6.21.1, right? 
That leaves 10 that need developer attention.

 John Stultz seems to be taking care of 3 of them.

 Oliver Neukum has 1.

 2 are particular drivers (ali_pata and rtl8139, according to the 
 reports).

 2 seem to be ACPI-related; at least one has a candidate patch now.

 1 seems to be an ALSA problem.

 1 is STD and being debugged.

It looks like all of the known regressions are being worked on, and 
getting fixes in for them is -stable material at this point. Furthermore, 
it doesn't look to me like anyone who is needed for dealing with these 
regressions is trying to get stuff into the 2.6.22 merge window.

I think it's clear that this is the right point for Linus to start the 
2.6.22 cycle and leave the rest of the 2.6.21 work to the -stable team, 
who are the experts of taking care of this sort of stuff. Furthermore, it 
seems like -rc testers at this point have found everything in 2.6.21-rc 
they're going to, so, again, it's time for new regressions. Personally, I'd 
vote for having Linus leave off at 2.6.X-final, and have 2.6.X be the 
first -stable release of the series, where the remaining known regressions 
get fixed, but that's an issue of nomenclature, not development process.

I think you've allowed for a well-tested 2.6.21, and a good chance of a 
2.6.21.1 or .2 with no known regressions against 2.6.20, which seems to me 
like you succeeded as far as everything except making Linus a release 
engineer.

	-Daniel
*This .sig left intentionally blank*

Re: Linux 2.6.21

[Oliver Neukum]

Am Donnerstag, 26. April 2007 08:46 schrieb Daniel Barkalow:
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release:
> > 14
> 
> I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found 
> to be inapplicable.
> 
> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release with patches available at the time of the 2.6.21 
> > release [1]:
> > 3
> 
> The -stable team can presumably take care of these in 2.6.21.1, right? 
> That leaves 10 that need developer attention.
> 
>  John Stultz seems to be taking care of 3 of them.
> 
>  Oliver Neukum has 1.

Which I haven't succeeded in reproducing and gotten no further info on.

	Regards
		Oliver

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 02:46:08AM -0400, Daniel Barkalow wrote:
> 
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release:
> > 14
> 
> I count 13. (v2) had 15 items, of which 2 were subsequently fixed or found 
> to be inapplicable.

Plus one that has been reported since my last list.

> > Number of different known regressions compared to 2.6.20 at the time
> > of the 2.6.21 release with patches available at the time of the 2.6.21 
> > release [1]:
> > 3
> 
> The -stable team can presumably take care of these in 2.6.21.1, right? 

Two of them are heavily discussed patches, and I'm therefore not sure 
they will ever reach the 2.6.21 branch now that the attention has been 
shifted away from 2.6.21 regressions.

> That leaves 10 that need developer attention.
> 
>  John Stultz seems to be taking care of 3 of them.
> 
>  Oliver Neukum has 1.
> 
>  2 are particular drivers (ali_pata and rtl8139, according to the 
>  reports).
> 
>  2 seem to be ACPI-related; at least one has a candidate patch now.
> 
>  1 seems to be an ALSA problem.
> 
>  1 is STD and being debugged.

You are overinterpreting the Handled-By field in my reports.

It does not imply that this person promised to fix this issue, it only 
says that this person is or was working on this issue.

And more than 50% of the issues were reported first last month or 
earlier and are still unfixed despite repeated reminder emails - if
they weren't fixed until now they might as well become similar to
"foo seems to be broken since at about 2.6.9" issues.

It sounds highly unrealistic that all issues unfixed for a month 
suddenly become fixed even though the main focus of everyone shifts to 
2.6.21.

> It looks like all of the known regressions are being worked on, and 
> getting fixes in for them is -stable material at this point. Furthermore, 
> it doesn't look to me like anyone who is needed for dealing with these 
> regressions is trying to get stuff into the 2.6.22 merge window.
>...

That's a wrong impression, nearly every active kernel developer has at 
least one patch pending for 2.6.22.

> 	-Daniel

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Soeren Sonnenburg]

On Thu, 2007-04-26 at 06:08 +0200, Adrian Bunk wrote:
[...]
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.

Adrian, please reconsider. Without you the issues I've reported (most
likely to the wrong people) would have been missed too. And also keep in
mind that it takes 2 to tango. If reporters can nail down issues to
single config options/patches or go further by adding kprint's chances
that things get fixed increase a lot.

Soeren.
-- 
Sometimes, there's a moment as you're waking, when you become aware of
the real world around you, but you're still dreaming.

Re: Linux 2.6.21

[Jens Axboe]

On Thu, Apr 26 2007, Adrian Bunk wrote:
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.

Adding my voice to the chorus, I too hope you'll reconsider and continue
doing a great job with the regressions lists. It's really useful!

-- 
Jens Axboe

Re: Linux 2.6.21

[Jesper Juhl]

On 26/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
> >...
> > So it's been over two and a half months, and while it's certainly not the
> > longest release cycle ever, it still dragged out a bit longer than I'd
> > have hoped for and it should have. As usual, I'd like to thank Adrian (and
> > the people who jumped on the entries Adrian had) for keeping everybody on
> > their toes with the regression list - there's a few entries there still,
> > but it got to the point where we didn't even know if they were real
> > regressions, and delaying things further just wasn't going to help.
> >...
>
>
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14
>
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release that were first reported in March or earlier:
> 8
>
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21
> release [1]:
> 3
>
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
>
>
> We have an astonishing amount of -rc testers, but obviously not the
> developer manpower for handling them.
>
> If we would take "no regressions" seriously, it might take 4 or 5 months
> between releases due to the lack of developer manpower for handling
> regressions. But that should be considered OK if avoiding regressions
> was considered more important than getting as quick as possible to the
> next two week regression-merge window.
>
> But releasing with so many known regressions is insulting for the many
> people who spent their time testing -rc kernels.
>

Many people have already said this, but it needs saying again.
You are doing a great job that really helps, both with the regression
lists and the trivial patch monkey stuff. Please keep up the good
work, you are really helping the kernel.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Linux 2.6.21

[Adrian Bunk]

A clarification:

I am aware that my work had some effect, and I am aware that my work 
gets appreciated - there's no need for everyone to repeat this.

The point is: I'm not satisfied with the result.

Linus said 2.6.20 was a stable kernel. My impression was that at least 
two of the regressions from my 2.6.20 regressions list should have been 
fixed before 2.6.20.

They have both been fixed through -stable, but I also remember a quite 
experienced kernel maintainer running into one of them after 2.6.20 was 
released and spending half a day tracking it down - and my answer was
"known unfixed regression, first reported more than a month ago".

There is a conflict between Linus trying to release kernels every
2 months and releasing with few regressions.

Trying to avoid regressions might in the worst case result in an -rc12 
and 4 months between releases. If the focus is on avoiding regressions 
this has to be accepted.

And a serious delay of the next regression-merge window due to unfixed 
regressions might even have the positive side effect of more developers 
becoming interested in fixing the current regressions for getting their 
shiny new regressions^Wfeatures faster into Linus' tree.

0 regressions is never realistic (especially since many regressions 
might not be reported during -rc), but IMHO we could do much better than 
what happened in 2.6.20 and 2.6.21.

These are just my personal opinions, and other people consider the 
resulting 2.6.20 and 2.6.21 kernels OK.

I'm not satisfied with the result, and the world won't stop turning when 
I'm not tracking 2.6.22-rc regressions.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> There is a conflict between Linus trying to release kernels every
> 2 months and releasing with few regressions.

No. 

Regressions _increase_ with longer release cycles. They don't get fewer.

The fact is, we have a -stable series for a reason. The reason is that the 
normal development kernel can work in three ways:

 (a) long release cycles, with two subcases:
	(a1) huge changes (ie a "long development series". This is what we 
	     used to have. There's no way to even track the regressions, 
	     because things just change too much.
	(a2) keep the development limited, just stretch out the 
	     "stabilization phase". This simply *does*not*work*. You might 
	     want it to work, but it's against human psychology. People 
	     get bored, and start wasting their time discussing esoteric 
	     scheduler issues which weren't regressions at all.
 (b) Short and staggered release cycle: keep changes limited (like a2), 
     but recognize when it gets counter-productive, and cut a release so 
     that the stable team can continue with it, while most developers (who 
     wouldn't have worked on the stable kernel _anyway_) don't get 
     frustrated.

And yes, we've gone for (b). With occasional "I'm not taking any half-way 
scary things at _all_" releases, like 2.6.20 was.

> Trying to avoid regressions might in the worst case result in an -rc12 
> and 4 months between releases. If the focus is on avoiding regressions 
> this has to be accepted.

No. You are ignoring the reality of development. The reality is that you 
have to balance things. If you have a four-month release cycle, where 
three and a half months are just "wait for reports to trickle in from 
testers", you simply won't get _anything_ done. People will throw their 
hands up in frustration and go somewhere else.

> And a serious delay of the next regression-merge window due to unfixed 
> regressions might even have the positive side effect of more developers 
> becoming interested in fixing the current regressions for getting their 
> shiny new regressions^Wfeatures faster into Linus' tree.

No. Quite the reverse. 

If we have a problem right now

> 0 regressions is never realistic (especially since many regressions 
> might not be reported during -rc), but IMHO we could do much better than 
> what happened in 2.6.20 and 2.6.21.

2.6.20 was actually really good. Yes, it had some regressions, but I do 
believe that it was one of the least buggy releases we've had. The process 
_worked_. 

2.6.21 was much less pleasant, but the timer thing really was 

> I'm not satisfied with the result, and the world won't stop turning when 
> I'm not tracking 2.6.22-rc regressions.

True. However, it's sad that you feel like you can't bother to track them. 
They were _very_ useful. The fact that you felt they weren't is just 
becasue I think you had unrealistic expectations, and you think that the 
stable people shouldn't have to have anything to do.

You're maintaining 2.6.16 yourself - do you not see what happens when you 
decide that "zero regressions" is the target? You have to stop 
development. And while that may sound like a good thing at any particular 
time, it's a total *disaster* in the long run (not even very long, 
actually: in the two-to-three release cycle kind of run), because while 
you are in a "regression fix" mode, people still go on developing, and 
you're just causing problems for the _next_ release by holding things up 
too long.

That's the *real* reality: 5 to 7 _million_ lines of diffs in a release 
every two to three months. Do you really think those changes stop just 
because of a release process? No. If you drag out the releases to be 4+ 
months, you'll just have 10-15 million lines of changes instead (or, more 
likely, you'll have developers who can't be bothered any more, and you may 
have just 2 million lines, and three years later you have a kernel that 
isn't relevant any more. Look at any of the other Unixes).

In other words, there's a _reason_ we have staggered development. We have 
the "crazy development trees" (aka -mm and various other trees), we have 
the "development tree" (aka Linus' tree), and we have the -stable tree. If 
the stable tree has a dozen known issues that they'll have to sort out 
over the next two months, that's *fine*. That's kind of the point of the 
stable tree.

And you would helpe them with the 2.6.22-stable releases if you'd maintain 
that list. Even if it is _designed_ not to go down to zero.

I suspect that you got overly optimistic from the fact that 2.6.20 really 
_was_ an easy release. It was designed that way. You feel that it was bad 
or average, but that's actually because of _your_ unrealistic 
expectations, not becasue there was anything wrong with 2.6.20.

		Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 08:47:26AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> > 
> > There is a conflict between Linus trying to release kernels every
> > 2 months and releasing with few regressions.
> 
> No. 
> 
> Regressions _increase_ with longer release cycles. They don't get fewer.
> 
> The fact is, we have a -stable series for a reason. The reason is that the 
> normal development kernel can work in three ways:
> 
>  (a) long release cycles, with two subcases:
> 	(a1) huge changes (ie a "long development series". This is what we 
> 	     used to have. There's no way to even track the regressions, 
> 	     because things just change too much.
> 	(a2) keep the development limited, just stretch out the 
> 	     "stabilization phase". This simply *does*not*work*. You might 
> 	     want it to work, but it's against human psychology. People 
> 	     get bored, and start wasting their time discussing esoteric 
> 	     scheduler issues which weren't regressions at all.
>  (b) Short and staggered release cycle: keep changes limited (like a2), 
>      but recognize when it gets counter-productive, and cut a release so 
>      that the stable team can continue with it, while most developers (who 
>      wouldn't have worked on the stable kernel _anyway_) don't get 
>      frustrated.

<SCNR>

They get frustrated because they focussed on developing new features 
instead of fixing regressions, and now it takes longer until their new 
features get merged because noone fixed the regressions...

</SCNR>

> And yes, we've gone for (b). With occasional "I'm not taking any half-way 
> scary things at _all_" releases, like 2.6.20 was.
> 
> > Trying to avoid regressions might in the worst case result in an -rc12 
> > and 4 months between releases. If the focus is on avoiding regressions 
> > this has to be accepted.
> 
> No. You are ignoring the reality of development. The reality is that you 
> have to balance things. If you have a four-month release cycle, where 

I'm not saying it always have to be 4 months.

> three and a half months are just "wait for reports to trickle in from 
> testers", you simply won't get _anything_ done. People will throw their 
> hands up in frustration and go somewhere else.

"wait for reports to trickle in from testers" is exactly the opposite of 
our problem.

I started the regression lists originally to prove the fairy tale
"noone tests -rc kernels" some kernel developers spread as wrong.

Look at the facts:
8 out of 14 regressions in my current list were reported in March or earlier.
And for many regressions fixed it took several weeks until debugging 
by a kernel developer was started.

We do not lack testers for getting bug reports quickly.
We lack developer manpower for debugging the many regression reports.

>...
> > 0 regressions is never realistic (especially since many regressions 
> > might not be reported during -rc), but IMHO we could do much better than 
> > what happened in 2.6.20 and 2.6.21.
> 
> 2.6.20 was actually really good. Yes, it had some regressions, but I do 
> believe that it was one of the least buggy releases we've had. The process 
> _worked_. 

In the country of the blind the one-eyed man is king...

> 2.6.21 was much less pleasant, but the timer thing really was 
> 
> > I'm not satisfied with the result, and the world won't stop turning when 
> > I'm not tracking 2.6.22-rc regressions.
> 
> True. However, it's sad that you feel like you can't bother to track them. 
> They were _very_ useful. The fact that you felt they weren't is just 
> becasue I think you had unrealistic expectations, and you think that the 
> stable people shouldn't have to have anything to do.
> 
> You're maintaining 2.6.16 yourself - do you not see what happens when you 
> decide that "zero regressions" is the target? You have to stop 
> development. And while that may sound like a good thing at any particular 
> time, it's a total *disaster* in the long run (not even very long, 
> actually: in the two-to-three release cycle kind of run), because while 
> you are in a "regression fix" mode, people still go on developing, and 
> you're just causing problems for the _next_ release by holding things up 
> too long.
> 
> That's the *real* reality: 5 to 7 _million_ lines of diffs in a release 
> every two to three months. Do you really think those changes stop just 
> because of a release process? No. If you drag out the releases to be 4+ 
> months, you'll just have 10-15 million lines of changes instead (or, more 
> likely, you'll have developers who can't be bothered any more, and you may 
> have just 2 million lines, and three years later you have a kernel that 
> isn't relevant any more. Look at any of the other Unixes).

There's not a realistic chance for 0 regressions, and 4 months was 
a worst case, not the average case.

But I am not happy with the current state of released kernels.

> In other words, there's a _reason_ we have staggered development. We have 
> the "crazy development trees" (aka -mm and various other trees), we have 
> the "development tree" (aka Linus' tree), and we have the -stable tree. If 
> the stable tree has a dozen known issues that they'll have to sort out 
> over the next two months, that's *fine*. That's kind of the point of the 
> stable tree.

And all the people who have to upgrade to 2.6.21 for getting an 
important security fix run into a dozen known (and many unknown) 
regressions.

I don't think that's fine.

> And you would helpe them with the 2.6.22-stable releases if you'd maintain 
> that list. Even if it is _designed_ not to go down to zero.
> 
> I suspect that you got overly optimistic from the fact that 2.6.20 really 
> _was_ an easy release. It was designed that way. You feel that it was bad 
> or average, but that's actually because of _your_ unrealistic 
> expectations, not becasue there was anything wrong with 2.6.20.

If we had the developer manpower to get each reported regression 
debugged and fixed [1] within three weeks, 2.6.21 might be in the shape 
I would have liked it to be today.

But there are the three interdependent variables time, developer 
manpower and quality. And few developer manpower and few time results in 
a lower quality of the release I'm not happy with.

Life has taught me that sometimes I'm right, sometimes I'm wrong, and 
sometimes both sides have a possible solution. We might agree to 
disagree, and you are the one who's opinion counts. I can only say that 
I am not happy with the result, and that I do therefore not spend my 
time on maintaining regression lists for 2.6.22 - and maintaining such 
lists is not something special noone else could do equally well.

> 		Linus

cu
Adrian

[1] "fixed" can also be e.g. "patch reverted" or "not a bug"

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Adrian Bunk wrote:
> 
> They get frustrated because they focussed on developing new features 
> instead of fixing regressions, and now it takes longer until their new 
> features get merged because noone fixed the regressions...

I agree. That's part of it. But part of it is not just the "it's 2 months 
until the next release", part of it is also very much a "nothing has 
happened in the normal kernel for the last 8 weeks, this is boring, so 
I'll do my own exciting stuff".

So one _fundmanetal_ issue is that all the people who aren't directly 
involved with a particular regression are simply bored. And bored is not 
good. You want people productive - and that meas that you want a 
active development kernel that they can work with, since they aren't going 
to help with the regressions anyway.

This is why the -stable tree is so useful. It's not only that users want a 
stable tree - it allows people who do *not* have regressions on their 
plate to not be stuck twiddling their thumbs - they can be on the regular 
kernel.

> I'm not saying it always have to be 4 months.

I'm saying that four months wouldn't even have *helped* in the case of 
2.6.21.

Do you really think bugs get fixed faster just because there wasn't a 
release? Quite the reverse. Bugs get _found_ faster thanks to a release 
(simply because you tend to get more information thanks to more users), 
giving the stable people more information, causing the bugs to be able to 
be found and fixed _more_quickly_ in the stable release than if we had 
waited for four months to release 2.6.21.

The two last weeks of 2.6.21-rc were almost entirely "wasted", apart from 
getting the e1000 issue at least resolved (which was the reason for that 
delay, so I'm not complaining - I'm just saying that not a lot of people 
actually were able to _help_ with regressions during that time, and for 
some of them, we might well be better off with more information about the 
issue).

Did we fix other bugs? Yes. There was one long-time bug (since 2.6.15 or 
something) that happened to come in during that time, and we had some 
cleanups, we had MIPS bugs, we found some networking issues etc etc. But 
the amount of combined effort people put on it was pretty weak. 

> "wait for reports to trickle in from testers" is exactly the opposite of 
> our problem.

I disagree. Quite often, having 5 people report the same thing is actually 
more useful (because you see a pattern) than having one known regression 
that you don't know _why_ that regression happened. And that's the case we 
had for most of them. 

You have things like the maintainer (see Oliver's reply, for example) 
simply unable to reproduce it, and needing more information. It 
*does*not*matter* that the original report may be old. If you need more 
information, you need more information, and a two-month-old report isn't 
any better just because it's two months old.

At some point, you need to say: we're not making progress, need to release 
it, that might get us *off* this stuck situation.

That's the part you seem unable to accept. You think that "we have a 
listed regression" means that you should be able to fix it. Not so. We 
*often* need more information. 

> But I am not happy with the current state of released kernels.

So you're going to help exactly how? By stopping to help? Or kvetching 
about developers that can't figure out why something regressed.

Sure, that makes tons of sense sense, Adrian.

NOT.

		Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 10:20:46AM -0700, Linus Torvalds wrote:
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
>...
> > But I am not happy with the current state of released kernels.
> 
> So you're going to help exactly how? By stopping to help? Or kvetching 
> about developers that can't figure out why something regressed.
> 
> Sure, that makes tons of sense sense, Adrian.
> 
> NOT.

It is my time, and it's therefore my decision what I consider to make 
sense spending it for.

Instead of continuing our discussion it makes more sense that we simply 
accept that we disagree regarding when a kernel is ready for being 
released instead of repeating the same arguments in a lengthy 
discussion.

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Krzysztof Halasa]

Adrian Bunk <bunk@stusta.de> writes:

> Look at the facts:
> 8 out of 14 regressions in my current list were reported in March or earlier.
> And for many regressions fixed it took several weeks until debugging 
> by a kernel developer was started.
>
> We do not lack testers for getting bug reports quickly.
> We lack developer manpower for debugging the many regression reports.

Quite possible, given the (very) limited range of the bugs. Most people
just can't debug them.

This isn't IMHO fundamentally wrong, and releasing a ".0" kernel with
known problems isn't fundamentally wrong either.

What is missing is easily accessible KNOWN_PROBLEMS information for
released kernels. While I think your work documenting etc. known
regressions is a very good thing, publishing it with the released
kernels (certainly .0 and next stable releases, perhaps "quite stable"
rc versions as well) would be ideal.

A pressure for fixing the bugs is, obviously, the other very good thing.

>> 2.6.20 was actually really good. Yes, it had some regressions, but I do 
>> believe that it was one of the least buggy releases we've had. The process 
>> _worked_.

I the process worked with 2.6.21 as well. Obviously no two releases
are equal, one has to be better than the other.

>> > I'm not satisfied with the result, and the world won't stop turning when 
>> > I'm not tracking 2.6.22-rc regressions.

Anyway, I and many others are satisfied with the result. I think it's
one of the few "quite recent" things which are a great improvements.
Other such things are using that weird git thing :-) and perhaps the
most important - the length of devel cycle under control (I mean the
lack of "2.5 series" thing).

> But I am not happy with the current state of released kernels.

We've got stable series.
With KNOWN_PROBLEMS information, sysadmins can decide if they can
safely upgrade to .0 or if they have to wait for .123. Pressing
the responsible people to fix the problems in .123 (would) help
it greatly.
-- 
Krzysztof Halasa

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 08:37:14PM +0200, Krzysztof Halasa wrote:
>...
> With KNOWN_PROBLEMS information, sysadmins can decide if they can
> safely upgrade to .0 or if they have to wait for .123.
>...

Listing regressions like the following will most likely be zero help for 
them when deciding whether to upgrade now or later (and BTW, the latter 
might imply running a kernel with known security issues):

Subject    : acpi_pm clocksource loses time on x86-64
References : http://lkml.org/lkml/2007/4/17/143
Submitter  : Mikael Pettersson <mikpe@it.uu.se>
Handled-By : John Stultz <johnstul@us.ibm.com>
             Len Brown <lenb@kernel.org>
Status     : problem is being debugged

> Krzysztof Halasa

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Krzysztof Halasa]

Adrian Bunk <bunk@stusta.de> writes:

> Listing regressions like the following will most likely be zero help for 
> them when deciding whether to upgrade now or later (and BTW, the latter 
> might imply running a kernel with known security issues):
>
> Subject    : acpi_pm clocksource loses time on x86-64
> References : http://lkml.org/lkml/2007/4/17/143
> Submitter  : Mikael Pettersson <mikpe@it.uu.se>
> Handled-By : John Stultz <johnstul@us.ibm.com>
>              Len Brown <lenb@kernel.org>
> Status     : problem is being debugged

I don't think so, any info is better than no info.
-- 
Krzysztof Halasa

Re: Linux 2.6.21

[Mel Gorman]

On (26/04/07 20:45), Adrian Bunk didst pronounce:
> On Thu, Apr 26, 2007 at 08:37:14PM +0200, Krzysztof Halasa wrote:
> >...
> > With KNOWN_PROBLEMS information, sysadmins can decide if they can
> > safely upgrade to .0 or if they have to wait for .123.
> >...
> 
> Listing regressions like the following will most likely be zero help for 
> them when deciding whether to upgrade now or later (and BTW, the latter 
> might imply running a kernel with known security issues):
> 
> Subject    : acpi_pm clocksource loses time on x86-64
> References : http://lkml.org/lkml/2007/4/17/143
> Submitter  : Mikael Pettersson <mikpe@it.uu.se>
> Handled-By : John Stultz <johnstul@us.ibm.com>
>              Len Brown <lenb@kernel.org>
> Status     : problem is being debugged
> 

On the other hand when I was responsible for a bug, it was a great help
to see this sort of mail. Not only as it reminded I was responsible for a
screw-up but when I sent a fix out, these mails let me know it was being
picked up and ensured it made it to mainline.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: Linux 2.6.21

[Stephen Clark]

Krzysztof Halasa wrote:

>Adrian Bunk <bunk@stusta.de> writes:
>
>  
>
>>Look at the facts:
>>8 out of 14 regressions in my current list were reported in March or earlier.
>>And for many regressions fixed it took several weeks until debugging 
>>by a kernel developer was started.
>>
>>We do not lack testers for getting bug reports quickly.
>>We lack developer manpower for debugging the many regression reports.
>>    
>>
>
>Quite possible, given the (very) limited range of the bugs. Most people
>just can't debug them.
>
>This isn't IMHO fundamentally wrong, and releasing a ".0" kernel with
>known problems isn't fundamentally wrong either.
>
>What is missing is easily accessible KNOWN_PROBLEMS information for
>released kernels. While I think your work documenting etc. known
>regressions is a very good thing, publishing it with the released
>kernels (certainly .0 and next stable releases, perhaps "quite stable"
>rc versions as well) would be ideal.
>
>A pressure for fixing the bugs is, obviously, the other very good thing.
>
>  
>
>>>2.6.20 was actually really good. Yes, it had some regressions, but I do 
>>>believe that it was one of the least buggy releases we've had. The process 
>>>_worked_.
>>>      
>>>
>
>I the process worked with 2.6.21 as well. Obviously no two releases
>are equal, one has to be better than the other.
>
>  
>
>>>>I'm not satisfied with the result, and the world won't stop turning when 
>>>>I'm not tracking 2.6.22-rc regressions.
>>>>        
>>>>
>
>Anyway, I and many others are satisfied with the result. I think it's
>one of the few "quite recent" things which are a great improvements.
>Other such things are using that weird git thing :-) and perhaps the
>most important - the length of devel cycle under control (I mean the
>lack of "2.5 series" thing).
>
>  
>
>>But I am not happy with the current state of released kernels.
>>    
>>
>
>We've got stable series.
>With KNOWN_PROBLEMS information, sysadmins can decide if they can
>safely upgrade to .0 or if they have to wait for .123. Pressing
>the responsible people to fix the problems in .123 (would) help
>it greatly.
>  
>
The biggest problem I see is that developers want to make
 improvements in an area, like ide, but they don't seem to
look at the old code and make it  sure the new code supports
everything the old code did. This causes hardware that used
to work to not work, or work in a degraded fashion.

My $.02
Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Michael Tokarev]

Krzysztof Halasa wrote:
[]
> We've got stable series.
> With KNOWN_PROBLEMS information, sysadmins can decide if they can
> safely upgrade to .0 or if they have to wait for .123. Pressing
> the responsible people to fix the problems in .123 (would) help
> it greatly.

For how long you plan to maintain 2.6.x.y -stable series for each
2.6.x release?  The thing is that tehere will probably be NO
.123 "revision" (with maybe the exception of 2.6.16, thanks to
Adrian again).  The end result is that there will be just no stable
kernels *at all*. because when the next 2.6.x will start looking
more or less useful due to 2.6.x.y series, there will be new 2.6.x+1,
and work with 2.6.x stops...

It's not the case currently, but this way ("let's fix the bugs
in 2.6.x.y -stable series, don't bother releasing 2.6.x in a good
shape"), we can finally come to the above situation...

/mjt

Re: Linux 2.6.21

[Stefan Richter]

Michael Tokarev wrote:
[...]
> there will be just no stable
> kernels *at all*. because when the next 2.6.x will start looking
> more or less useful due to 2.6.x.y series, there will be new 2.6.x+1,
> and work with 2.6.x stops...
> 
> It's not the case currently, but this way ("let's fix the bugs
> in 2.6.x.y -stable series, don't bother releasing 2.6.x in a good
> shape"), we can finally come to the above situation...

Actually, bugs are fixed in --> 2.6.(x+1)-rc <--, and shortly thereafter
a small selection of the fixes are "backported" to 2.6.x.y and to
various distributor kernels.
-- 
Stefan Richter
-=====-=-=== -=-- ==-==
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Krzysztof Halasa]

Michael Tokarev <mjt@tls.msk.ru> writes:

> For how long you plan to maintain 2.6.x.y -stable series for each
> 2.6.x release?  The thing is that tehere will probably be NO
> .123 "revision"

Actually I meant .1, .2 and maybe even .3 :-)
-- 
Krzysztof Halasa

Re: Linux 2.6.21

[Alan Cox]

> They get frustrated because they focussed on developing new features 
> instead of fixing regressions, and now it takes longer until their new 
> features get merged because noone fixed the regressions...

I would disagree: They get frustrated because they are blocked on some
small regression which is stopping a ton of other fixed including
features people need (like new hardware support) from being released.

The "no regressions" model doesn't really work when you ask about the
greater good of the userbase.

The goal of no regressions is great and the regression lists for ATA were
certainly very helpful but the greater good comes first.

Alan

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 09:50:15PM +0100, Alan Cox wrote:
> > They get frustrated because they focussed on developing new features 
> > instead of fixing regressions, and now it takes longer until their new 
> > features get merged because noone fixed the regressions...
> 
> I would disagree: They get frustrated because they are blocked on some
> small regression which is stopping a ton of other fixed including
> features people need (like new hardware support) from being released.
> 
> The "no regressions" model doesn't really work when you ask about the
> greater good of the userbase.
> 
> The goal of no regressions is great and the regression lists for ATA were
> certainly very helpful but the greater good comes first.

"no regressions" is definitely not feasible.

14 known regressions, some of them not yet debugged at all, are 
different from your "some small regression".

And look e.g. at the many (and non-trivial) changes between -rc7 and 
-final, resulting in more than one report from people who were running 
-rc7 without problems - and 2.6.21 doesn't work for them.

It's not a choice between "regressions don't matter" and "no regressions",
it's about the place in the area between these two extremes. I have my 
opinions on what I want to expect from a stable Linux kernel, and other 
people have different opinins.

> Alan

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Theodore Tso]

On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
> "no regressions" is definitely not feasible.
> 
> 14 known regressions, some of them not yet debugged at all, are 
> different from your "some small regression".

Yes, but when were some of these regressions reported?  Past a certain
point, I think it's reasonable to look at the regression, decide how
many people would be affected by it, and why it hadn't been noticed
earlier, and in some cases, decide that it's better to get this
debugged and fixed in the stable and development trees in parallel.

> And look e.g. at the many (and non-trivial) changes between -rc7 and 
> -final, resulting in more than one report from people who were running 
> -rc7 without problems - and 2.6.21 doesn't work for them.

I agree that's unfortunate.

> It's not a choice between "regressions don't matter" and "no regressions",
> it's about the place in the area between these two extremes. I have my 
> opinions on what I want to expect from a stable Linux kernel, and other 
> people have different opinins.

Everyone is going to disagree to some extent; and their own comfort
zone.  So a certain amount compromise is always going to be necessary.
Of course, it's up to you decide whether this has gone beyond the zone
where you aren't comfortable working with other people's development
style.

Regards,

					- Ted

Re: Linux 2.6.21

[Adrian Bunk]

On Fri, Apr 27, 2007 at 12:31:43PM -0400, Theodore Tso wrote:
> On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
> > "no regressions" is definitely not feasible.
> > 
> > 14 known regressions, some of them not yet debugged at all, are 
> > different from your "some small regression".
> 
> Yes, but when were some of these regressions reported?  Past a certain
> point, I think it's reasonable to look at the regression, decide how
> many people would be affected by it, and why it hadn't been noticed
> earlier, and in some cases, decide that it's better to get this
> debugged and fixed in the stable and development trees in parallel.

8 of them have been reported in March or earlier. [1]

Patches for 2 of these 8 were available at the time of the release. [2]
While the question whether to merge one of them into 2.6.21 was 
controversial, the other one was not controversial.

For one of the bugs, it became obvious when someone looked at it after 
the release of 2.6.21 that between the bug report on March 31th and the 
release of 2.6.21 on April 21th, noone started debugging this bug. [3] [4]

> > And look e.g. at the many (and non-trivial) changes between -rc7 and 
> > -final, resulting in more than one report from people who were running 
> > -rc7 without problems - and 2.6.21 doesn't work for them.
> 
> I agree that's unfortunate.
> 
> > It's not a choice between "regressions don't matter" and "no regressions",
> > it's about the place in the area between these two extremes. I have my 
> > opinions on what I want to expect from a stable Linux kernel, and other 
> > people have different opinins.
> 
> Everyone is going to disagree to some extent; and their own comfort
> zone.  So a certain amount compromise is always going to be necessary.
> Of course, it's up to you decide whether this has gone beyond the zone
> where you aren't comfortable working with other people's development
> style.
> 
> Regards,
> 
> 					- Ted

cu
Adrian

[1] http://lkml.org/lkml/2007/4/26/2
[2] http://lkml.org/lkml/2007/4/25/496
[3] http://lkml.org/lkml/2007/4/26/496
[4] and although it turned out this specific regression was already 
    fixed in 2.6.21, I hope you get my point

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Stephen Clark]

Adrian Bunk wrote:

>On Fri, Apr 27, 2007 at 12:31:43PM -0400, Theodore Tso wrote:
>  
>
>>On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
>>    
>>
>>>"no regressions" is definitely not feasible.
>>>
>>>14 known regressions, some of them not yet debugged at all, are 
>>>different from your "some small regression".
>>>      
>>>
>>Yes, but when were some of these regressions reported?  Past a certain
>>point, I think it's reasonable to look at the regression, decide how
>>many people would be affected by it, and why it hadn't been noticed
>>earlier, and in some cases, decide that it's better to get this
>>debugged and fixed in the stable and development trees in parallel.
>>    
>>
>
>8 of them have been reported in March or earlier. [1]
>
>Patches for 2 of these 8 were available at the time of the release. [2]
>While the question whether to merge one of them into 2.6.21 was 
>controversial, the other one was not controversial.
>
>For one of the bugs, it became obvious when someone looked at it after 
>the release of 2.6.21 that between the bug report on March 31th and the 
>release of 2.6.21 on April 21th, noone started debugging this bug. [3] [4]
>
>  
>
>>>And look e.g. at the many (and non-trivial) changes between -rc7 and 
>>>-final, resulting in more than one report from people who were running 
>>>-rc7 without problems - and 2.6.21 doesn't work for them.
>>>      
>>>
>>I agree that's unfortunate.
>>
>>    
>>
>>>It's not a choice between "regressions don't matter" and "no regressions",
>>>it's about the place in the area between these two extremes. I have my 
>>>opinions on what I want to expect from a stable Linux kernel, and other 
>>>people have different opinins.
>>>      
>>>
>>Everyone is going to disagree to some extent; and their own comfort
>>zone.  So a certain amount compromise is always going to be necessary.
>>Of course, it's up to you decide whether this has gone beyond the zone
>>where you aren't comfortable working with other people's development
>>style.
>>
>>Regards,
>>
>>					- Ted
>>    
>>
>
>cu
>Adrian
>
>[1] http://lkml.org/lkml/2007/4/26/2
>[2] http://lkml.org/lkml/2007/4/25/496
>[3] http://lkml.org/lkml/2007/4/26/496
>[4] and although it turned out this specific regression was already 
>    fixed in 2.6.21, I hope you get my point
>
>  
>
What Adrian was doing, or anybody in the future, is not going to be 
productive unless
Linus holds the people who are responsible for the bug to get it fixed 
or report why they
can't.

My $.02
Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Markus Rechberger]

On 4/27/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Fri, Apr 27, 2007 at 12:31:43PM -0400, Theodore Tso wrote:
> > On Fri, Apr 27, 2007 at 04:58:05PM +0200, Adrian Bunk wrote:
> > > "no regressions" is definitely not feasible.
> > >
> > > 14 known regressions, some of them not yet debugged at all, are
> > > different from your "some small regression".
> >
> > Yes, but when were some of these regressions reported?  Past a certain
> > point, I think it's reasonable to look at the regression, decide how
> > many people would be affected by it, and why it hadn't been noticed
> > earlier, and in some cases, decide that it's better to get this
> > debugged and fixed in the stable and development trees in parallel.
>
> 8 of them have been reported in March or earlier. [1]
>
> Patches for 2 of these 8 were available at the time of the release. [2]
> While the question whether to merge one of them into 2.6.21 was
> controversial, the other one was not controversial.
>

I can imagine that the dvb oops bugfix got held back to avoid some
noise with dvb developers who claimed that they didn't get notified
about how that patch got into the v4l-dvb repository (it didn't get
reviewed by these people for weeks because it simply got ignored and
some of them were aware of that)

On the other side if I read bugreports like the following one:
http://www.mail-archive.com/linux-dvb%40linuxtv.org/msg23028.html

(My Nova-T 500 crashes quite regularly. My machine has been running for
about a week and in that time has had 5 oopses.)

It doesn't solve the Nova-T disconnects, but it at least solves that
the machine doesn't go down when this happens till the driver gets
fixed.
I think it would have been nice to get that patch into it

Markus

> For one of the bugs, it became obvious when someone looked at it after
> the release of 2.6.21 that between the bug report on March 31th and the
> release of 2.6.21 on April 21th, noone started debugging this bug. [3] [4]
>
> > > And look e.g. at the many (and non-trivial) changes between -rc7 and
> > > -final, resulting in more than one report from people who were running
> > > -rc7 without problems - and 2.6.21 doesn't work for them.
> >
> > I agree that's unfortunate.
> >
> > > It's not a choice between "regressions don't matter" and "no
> regressions",
> > > it's about the place in the area between these two extremes. I have my
> > > opinions on what I want to expect from a stable Linux kernel, and other
> > > people have different opinins.
> >
> > Everyone is going to disagree to some extent; and their own comfort
> > zone.  So a certain amount compromise is always going to be necessary.
> > Of course, it's up to you decide whether this has gone beyond the zone
> > where you aren't comfortable working with other people's development
> > style.
> >
> > Regards,
> >
> > 					- Ted
>
> cu
> Adrian
>
> [1] http://lkml.org/lkml/2007/4/26/2
> [2] http://lkml.org/lkml/2007/4/25/496
> [3] http://lkml.org/lkml/2007/4/26/496
> [4] and although it turned out this specific regression was already
>     fixed in 2.6.21, I hope you get my point
>
> --
>
>        "Is there not promise of rain?" Ling Tan asked suddenly out
>         of the darkness. There had been need of rain for many days.
>        "Only a promise," Lao Er said.
>                                        Pearl S. Buck - Dragon Seed
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


-- 
Markus Rechberger

Re: Linux 2.6.21

[Bill Davidsen]

Adrian Bunk wrote:

> Life has taught me that sometimes I'm right, sometimes I'm wrong, and 
> sometimes both sides have a possible solution. We might agree to 
> disagree, and you are the one who's opinion counts. I can only say that 
> I am not happy with the result, and that I do therefore not spend my 
> time on maintaining regression lists for 2.6.22 - and maintaining such 
> lists is not something special noone else could do equally well.
>
And the next kernel will go out with no list to warn users, and no to-do 
list for -stable.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Chuck Ebbert]

Linus Torvalds wrote:
> 
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
>> There is a conflict between Linus trying to release kernels every
>> 2 months and releasing with few regressions.
> 
> No. 
> 
> Regressions _increase_ with longer release cycles. They don't get fewer.
> 
> The fact is, we have a -stable series for a reason.

Problem is, not enough developers pay attention to the -stable
series. Adrian, maybe you could shift your attention there and
stop trying to track the bleeding edge?

Re: Linux 2.6.21

[Diego Calleja]

El Thu, 26 Apr 2007 13:02:28 -0400, Chuck Ebbert <cebbert@redhat.com> escribió:

> Problem is, not enough developers pay attention to the -stable
> series. Adrian, maybe you could shift your attention there and
> stop trying to track the bleeding edge?

>From my humble POV, it's a problem that all this discussion was generated
on what Adrian does or stop doing. Apparently, unless Adrian posts his
list of know regressions, most of the people doesn't look at the bugzilla
at all. Maybe it'd be useful to create a per-release bug tracker in the
bugzilla or collect them into one of the a kernel.org's wiki, to make easier
to follow the current state of all the "important" regressions. 

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Diego Calleja wrote:
> 
> From my humble POV, it's a problem that all this discussion was generated
> on what Adrian does or stop doing. Apparently, unless Adrian posts his
> list of know regressions, most of the people doesn't look at the bugzilla
> at all. Maybe it'd be useful to create a per-release bug tracker in the
> bugzilla or collect them into one of the a kernel.org's wiki, to make easier
> to follow the current state of all the "important" regressions. 

Any web-based interface is a no-no. It's one reason I don't use bugzilla a 
lot. If I can't get it by email, it doesn't exist, as far as I'm 
concerned.

I bet that's true even of a lot of people who are more "web oriented" than 
I am. They may look at webpages, but getting notified by email is still 
the wakeup call. There's a difference between "active and directed pushing 
to the involved people" and "the resource exists, that people could look 
at".

So it would have to be more than just a wiki or a bugzilla entry. It would 
have to have that weekly email status thing, and I think that it needs to 
have some human who tries to find messages on the kernel mailing list too, 
and make a first-level judgement on the bugs. Adrian was doing a good job.

But it doesn't necessarily need somebody with intimate knowledge of the 
kernel. In fact, almost everybody who *does* have intimate knowledge tends 
to have so in a very specific area (nobody knows everything - and that 
very much includes people like me and Andrew too) and maybe be skewed in 
other ways too, so a "generalist" is probably more useful than somebody 
who is a "deep coder" in some subsystem.

And it almost certainly doesn't have/prefer to be _one_ person. I suspect 
that this is something where it actually might be better to have some 
collection of people interested in it, and yes, perhaps editing a wiki is 
part of the process, but with at least that "automated email" thing going 
on in additin (and it needs to go to the people involved, not just the 
kernel mailing list - so part of it is not just gathering the reports 
themselves, but also gathering target addresses from MAINTAINERS files and 
perhaps git logs etc).

And yes, it's quite possibly a good way to get into kernel development - 
it definitely helps to know about programming, but as mentioned, I don't 
think it is something where you even need to know specifically about 
*kernel* programming per se.

For example, I don't think it was an accident that Adrian (who has been 
involved in kernelnewbies, janitors and the trivial tree) was the one who 
picked it up. That's exactly the kind of involvement that the regression 
tracking is all about!

(In fact, I think regression tracking might be "easier" to get into than 
actually getting into some of the janitorial projects, exactly because 
it's less about coding. The fact that a person who tracks regressions 
might then *also* indirectly get into the code itself would just be a big 
additional bonus!)

So yes, some automation can almost certainly help (especially if there are 
multiple people involved), but I think a lot of it is that "human 
judgement" and ability to group things and communicate. Are there any 
kernel janitors/newbies/mentors that can think of people who would want to 
do something like this?

			Linus

Re: Linux 2.6.21

[Diego Calleja]

El Thu, 26 Apr 2007 11:42:22 -0700 (PDT), Linus Torvalds <torvalds@linux-foundation.org> escribió:

> I bet that's true even of a lot of people who are more "web oriented" than 
> I am. They may look at webpages, but getting notified by email is still 
> the wakeup call. There's a difference between "active and directed pushing 

Bugzilla sucks quite a lot at email, but you can answer emails and they get
into the bugzilla database; and there're two mailing lists (listed in
Documentation/HOWTO) that send notifications about every new bug
added/modified- I know it's not the perfect email interface every hacker
wants, but it's better than nothing.

I suggested some time ago that it'd be useful to send every new bug
notification from bugme-new to the LKML (and/or other lists). The
volume should not be so high to make it so annoying that it makes it
unuseful, and at least it makes the bugzilla-haters aware of the bugs
reported, and since bugzilla tracks the answers to emails and the
reporter email address is in the email, it makes easier for bugzilla-haters
to ask for more data and try to fix the problem, without starting
any browser.

I can understand Adrian's resign. Bugzilla is crap, but there're users
reporting bugs there and willing to cooperate to fix them, and they're
not getting listened. There're even a few description of patches (ie: "line
6 in foo.c is wrong and it breaks our testing, it should read like this:")
that have been sitting there for *years* and not getting merged. I guess
that Adrian tried to canalize the important regressions to the hackers,
and he got tired of apparently being the only one that cares about getting
them fixed.

So I, or anyone else, could try to do Adrian's job. But if Adrian (a guy
that sends patches to make global functions static 8) got tired
of doing that job, I suspect that I, or anyone else would also got 
tired of it even sooner. There're other big projects with probably
more bug reports than linux, they don't work this way, and they
look more succesful in their bug handling.

So in my humble opinion there's a problem, about how the whole
bug reporting/fixing process works. With the current linux
development model, a good bug reporting/fixing process doesn't
looks optional, since it's important to fix bugs ASAP to get the
fixes into -stable. The fix may go as further as "writing our own
bug tracking software" in the same way git fixed other
development issues, or it may be a human issue, or a mix of the
two.

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Diego Calleja wrote:
> 
> Bugzilla sucks quite a lot at email, but you can answer emails and they get
> into the bugzilla database; and there're two mailing lists (listed in
> Documentation/HOWTO) that send notifications about every new bug
> added/modified- I know it's not the perfect email interface every hacker
> wants, but it's better than nothing.

No, it's *not* better than nothing.

The thing is, these reports MUST NOT go to "everybody". If they do, that 
is actually *worse* than nothing, because people will just ignore them 
entirely, since they aren't "directed".

The emails need to be directed to the appropriate parties, not go to 
everybody. There is nobody who is interested in seeing all regressions, 
except perhaps me and Andrew. Most *real* developers (as opposed to people 
like me, who are integrators, not "real developers") want to be notified 
about problems in *their* area, and if it's just automation that sends out 
everything, it just dilutes the value of the thing, to the point where 
people will ignore it even for the cases when they happen to be related to 
what they do.

> I suggested some time ago that it'd be useful to send every new bug
> notification from bugme-new to the LKML (and/or other lists).

I don't know a lot of developers who actually read LKML. I know a lot of 
people who look for interesting subject lines and interesting people, but 
read LKML in the sense of reading everything? Not likely.

That's why I think Adrian did a great job: he took the "noise" and made it 
somethng worth looking at! And part of that is very much to make it 
directred to only relevant parties (yes, they *also* got cc'd to 
linux-kernel, but people would get them in their personal mailboxes and 
*not* feel like it was just noise that didn't matter to them!)

> I can understand Adrian's resign. Bugzilla is crap, but there're users
> reporting bugs there and willing to cooperate to fix them, and they're
> not getting listened.

I personally refuse to have anything at all with bugzilla. The interface 
is so horrible that it's just not worth my time. I know there are a few 
people who use it productively, but I'm always amazed that they can do 
that.

The *big* problem with bugzilla is that it's such a "detail-oriented" 
thing. It's fine if you have *one* bug that you're tracking. But whenever 
that's not the case, it's almost totally useless.

Let me put it another way: I would never use a source control system that 
forces me to look at my 22,000 files one at a time. I think such a system 
is fundamentally broken, because it makes it impossible to get the big 
picture ("what changed in the last week" kind of thing). The same is true 
of bugzilla: if you *know* which bug you're looking at, it's good. For 
anythign else, it's almost worse than useless, exactly because there is no 
way to get an overview.

> There're even a few description of patches (ie: "line
> 6 in foo.c is wrong and it breaks our testing, it should read like this:")
> that have been sitting there for *years* and not getting merged.

.. and you claim that this shows that developers don't listen. I'd say it 
shows the exact *opposite*: the users don't listen. There's a lot mroe 
users than developers, and bugzilla is pretty much designed to let the 
users "report and forget", which is exactly the *wrong* thing to do, 
because it puts the onus on the developer.

(I've said this before, but I'll say it again: one thing that would 
already make bugzilla better is to just always drop any bug reports that 
are more than a week old and haven't been touched. It wouldn't need *much* 
touching, but if a reporter cannot be bothered to say "still true with 
current snapshot" once a week, then it shouldn't be seen as being somehow 
up to those scare resources we call "developers" to have to go through 
it).

So there are probably things that bugzilla could do to become more useful, 
but I don't see that happening. We'd need either a smarter/better 
bugzilla, or somebody who actually turns noise into real information. 
Adrian did that (although in fairness to others, other people definitely 
do it too. Dave Jones, for example. Very useful).

> So I, or anyone else, could try to do Adrian's job. But if Adrian (a guy
> that sends patches to make global functions static 8) got tired
> of doing that job, I suspect that I, or anyone else would also got 
> tired of it even sooner.

I do agree - one of the problems with the job is not that it's thankless 
(I think we've had at least ten kernel developers, very much including me, 
talking about how _useful_ it is), but there is definitely a lack of 
glamour and probably interest.

I think it could be more interesting if part of the job was doing the 
tools. Tools *are* important. Most of my actual _development_ for the last 
couple of years has been on "git", not the kernel, but I think I was more 
productive that way, so I don't think that's wasted time at all.

So yes, automation would be a good idea, but I don't think bugzilla is it. 

> There're other big projects with probably more bug reports than linux, 
> they don't work this way, and they look more succesful in their bug 
> handling.

Well, one thing to keep in mind is that the kernel really does have a 
*lot* more development going on that most other projects.

I don't think you'll find another project that has about six megabytes of 
diffs every release (every two months). That's really one of the 
fundamental issues - things really *happen* in the kernel. A *lot* of 
things. You can't take a breather - I can do "stabilization releases" 
every once in a while, and Andrew can kick out patches he decides aren't 
ready to be merged rather than maintain them in his tree (and he does do 
that), but the kernel simply tends to have a different *scale* than other 
projects.

And almost all hard bugs are about hardware interactions. Drivers. Big 
iron. Things like that - ie unlike something like a compiler, you can 
seldom say "this test-case crashes". Yes, that does happen for the kernel 
too, but those are the *easy* bugs. Those generally get fixed in a day or 
two.

So I really don't think you can compare to "other projects". They simply 
don't have these issues.

		Limis

Re: Linux 2.6.21

[Marek Wawrzyczny]

On Fri, 27 Apr 2007 07:13:08 Linus Torvalds wrote:
> I think it could be more interesting if part of the job was doing the
> tools. Tools *are* important. Most of my actual _development_ for the last
> couple of years has been on "git", not the kernel, but I think I was more
> productive that way, so I don't think that's wasted time at all.
>
> So yes, automation would be a good idea, but I don't think bugzilla is it.

<...>

> So I really don't think you can compare to "other projects". They simply
> don't have these issues.

Seems like the bug tracking system needs to be re-evaluated. Perhaps Bugzilla 
can be modified to better serve the needs of kernel developers.

There might be alternatives, JIRA (http://www.atlassian.com/software/jira/) 
for example. I am sure there are others.

Finally, I have over 7 years experience writing various web based systems 
(using WebObjects, J2EE and others). I would be quite willing to volunteer my 
spare time and experience if this community decides that a custom written 
solution (one that would handle email bug submissions/resolutions :) is in 
order.

Kind regards,

Marek Wawrzyczny

Re: Linux 2.6.21

[Martin J. Bligh]

> The thing is, these reports MUST NOT go to "everybody". If they do, that 
> is actually *worse* than nothing, because people will just ignore them 
> entirely, since they aren't "directed".
> 
> The emails need to be directed to the appropriate parties, not go to 
> everybody. There is nobody who is interested in seeing all regressions, 
> except perhaps me and Andrew. Most *real* developers (as opposed to people 
> like me, who are integrators, not "real developers") want to be notified 
> about problems in *their* area, and if it's just automation that sends out 
> everything, it just dilutes the value of the thing, to the point where 
> people will ignore it even for the cases when they happen to be related to 
> what they do.

It's easy to send the different categories to different mailing lists,
if that's what we want to do. Apart from some aggressive filtering on
the SCSI lists etc stops me from bouncing messages to it, but that's
fixable.

Yes, human involvement from someone with half a brain would be better.
Andrew does a lot of that. Not a particularly good use of talent really.
but still.

As Andrew has pointed out before though - even though he forwards
the bugs, nobody does anything with it. The sad truth seems to be
that people have very little interest in fixing bugs when they are
reported - it's not sexy, I guess.

> Let me put it another way: I would never use a source control system that 
> forces me to look at my 22,000 files one at a time. I think such a system 
> is fundamentally broken, because it makes it impossible to get the big 
> picture ("what changed in the last week" kind of thing). The same is true 
> of bugzilla: if you *know* which bug you're looking at, it's good. For 
> anythign else, it's almost worse than useless, exactly because there is no 
> way to get an overview

Go to http://bugzilla.kernel.org. Hit query. Find the box that says
"Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.

74 bugs found.

Not hard to do.

> (I've said this before, but I'll say it again: one thing that would 
> already make bugzilla better is to just always drop any bug reports that 
> are more than a week old and haven't been touched. It wouldn't need *much* 
> touching, but if a reporter cannot be bothered to say "still true with 
> current snapshot" once a week, then it shouldn't be seen as being somehow 
> up to those scare resources we call "developers" to have to go through 
> it).

I'm reluctant to drop / close them. We could fairly easily move them to
a "STALE" state if you want, and have that ping the user. Not sure what
we'd ping them with apart from "Nobody seems to give a toss about your
bug. Life's a bitch. Try sending chocolates, flowers, or fireworks".
I'm still unconvinced the users or the tool are the problem, but if it
makes you happier, we can do that.

> So there are probably things that bugzilla could do to become more useful, 
> but I don't see that happening. We'd need either a smarter/better 
> bugzilla, or somebody who actually turns noise into real information. 
> Adrian did that (although in fairness to others, other people definitely 
> do it too. Dave Jones, for example. Very useful).

What would you want from a smarter / better bugzilla or other bug
tracking tool? A list of requirements / suggestions would be nice. The
main complaint we had before was lack of an email interface, and that
was fixed a long time ago. I admit development has not exactly been
active since, but the only person I got real feedback from was Dave J,
and we've been fixing his UI issues.

M.

Re: Linux 2.6.21

[Neil Brown]

On Saturday April 28, mbligh@mbligh.org wrote:
> 
> Yes, human involvement from someone with half a brain would be better.
> Andrew does a lot of that. Not a particularly good use of talent really.
> but still.

I think more than half a brain is needed to do this well.  You need a
reasonable understanding of how all the bits of the kernel work
together so that you have a good chance of sending the bug in the right
direction.  You need a good understanding of the kernel community and
various sub communities so that you know who might be both able and
willing to deal with the bug.  And it wouldn't hurt to have a good
over-view of the current 'hot' areas of the kernel so you know if it
is really worth suggesting "try with the latest -mm" or not.
And you need good people skills.

So I think you really need a lot of up-to-date knowledge to do this
well.  Because of Andrew's position as a funnel, he has a lot of that
knowledge.  It would be really nice if he had some help though.  And I
really think that would mean finding someone in the community who
would rather be coding (and currently are) and convincing them that
there is a higher calling for them.  Finding someone out side or on
the edge of the community is less likely to be effective.

> 
> As Andrew has pointed out before though - even though he forwards
> the bugs, nobody does anything with it. The sad truth seems to be
> that people have very little interest in fixing bugs when they are
> reported - it's not sexy, I guess.

Not sexy, and also not at all easy.   A lot of the interesting bugs
seem to be subtle interactions between separate parts of the kernel -
one part making an assumption or exhibiting a behaviour that the other
part didn't expect.  And we all know that writing bug^Wcode is easier
than removing bugs.  I can spend hour and hours reading through code
trying to get the big picture, and end up finding a one-line change
that then needs documenting, testing and external review.  It's not
easy.

> I'm still unconvinced the users or the tool are the problem, but if it
> makes you happier, we can do that.

No, they aren't the problem.  Bugs are the problem.  But they might be
a more effective part of the solution.

My perception of the kernel bugzilla is that visibility is very low.

I think there is value in weekly reminders, and I wouldn't mind seeing
a weekly Email on linux-kernel with something like a list of open bugs
that have not seen any activity in between 1 and 2 weeks.  It might
get someone out-of-area interested, and might be noticed by someone
who thinks they are in-area and get them wondering why they didn't
find out when the bug was first reported.

NeilBrown

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 08:11:30AM +1000, Neil Brown wrote:
> On Saturday April 28, mbligh@mbligh.org wrote:
>...
> > As Andrew has pointed out before though - even though he forwards
> > the bugs, nobody does anything with it. The sad truth seems to be
> > that people have very little interest in fixing bugs when they are
> > reported - it's not sexy, I guess.
> 
> Not sexy, and also not at all easy.   A lot of the interesting bugs
> seem to be subtle interactions between separate parts of the kernel -
> one part making an assumption or exhibiting a behaviour that the other
> part didn't expect.  And we all know that writing bug^Wcode is easier
> than removing bugs.  I can spend hour and hours reading through code
> trying to get the big picture, and end up finding a one-line change
> that then needs documenting, testing and external review.  It's not
> easy.
> 
> > I'm still unconvinced the users or the tool are the problem, but if it
> > makes you happier, we can do that.
> 
> No, they aren't the problem.  Bugs are the problem.  But they might be
> a more effective part of the solution.
> 
> My perception of the kernel bugzilla is that visibility is very low.
> 
> I think there is value in weekly reminders, and I wouldn't mind seeing
> a weekly Email on linux-kernel with something like a list of open bugs
> that have not seen any activity in between 1 and 2 weeks.  It might
> get someone out-of-area interested, and might be noticed by someone
> who thinks they are in-area and get them wondering why they didn't
> find out when the bug was first reported.

The 100 kB email limit has to be lifted for this... 

More seriously, there are > 1000 open bugs in the kernel Bugzilla 
without any activity during the last 2 weeks.

The problem is usually either "Not sexy, and also not at all easy." or
"no maintainer".

Technology can assist, but there are non-technical problems you can't 
solve through technology.

> NeilBrown

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Neil Brown]

On Sunday April 29, bunk@stusta.de wrote:
> On Sun, Apr 29, 2007 at 08:11:30AM +1000, Neil Brown wrote:
> > 
> > I think there is value in weekly reminders, and I wouldn't mind seeing
> > a weekly Email on linux-kernel with something like a list of open bugs
> > that have not seen any activity in between 1 and 2 weeks.  It might
> > get someone out-of-area interested, and might be noticed by someone
> > who thinks they are in-area and get them wondering why they didn't
> > find out when the bug was first reported.
> 
> The 100 kB email limit has to be lifted for this... 
> 
> More seriously, there are > 1000 open bugs in the kernel Bugzilla 
> without any activity during the last 2 weeks.

I meant "between 1 and 2 weeks"  - so each bug hits linux-kernel once,
any only if no-one seem to care.
If I ask for bugs changed in the last 7 days I get 80
If I ask for bugs changed in the last 14 days I get 114.
So today, that list would have (114-80) = 34 bug, which I think it is
manageable to glance through and think "Would I like to care".

NeilBrown

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 00:33, Adrian Bunk wrote:
> On Sun, Apr 29, 2007 at 08:11:30AM +1000, Neil Brown wrote:
> > On Saturday April 28, mbligh@mbligh.org wrote:
> >...
> > > As Andrew has pointed out before though - even though he forwards
> > > the bugs, nobody does anything with it. The sad truth seems to be
> > > that people have very little interest in fixing bugs when they are
> > > reported - it's not sexy, I guess.
> > 
> > Not sexy, and also not at all easy.   A lot of the interesting bugs
> > seem to be subtle interactions between separate parts of the kernel -
> > one part making an assumption or exhibiting a behaviour that the other
> > part didn't expect.  And we all know that writing bug^Wcode is easier
> > than removing bugs.  I can spend hour and hours reading through code
> > trying to get the big picture, and end up finding a one-line change
> > that then needs documenting, testing and external review.  It's not
> > easy.
> > 
> > > I'm still unconvinced the users or the tool are the problem, but if it
> > > makes you happier, we can do that.
> > 
> > No, they aren't the problem.  Bugs are the problem.  But they might be
> > a more effective part of the solution.
> > 
> > My perception of the kernel bugzilla is that visibility is very low.
> > 
> > I think there is value in weekly reminders, and I wouldn't mind seeing
> > a weekly Email on linux-kernel with something like a list of open bugs
> > that have not seen any activity in between 1 and 2 weeks.  It might
> > get someone out-of-area interested, and might be noticed by someone
> > who thinks they are in-area and get them wondering why they didn't
> > find out when the bug was first reported.
> 
> The 100 kB email limit has to be lifted for this... 
> 
> More seriously, there are > 1000 open bugs in the kernel Bugzilla 
> without any activity during the last 2 weeks.
> 
> The problem is usually either "Not sexy, and also not at all easy." or
> "no maintainer".

Please add "limited human resources" to this list.

Greetings,
Rafael

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Neil Brown wrote:
> 
> I think more than half a brain is needed to do this well.  You need a
> reasonable understanding of how all the bits of the kernel work
> together so that you have a good chance of sending the bug in the right
> direction.

Yes. However, even if you just end up summarizing the current outstanding 
issues (and are not able to necessarily point to the right person), I 
suspect that all the top-level maintainers would be interested in it. As 
long as it's a "summary report twice a week" kind of thing.

In fact, I suspect a lot of non-maintainers would be interested too!

And if you then have some way for people to add commentary (and directing) 
to entries, you might start out not knowing where some bugreport should 
go, but you'll have people able to forward them.

I end up doing that fairly regularly - adding people to the Cc when I'm 
involved in a bug, and we suddenly notice that "hey, it looks like it's a 
timer" issue or something, and we end up adding Ingo and Thomas Gleixner 
to the cc.

This, btw, is why email is *so* much nicer than a web interface. A web 
interface is always "a single person looking at it". An email, on the 
other hand, is always "you can bounce it to make _another_ person look at 
it".

But I agree that the more involved in the kernel the initiating person has 
been (at least to *some* degree), the better. No question. I just don't 
think it necessarily needs to be a hugely core person. Anybody who has 
been reading lkml for the last few months will already know a lot of the 
people involved in different subsystems!

			Linus

Re: Linux 2.6.21

[Andrew Morton]

On Sun, 29 Apr 2007 08:11:30 +1000 Neil Brown <neilb@suse.de> wrote:

> On Saturday April 28, mbligh@mbligh.org wrote:
> > 
> > Yes, human involvement from someone with half a brain would be better.
> > Andrew does a lot of that. Not a particularly good use of talent really.
> > but still.
> 
> I think more than half a brain is needed to do this well.  You need a
> reasonable understanding of how all the bits of the kernel work
> together so that you have a good chance of sending the bug in the right
> direction.  You need a good understanding of the kernel community and
> various sub communities so that you know who might be both able and
> willing to deal with the bug.  And it wouldn't hurt to have a good
> over-view of the current 'hot' areas of the kernel so you know if it
> is really worth suggesting "try with the latest -mm" or not.
> And you need good people skills.
> 
> So I think you really need a lot of up-to-date knowledge to do this
> well.  Because of Andrew's position as a funnel, he has a lot of that
> knowledge.

yup

>  It would be really nice if he had some help though.

Amen, Brother Neil.

>  And I
> really think that would mean finding someone in the community who
> would rather be coding (and currently are) and convincing them that
> there is a higher calling for them.  Finding someone out side or on
> the edge of the community is less likely to be effective.

	http://www.google.com/support/jobs/bin/answer.py?answer=53317

This is a fully funded regular full-time position @google's Mountain View
HQ, sitting within nagging range of myself, doing precisely what you
describe.

Unfortunately the recruiting has been a bit tricky - this is not a typical
job and it's a funny mixture of bureaucracy/politics/social engineering
and programming.  People who are skilled in both areas, are, ah, uncommon.
But it will happen eventually.

Meanwhile I

- ensure that all bugzilla reports are routed to the relevant maintainer

- ensure that all those who reported bugs via email are later asked to
  raise bugzilla reports if it didn't get fixed (but I only monitor one list!)

- continue to file away all the real-looking bugzillas with the intention
  of generating aggregated reports, but you know how it is.

Re: Linux 2.6.21

[Linus Torvalds]

On Sat, 28 Apr 2007, Martin J. Bligh wrote:
> 
> Go to http://bugzilla.kernel.org. Hit query. Find the box that says
> "Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.
> 
> 74 bugs found.
> 
> Not hard to do.

And what part of the "directed" did you miss?

Do you really expect me to go there every day to look at all bugs? That's 
nbot a bug tracker. That's just a noise-maker.

It needs to be email, not some "mouse around for 30 seconds and type 
thing", and it needs to be *directed*. Preferably with somebody who 
actually did some manual scanning over it and spent a few minutes just 
looking at whether it looks like a worthy bug.

In other words: we shouldn't have all developers wasting time doing this. 
It would be much better to have _one_ person (or a group of people) doing 
it, and actually turnign your "Not hard to do" into real information, 
rather than just random data. Adrian did.

The good news is that it looks like now that people are aware of it, we 
hopefully have others who will help do this kind of thing.

		Linus

Re: Linux 2.6.21

[Andrew Morton]

On Sat, 28 Apr 2007 17:07:30 -0700 (PDT) Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Sat, 28 Apr 2007, Martin J. Bligh wrote:
> > 
> > Go to http://bugzilla.kernel.org. Hit query. Find the box that says
> > "Bug Changes, Only bugs changed in the last __ days". Stick 7 in it.
> > 
> > 74 bugs found.
> > 
> > Not hard to do.
> 
> And what part of the "directed" did you miss?
> 
> Do you really expect me to go there every day to look at all bugs? That's 
> nbot a bug tracker. That's just a noise-maker.
> 
> It needs to be email, not some "mouse around for 30 seconds and type 
> thing", and it needs to be *directed*. Preferably with somebody who 
> actually did some manual scanning over it and spent a few minutes just 
> looking at whether it looks like a worthy bug.
> 
> In other words: we shouldn't have all developers wasting time doing this. 

yup.

> It would be much better to have _one_ person (or a group of people) doing 
> it,

I am doing that.  It's only 5-10 a day - routing them to the relevant
culprit is very little work.  It's also very little work for said culprits
to totally ignore said routing, which is a tougher problem.

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 02:13:08PM -0700, Linus Torvalds wrote:
>...
> (I've said this before, but I'll say it again: one thing that would 
> already make bugzilla better is to just always drop any bug reports that 
> are more than a week old and haven't been touched. It wouldn't need *much* 
> touching, but if a reporter cannot be bothered to say "still true with 
> current snapshot" once a week, then it shouldn't be seen as being somehow 
> up to those scare resources we call "developers" to have to go through 
> it).
>...

And if it's a bug in an unmaintained subsystem, a user could do this for 
100 weeks without any effect.
There's no value in keeping reporters busy with useless tasks. [1]

Don't forget:
A good bug report is an important contribution.

We are already quite good at ignoring bug reports that come through 
linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
than 1600 open bugs because this tells how bad we are at handling bugs.
How many thousand bug reports have been ignored during the same time on 
linux-kernel?

If a developer asked for further information and the submitter didn't
answer within 1 months, I will close this bug. [2] And "I will" is not
talking about the future, I'm doing this in the kernel Bugzilla for
three years or so.

The problem is we have tree states of subsystems and drivers:
- unmaintained
- maintained [3]
- maintained and maintainer looks after bug reports

I do hereby promise you to manually ask the submitters of all 1600 open 
bugs in the kernel Bugzilla within one month whether their problem is 
still present with 2.6.21 and forwarding all bugs if the answer was 
"yes" to whoever is the right recipient if you promise me that all bugs 
where the submitter said "yes" will be debugged by a kernel developer 
who knows the corresponding subsystem. [4]

> 		Limis

cu
Adrian

[1] "Still present with the latest kernel?" is a valid question
    if a developer intends to debug further, but otherwise it's silly.
[2] sometimes a bit later, but I'll do it
[3] I'll not do public maintainer bashing, but we have very active 
    maintainers with nearly zero activity in debugging user bug reports
[4] I don't think you'll do - but if you do that's a serious offer

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Russell King]

On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> We are already quite good at ignoring bug reports that come through 
> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> than 1600 open bugs because this tells how bad we are at handling bugs.
> How many thousand bug reports have been ignored during the same time on 
> linux-kernel?

However, look at this bug:

  http://bugme.osdl.org/show_bug.cgi?id=7760

It's outside my knowledge to be able to fix for various reasons:

1. I don't know _anything_ about IXP4xx hardware, so I'm not the right
   person to own this bug.

2. I've no idea who's looking after IXP4xx stuff now.  (When it was
   posted to the ARM kernel list by Rob, it was ignored so I guess the
   IXP4xx maintainer isn't watching for bugs, if such a person even
   exists.)

3. I've little idea about why the MM page allocation is failing early.

4. The ARM DMA bounce code is a hack, and I'm pretty sure that this
   failure is a result of trying to use this contorted code instead
   of the relevant subsystems having a correct DMA mask.

Can you make any suggestion what should be done about this bug?

I'm personally very tempted to close it as "won't fix" (I wish there was
a "can't fix" category.)

As for my other bugs:

  http://bugme.osdl.org/show_bug.cgi?id=8149

Again, EP93xx is not my thing, but I've recently merged a patch to allow
AMBA PL010 uarts (which are present on this platform) to use the clock
control API.

The EP93xx people (provided they're willing) now need to do whatever's
required to resolve that bug.  (Hopefully they've taken ownership of
that bug now.)

  http://bugme.osdl.org/show_bug.cgi?id=4270
  http://bugme.osdl.org/show_bug.cgi?id=5875
  http://bugme.osdl.org/show_bug.cgi?id=7750

I'm no longer serial maintainer.  Bug IDs after about 7000 reflect bugs
submitted since I've resigned my serial maintainership, and therefore
I've ignored them.  It's far easier to ignore bug reports in bugzilla
than it is to get categories reassigned (to whom? - dunno) or even
deleted (if no one steps up presumably that's what needs to happen?)

  http://bugme.osdl.org/show_bug.cgi?id=7389

This one isn't a regression, or even a bug IMHO, but could be viewed as
undesirable behaviour.  Fixing it is utterly non-trivial though.

A serial port which happens to have an IrDA device on it is still a serial
port at the end of the day, and as long as we have legacy probing, a serial
port at a standard COM port address will be detected as such.  There's
never been any facility in the kernel to get around serial ports being
at standard COM port addresses.

Moreover, there's a whole class of applications which want IrDA devices
to be presented to userspace as a serial port rather than a special
network device, so unilaterally ignoring IrDA devices which look like
serial is not really an option.

Basically, IrDA and serial need to develop a way to work in harmony
with each other.  That's a long-term thing though.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

irks with bugzilla (was Re: Linux 2.6.21)

[Stefan Richter]

Russell King wrote:
> On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
>> We are already quite good at ignoring bug reports that come through 
>> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
>> than 1600 open bugs because this tells how bad we are at handling bugs.
>> How many thousand bug reports have been ignored during the same time on 
>> linux-kernel?
> 
> However, look at this bug:
> 
>   http://bugme.osdl.org/show_bug.cgi?id=7760
> 
> It's outside my knowledge to be able to fix for various reasons:
[...]

http://bugzilla.kernel.org/faq.cgi says, although it doesn't make a lot
of sense:

	"Q. If a bug has an owner does that mean they are working on it?

	A. No. If it is not in the ASSIGNED state then no one is working
	on it. The owner defaults to the subsytem maintainer. However,
	anyone who wants to submit a patch or add more info to a bug can
	do so. If the bug is reassigned to someone then the owner field
	will reflect that change."

So the "owner" field is bogus per default.  It would be better if the
bugzilla admins used only meta-addresses instead of a person's address
for any automatically filled-in "owner" field, unless a person
specifically wants to assume this automatic owner role.

I for example am not automatic owner of IEEE 1394 bugs;
drivers_ieee1394@kernel-bugs.osdl.org is.  And I am watching this pseudo
owner.

So in fact, the "owner" field should be replaced by
  - a mail exploder for each component which can be watched by
    interested people,
  - an "assignee" field which is filled in when a bug is assigned to a
    person.

Now that I am at it, another quote from http://bugzilla.kernel.org/faq.cgi :

	"Q. What does a subsystem maintainer do?

	A. He or She will track new bugs and assign them to people or
	reject it for various reasons. They periodically check to make
	sure things are getting worked on and review fixes to make sure
	they are well written."

A maintainer in the project called linux kernel will almost never assign
bugs to people (besides to himself).  He could if he employed or
otherwise supervised people to assign bugs to.

This especially applies to so-called "subsystem maintainers in kernel
tracker", which are not what many people think "subsystem maintainers" are:

	"Q. Why are the subsystem maintainers in kernel tracker
	sometimes different than the person listed in the MAINTAINER
	file?

	A. The subsystem maintainers in kernel tracker are volunteers to
	help track bugs in an area they are interested in. Sometimes
	they are the same person as on kernel.org sometimes they are
	not. There are still some categories with no maintainers so more
	volunteers are needed."

Another quibble:  This FAQ I'm quoting can be reached from the cover
page of bugzilla.kernel.org, i.e. from http://bugzilla.kernel.org/.
However, none of the other web pages of this site link to
http://bugzilla.kernel.org/.  Same for the bugme-admin contact.

Because I find bugzilla.kernel.org quite useful for myself, I filed
these complaints of mine under
http://bugzilla.kernel.org/show_bug.cgi?id=8396
http://bugzilla.kernel.org/show_bug.cgi?id=8397
http://bugzilla.kernel.org/show_bug.cgi?id=8394
in hope that they are taken into consideration by those who maintain the
site.
-- 
Stefan Richter
-=====-=-=== -=-- ===--
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Adrian Bunk]

On Sat, Apr 28, 2007 at 09:27:01PM +0100, Russell King wrote:
> On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> > We are already quite good at ignoring bug reports that come through 
> > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> > than 1600 open bugs because this tells how bad we are at handling bugs.
> > How many thousand bug reports have been ignored during the same time on 
> > linux-kernel?
> 
> However, look at this bug:
> 
>   http://bugme.osdl.org/show_bug.cgi?id=7760
> 
> It's outside my knowledge to be able to fix for various reasons:
>...
> I'm personally very tempted to close it as "won't fix" (I wish there was
> a "can't fix" category.)
>...

So this is a completely debugged bug in a well-maintained subsystem
(no matter what the status in Bugzilla is).

The problem is that the other 1600 open bugs aren't in this state.

> I'm no longer serial maintainer.  Bug IDs after about 7000 reflect bugs
> submitted since I've resigned my serial maintainership, and therefore
> I've ignored them.
>...

That's one of the problems: Unmaintained subsystems.

Since you stepped down as serial maintainer (and it's your right as 
maintainer to do so), the serial subsystem is unmaintained.

That's exactly where Linus' "drop any bug reports that are more than a 
week old" suggestion is completely flawed - no matter what the submitter 
does, how often he tests latest kernels, noone will help him.

> It's far easier to ignore bug reports in bugzilla
> than it is to get categories reassigned (to whom? - dunno) or even
> deleted (if no one steps up presumably that's what needs to happen?)
>...

New subsystems always get default owners like   	 
drivers_ieee1394@kernel-bugs.osdl.org, and people interested in such 
bugs can edit their preferences to watching all (pseudo) users in whose 
bugs they are interested. drivers_serial@kernel-bugs.osdl.org [1] would 
be the logical defult owner.

> Russell King

cu
Adrian

[1] or @linux-foundation.org

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> 
> That's exactly where Linus' "drop any bug reports that are more than a 
> week old" suggestion is completely flawed - no matter what the submitter 
> does, how often he tests latest kernels, noone will help him.

You talk, but what do you actually *suggest*?

Talk is cheap. You use to do the walk too, but you've already said that 
you're not interested in that any more. So excuse me if I'm not impressed. 

The thing is, bugzilla is totally broken because it's designed to help 
track bugs, but it's *not* designed to actually handle the much harder 
problem, which is to actually get the *right* developers to be aware of 
the *right* bugs!

And both of those "right"s are important. Spamming everybody will just 
mean that everybody tunes them out. And spamming even the right developers 
with useless bugreports will also just cause them to tune out the good 
ones.

The thing is, the "tracking bugs" part that bugzilla _can_ do is also 
totally _useless_ without that much more important phase, namely the 
"connect the parties". That's what you really used to do. You made 
developers connect to the reports, because your reports were _useful_ and 
not overly noisy.

But go back and look - did you notice that once you connect the dots, it 
turns out that bugzilla itself isn't all that wonderful. Quite a lot of 
your regression reports had other ways of pointing to the problems, 
including very much mailing list archive pointers etc!

So bugzilla isn't actually all it's touted to be even _once_ the 
connection between reporter and developer has been established. I really 
don't see why you are so hung up about bugzilla, when your own regression 
reports didn't generally point to it all that often!

(I just went back and double-checked: you had more than twice as many 
pointers to kernel mailing list archives than you had pointers to bugzilla 
in the one series I looked at. And I'm _not_ saying that's wrong at all: I 
think the mailing list is actually likely to be at LEAST as useful as 
bugzilla is as a bug-tracker!)

And bugzilla actually falls down even more than the mailing list does for 
the whole (and MUCH MORE IMPORTANT!) phase of connecting developers to bug 
reports. And THAT is really the problem here.

And no, I don't actually think that automatically closing entries that 
haven't gotten any attention in the last two weeks would "fix" bugzilla. 
But I think that it might actually make people 
 (a) more likely to look into bugzilla (and thus maybe improve the 
     "connecting" developers/reporters thing)
 (b) act as a trivial noise-removal thing, because it would give 
     preferential treatment to people who are diligent about their 
     bug-reports and are willing to follow up on them and it would also 
     remove the noise that comes from obvious things that broke and got 
    fixed.

But I think the real fix is to have real humans that help de-noisify the 
bug reports. Let's call them the "bunk" group, just to pick a random 
four-letter combination. The kinds of people who help turn the mindless 
noise that is bugzilla (and the kernel mailing list) into directed 
_information_. 

No, nothing is ever going to be perfect. And "filtering" the noise will 
inevitably also end up dropping real information. But not filtering it 
will guarantee that even more is dropped.

		Linus

Re: Linux 2.6.21

[Andi Kleen]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > 
> > That's exactly where Linus' "drop any bug reports that are more than a 
> > week old" suggestion is completely flawed - no matter what the submitter 
> > does, how often he tests latest kernels, noone will help him.
> 
> You talk, but what do you actually *suggest*?
> 
> Talk is cheap. You use to do the walk too, but you've already said that 
> you're not interested in that any more. So excuse me if I'm not impressed. 
> 
> The thing is, bugzilla is totally broken because it's designed to help 
> track bugs, but it's *not* designed to actually handle the much harder 
> problem, which is to actually get the *right* developers to be aware of 
> the *right* bugs!

This means we need people who figure out who to assign bugs too.
Aka bugmasters.

In theory it could be nearly automated. Figure out what files related
to the bug and assign to the last 5 people who submitted patches
for them and/or signed off. 

Ok I suppose it's not that easy -- you would need some human judgement.

BTW one big problem in our current bugzilla is that a lot of people
cannot reassign bugs they don't own. I sometimes see bugs that I don't
own bug I know who is responsible, but bugzilla doesn't allow me to do it.

So I think what would help:

- Ask more people to just categorize and reassign bugs (anybody interested?)
- Give more people in bugzilla the power to reassign arbitary bugs
(bugzilla maintainers would need to do that)

-Andi

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> Linus Torvalds <torvalds@linux-foundation.org> writes:
> > 
> > The thing is, bugzilla is totally broken because it's designed to help 
> > track bugs, but it's *not* designed to actually handle the much harder 
> > problem, which is to actually get the *right* developers to be aware 
> > of the *right* bugs!
> 
> This means we need people who figure out who to assign bugs too.
> Aka bugmasters.

Yes. But not using bugzilla.

I don't know if you've noticed already, but I'm not the only one that 
doesn't have a very high opinon of bugzilla ;)

What works is somebody who is a bugmaster, and it doesn't really matter 
*what* bug tracker he points to (bugzilla being one of the possibilities, 
although not necessarily the best, and absolutely NOT the only choice), 
and turn them into emails. Once they are emails, bugzilla can track them.

Andrew does this, and yes, Adrian did it.

> In theory it could be nearly automated. Figure out what files related
> to the bug and assign to the last 5 people who submitted patches
> for them and/or signed off. 

I do think you're pretty optimistic. I think it's true for trivial driver 
bugs, but even for trivial driver bugs the initial report is often not 
enough to pinpoint the driver. 

Let's take the sis900 bug as an example (not because I want to rag on that 
being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's 
a good case of something really really easy - and if that easy case isn't 
handled trivially and obviously, then the bug-reporting doesn't work).

In that case, the initial report was (condensed version, but fairly 
accurate): "system hangs on boot at random points. 2.6.21-rc7 worked 
well".

Now, realistically, if that entry had been in bugzilla, what would you do?

Equally realistically, let's ignore bugzilla for a moment, and ask what 
the best method for handling something like this would be? Have an open 
mind, no rules on "have to use bug tracker XYZ". 

You know what? The report went to me and the kernel mailing list as email. 
And that was the *right* thing in that case. There was no good sign of who 
it should go to, and while there wasn't a whole lot of information there, 
there *was* a very tight timeframe (ie it could pinpoint to within about a 
week when it started). But the only thing I could really ask for was for 
the person to bisect it.

Would bugzilla have helped? HELL NO. It would have been a disaster. It 
would have wasted reporter time, it would have wasted developer time, and 
it would likely have been ignored because the bug report wasn't specific 
enough to really trigger any good queries or trigger a maintainer.

So bugzilla wouldn't have helped even for a _trivial_ bug. 

I personally suspect that bugzilla helps more if a bug actually has a real 
history, and people want to actually save that history because they are 
stumped. But I think it should come in *then*, not immediately. Because it 
is actually too intrustive for the simple stuff - and most bugs actually 
are simple, it's just that they also get resolved so simply that people 
don't even think of them as bugs (ie we have a lot of "duh" kind of fixes 
too).

> BTW one big problem in our current bugzilla is that a lot of people
> cannot reassign bugs they don't own. I sometimes see bugs that I don't
> own bug I know who is responsible, but bugzilla doesn't allow me to do it.

I think that's a problem, but I think the thing that I react to most (and 
I know some other people do too) is that I just don't want a broken mousy 
interface that I have to explicitly go and look at. I'd much rather get 
involved by email once it's clear that I need to be involved. Having a 
link to "more information at xyz" (where xyz _can_ be bugzilla too, but 
doesn't have to be!) is fine, but but it really cannot be the first 
interface, not in the current form, at least.

> So I think what would help:
> 
> - Ask more people to just categorize and reassign bugs (anybody interested?)
> - Give more people in bugzilla the power to reassign arbitary bugs
>   (bugzilla maintainers would need to do that)

I think both of these are good ideas, but I don't think it's enough. There 
must be some better form of bug tracking than bugzilla. Some relly 
distributed way of letting people work together, *without* having to 
congregate on some central web-site kind of thing. A "git for bugs", where 
you can track bugs locally and without a web interface.

And I think it would be something much closer to "distributed email mbox 
with status tracking facilities", _not_ a web clicky interface.

		Linus

Re: Linux 2.6.21

[Stefan Richter]

Linus Torvalds wrote:
> On Sun, 29 Apr 2007, Andi Kleen wrote:
>> - Ask more people to just categorize and reassign bugs (anybody interested?)
>> - Give more people in bugzilla the power to reassign arbitary bugs
>>   (bugzilla maintainers would need to do that)
> 
> I think both of these are good ideas, but I don't think it's enough. There 
> must be some better form of bug tracking than bugzilla. Some relly 
> distributed way of letting people work together, *without* having to 
> congregate on some central web-site kind of thing. A "git for bugs", where 
> you can track bugs locally and without a web interface.
> 
> And I think it would be something much closer to "distributed email mbox 
> with status tracking facilities", _not_ a web clicky interface.

It has to be able to suck in reports from people who don't know much
about how the Linux guys handle bugs, and has to keep the reporters
involved up until a bug can be closed.  IOW it has to be compatible with
integrators, developers, and reporters/testers.  Luckily though, not all
of them need all of that system's features.  ('system' == the right
people with the right tools)
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 18:07, Linus Torvalds wrote:
> 
> On Sun, 29 Apr 2007, Andi Kleen wrote:
> > Linus Torvalds <torvalds@linux-foundation.org> writes:
> > > 
> > > The thing is, bugzilla is totally broken because it's designed to help 
> > > track bugs, but it's *not* designed to actually handle the much harder 
> > > problem, which is to actually get the *right* developers to be aware 
> > > of the *right* bugs!
> > 
> > This means we need people who figure out who to assign bugs too.
> > Aka bugmasters.
> 
> Yes. But not using bugzilla.
> 
> I don't know if you've noticed already, but I'm not the only one that 
> doesn't have a very high opinon of bugzilla ;)
> 
> What works is somebody who is a bugmaster, and it doesn't really matter 
> *what* bug tracker he points to (bugzilla being one of the possibilities, 
> although not necessarily the best, and absolutely NOT the only choice), 
> and turn them into emails. Once they are emails, bugzilla can track them.
> 
> Andrew does this, and yes, Adrian did it.
> 
> > In theory it could be nearly automated. Figure out what files related
> > to the bug and assign to the last 5 people who submitted patches
> > for them and/or signed off. 
> 
> I do think you're pretty optimistic. I think it's true for trivial driver 
> bugs, but even for trivial driver bugs the initial report is often not 
> enough to pinpoint the driver. 
> 
> Let's take the sis900 bug as an example (not because I want to rag on that 
> being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's 
> a good case of something really really easy - and if that easy case isn't 
> handled trivially and obviously, then the bug-reporting doesn't work).
> 
> In that case, the initial report was (condensed version, but fairly 
> accurate): "system hangs on boot at random points. 2.6.21-rc7 worked 
> well".
> 
> Now, realistically, if that entry had been in bugzilla, what would you do?
> 
> Equally realistically, let's ignore bugzilla for a moment, and ask what 
> the best method for handling something like this would be? Have an open 
> mind, no rules on "have to use bug tracker XYZ". 
> 
> You know what? The report went to me and the kernel mailing list as email. 
> And that was the *right* thing in that case. There was no good sign of who 
> it should go to, and while there wasn't a whole lot of information there, 
> there *was* a very tight timeframe (ie it could pinpoint to within about a 
> week when it started). But the only thing I could really ask for was for 
> the person to bisect it.
> 
> Would bugzilla have helped? HELL NO. It would have been a disaster. It 
> would have wasted reporter time, it would have wasted developer time, and 
> it would likely have been ignored because the bug report wasn't specific 
> enough to really trigger any good queries or trigger a maintainer.
> 
> So bugzilla wouldn't have helped even for a _trivial_ bug. 
> 
> I personally suspect that bugzilla helps more if a bug actually has a real 
> history, and people want to actually save that history because they are 
> stumped. But I think it should come in *then*, not immediately. Because it 
> is actually too intrustive for the simple stuff - and most bugs actually 
> are simple, it's just that they also get resolved so simply that people 
> don't even think of them as bugs (ie we have a lot of "duh" kind of fixes 
> too).

I completely agree.

My personal experience with bugzilla is that it's very unfriendly to
reporters.  IMHO it's suitable for tracking unresolved problems along with
debug patches, system information etc., but not for _reporting_ new ones.

Greetings,
Rafael

Re: Linux 2.6.21

[Andi Kleen]

> My personal experience with bugzilla is that it's very unfriendly to
> reporters.  IMHO it's suitable for tracking unresolved problems along with
> debug patches, system information etc., but not for _reporting_ new ones.

What did you find unfriendly? 

While I also cannot say I love it I don't find it any less unfriendly
than an email.

Besides the primary point of bug tracking is not to be friendly
to someone, but to (a) fix the bugs and (b) know how many bugs
there for a given release. Any replacement would need to solve
this problem too.

Email does not solve it as far as I can see.

-Andi

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> 
> Besides the primary point of bug tracking is not to be friendly
> to someone, but to (a) fix the bugs and (b) know how many bugs
> there for a given release. Any replacement would need to solve
> this problem too.
> 
> Email does not solve it as far as I can see.

Email fixes a _lot_ more bugs than bugzilla does. 

End of story. I don't think anybody who cannot accept that UNDENIABLE FACT 
should even participate in this discussion. Wake up and look at all the 
bugs we fix - most of them have never been in bugzilla.

That's a FACT.

Don't go around ignoring reality.

		Linus

regression tracking (Re: Linux 2.6.21)

[Oleg Verych]

* Newsgroups: gmane.linux.kernel
* Date: Sun, 29 Apr 2007 10:50:22 -0700 (PDT)

* From: Linus Torvalds
>
> On Sun, 29 Apr 2007, Andi Kleen wrote:
>> 
>> Besides the primary point of bug tracking is not to be friendly
>> to someone, but to (a) fix the bugs and (b) know how many bugs
>> there for a given release. Any replacement would need to solve
>> this problem too.
>> 
>> Email does not solve it as far as I can see.
>
> Email fixes a _lot_ more bugs than bugzilla does. 
>
> End of story. I don't think anybody who cannot accept that UNDENIABLE FACT 
> should even participate in this discussion. Wake up and look at all the 
> bugs we fix - most of them have never been in bugzilla.
>
> That's a FACT.
>
> Don't go around ignoring reality.

I'm seeing this long (198) thread and just have no idea how it has
ended (wiki? hand-mailing?).

Just two general questions to Adrian.

1) You was maintainer of the woody backports, isn't it[0]? Why you didn't
proposed (used) Debian's BTS as alternative to bugzilla, and how you did
your regression tracking?

What exactly doesn't fit? Full control by e-mail, comprehensive
management, ML handling/redirection, tagging, sorting, searching? Finally,
reportbug tool and web-inteface?

2) Your decision to stop activity, was that with debian because Sarge was
release with known security hole in the kernel[1]?

I'm just wonder.

[0] google: "woody backports Adrian Bunk"

[1] Message-ID: <20070331194728.GA31853@powerlinux.fr>
    Xref: news.gmane.org gmane.linux.debian.devel.kernel:27730
    [Just take your news readers and have fun with Gmane!]
    [For those, who don't know what it is -- web :]
    Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/27730>
--*--
    Unfortunately this message is from a man, who was punished in very
    unfair manner by "fellow developers". I'm not trying to rise this
    issue (sorry, if i'm trolling), just want to say, that life can be
    very unfair, when some wrong people are in power...
    
    Message-ID: <20070529053026.GA28352@powerlinux.fr>
    Xref: news.gmane.org gmane.linux.debian.devel.project:12330
    Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.project/12330>
____

Re: regression tracking (Re: Linux 2.6.21)

[Stefan Richter]

Oleg Verych wrote:
> I'm seeing this long (198) thread and just have no idea how it has
> ended (wiki? hand-mailing?).

Direct or indirect results:

  - See Michal Piotrowski's periodic posts and
    http://kernelnewbies.org/known_regressions .

  - Meanwhile, the people who maintain bugzilla.kernel.org seem to work
    on improvements.  I noticed that (a) each page now has a backlink to
    the bugzilla.kernel.org start page, (b) the show_bug.cgi=... page
    layout is now an unreadable mess, (c) e-mail integration is still
    the same (it's impossible at least for me to send e-mails to bugs).

[...]
> Why you didn't proposed (used) Debian's BTS as alternative to bugzilla,
[...]

BTS has been mentioned in that thread in a few posts; mostly positively
as I recall.
-- 
Stefan Richter
-=====-=-=== -==- -===-
http://arcgraph.de/sr/

Re: regression tracking (Re: Linux 2.6.21)

[Oleg Verych]

On Thu, Jun 14, 2007 at 05:33:40PM +0200, Stefan Richter wrote:
[]
> [...]
> > Why you didn't proposed (used) Debian's BTS as alternative to bugzilla,
> [...]
> 
> BTS has been mentioned in that thread in a few posts; mostly positively
> as I recall.

I know, that most developers here are not working/familiar with what
Debian has as its bug shooting weapon ``The system is mainly controlled
by  e-mail, but the bug reports can be viewed using the WWW.''[0].

I thought somebody, who familiar with that, might propose to setup/tune
it, but not doing yet another NIH thing, especially from e-mail
integration POV. I doubt mozilla guys can think about it without
javascript and/or java servlets :)

[0] <http://www.debian.org/Bugs/>
____

Re: regression tracking (Re: Linux 2.6.21)

[Stefan Richter]

Oleg Verych wrote:
> I thought somebody, who familiar with that, might propose to setup/tune
> it, but not doing yet another NIH thing,

I may have missed something, but I recall that Adrian's bugtracking,
while it lasted, and now Michal's continuing it mostly came into being
because Adrian just started doing it and others soon found it very useful.
-- 
Stefan Richter
-=====-=-=== -==- -===-
http://arcgraph.de/sr/

Re: regression tracking (Re: Linux 2.6.21)

[Adrian Bunk]

On Thu, Jun 14, 2007 at 06:39:23PM +0200, Oleg Verych wrote:
> On Thu, Jun 14, 2007 at 05:33:40PM +0200, Stefan Richter wrote:
> []
> > [...]
> > > Why you didn't proposed (used) Debian's BTS as alternative to bugzilla,
> > [...]
> > 
> > BTS has been mentioned in that thread in a few posts; mostly positively
> > as I recall.
> 
> I know, that most developers here are not working/familiar with what
> Debian has as its bug shooting weapon ``The system is mainly controlled
> by  e-mail, but the bug reports can be viewed using the WWW.''[0].
> 
> I thought somebody, who familiar with that, might propose to setup/tune
> it, but not doing yet another NIH thing, especially from e-mail
> integration POV. I doubt mozilla guys can think about it without
> javascript and/or java servlets :)
>...

The problem isn't Bugzilla, and the Debian BTS wouldn't solve any 
problem.

What is missing?

We need people who know one or more subsystems and who are willing to 
regularly handle bug reports in their area.

And we need a release process that makes debugging, and if possible 
fixing, all regressions prior to the release mandatory. You might never 
come down to zero regressions and might not be able to handle all 
last-minute reported regressions, but the 2.6.21 situation with 3 week 
old known regressions not ever being debugged by a kernel developer 
before the release has much room for improvements.

Changing the BTS would make sense if some core developers would state 
that they would start using the BTS after this change. But otherwise it 
doesn't matter which BTS to use.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: regression tracking (Re: Linux 2.6.21)

[Oleg Verych]

On Thu, Jun 14, 2007 at 07:30:49PM +0200, Adrian Bunk wrote:
> On Thu, Jun 14, 2007 at 06:39:23PM +0200, Oleg Verych wrote:
[]
> > I know, that most developers here are not working/familiar with what
> > Debian has as its bug shooting weapon ``The system is mainly controlled
> > by  e-mail, but the bug reports can be viewed using the WWW.''[0].
> > 
> > I thought somebody, who familiar with that, might propose to setup/tune
> > it, but not doing yet another NIH thing, especially from e-mail
> > integration POV. I doubt mozilla guys can think about it without
> > javascript and/or java servlets :)
> >...
> 
> The problem isn't Bugzilla, and the Debian BTS wouldn't solve any 
> problem.
> 
> What is missing?
> 
> We need people who know one or more subsystems and who are willing to 
> regularly handle bug reports in their area.

I think if somebody, by example will show how it can be handled in more
convenient way, that will eventually become mainstream. As we know,
nothing gets from vacuum just like that, without taking energy and time.
And my question was not about this social problem of acceptance, support
etc.

Linus had spent some time in this thread trying to explain what problems
are: as from that (social, think scheduler :) POV, as also from
"zarro bogs found" one.

Also, after i saw Linus' message about doing mostly tools last couple of
years, i wonder why you, Adrian, didn't think about your tools first,
before you've started regression tracking? You are not running in front
of a train, unlike you know who does, plus bugzilla issues are known for
years. Luckily Fedora kernel guys also upstream developers, thus lkml and
other MLs under their view.

After having read all that, i've asked you, my question, as the person
who supposedly used BTS as a maintainer.

Yes, in current form it might not be in suitable configuration, i.e.
kernel sub-systems instead of packages etc, anyway main thing is the way
BTS is handled. While i was looking and replying for bug reports in the
Debian kernel, that i saw in lkml, i've noticed, just how guys work with
it there. Now they even came up with tracking upstream bugzilla, it
seems [0]. I left that activity due to RL some months ago, but now trying
to catch up things again.

Thus it's just my curiosity about all this. And BTS is like, you know,
why not, if it fits by mostly all parameters?

[0] Message-ID: <handler.s.C.118074292516912.transcript@bugs.debian.org>
    Xref: news.gmane.org gmane.linux.debian.devel.kernel:29426
    Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29426>


> And we need a release process that makes debugging, and if possible 
> fixing, all regressions prior to the release mandatory. You might never 
> come down to zero regressions and might not be able to handle all 
> last-minute reported regressions, but the 2.6.21 situation with 3 week 
> old known regressions not ever being debugged by a kernel developer 
> before the release has much room for improvements.
> 
> Changing the BTS would make sense if some core developers would state 
> that they would start using the BTS after this change. But otherwise it 
> doesn't matter which BTS to use.

So, as i've wrote before: one must give them pretty-shiny tool, kindly
barking in their inboxes, instead of for example

"Guilty: **** ***** <????@****.com>",

as it was on the very beginning.
____

Re: regression tracking (Re: Linux 2.6.21)

[Adrian Bunk]

On Thu, Jun 14, 2007 at 10:33:38PM +0200, Oleg Verych wrote:
>...
> Also, after i saw Linus' message about doing mostly tools last couple of
> years, i wonder why you, Adrian, didn't think about your tools first,
> before you've started regression tracking? You are not running in front
> of a train, unlike you know who does, plus bugzilla issues are known for
> years. Luckily Fedora kernel guys also upstream developers, thus lkml and
> other MLs under their view.

My tool was a textfile with a text editor.
For a smaller amount of regressions that works fine.

And it's not that Linus started developing the Linux kernel with writing 
git, the first 10 years of Linux development were without any SCM.

> After having read all that, i've asked you, my question, as the person
> who supposedly used BTS as a maintainer.
> 
> Yes, in current form it might not be in suitable configuration, i.e.
> kernel sub-systems instead of packages etc, anyway main thing is the way
> BTS is handled. While i was looking and replying for bug reports in the
> Debian kernel, that i saw in lkml, i've noticed, just how guys work with
> it there. Now they even came up with tracking upstream bugzilla, it
> seems [0]. I left that activity due to RL some months ago, but now trying
> to catch up things again.
>...

Both the Debian BTS and Bugzilla are usable programs with their own
advantages and disadvantages.

I don't believe switching to the Debian BTS would solve any problem.

> > And we need a release process that makes debugging, and if possible 
> > fixing, all regressions prior to the release mandatory. You might never 
> > come down to zero regressions and might not be able to handle all 
> > last-minute reported regressions, but the 2.6.21 situation with 3 week 
> > old known regressions not ever being debugged by a kernel developer 
> > before the release has much room for improvements.
> > 
> > Changing the BTS would make sense if some core developers would state 
> > that they would start using the BTS after this change. But otherwise it 
> > doesn't matter which BTS to use.
> 
> So, as i've wrote before: one must give them pretty-shiny tool, kindly
> barking in their inboxes, instead of for example
> 
> "Guilty: **** ***** <????@****.com>",
> 
> as it was on the very beginning.

A pretty-shiny tools wouldn't change anything.

What you need are humans debugging the regresssions and humans remining 
other humans that they should debug the regressions.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: regression tracking (Re: Linux 2.6.21)

[Linus Torvalds]

On Thu, 14 Jun 2007, Oleg Verych wrote:
> 
> I'm seeing this long (198) thread and just have no idea how it has
> ended (wiki? hand-mailing?).

I'm hoping it's not "ended".

IOW, I really don't think we _resolved_ anything, although the work that 
Adrian started is continuing through the wiki and other people trying to 
track regressions, and that was obviously something good.

But I don't think we really know where we want to take this thing in the 
long run. I think everybody wants a better bug-tracking system, but 
whether something that makes people satisfied can even be built is open. 
It sure doesn't seem to exist right now ;)

		Linus

Re: regression tracking (Re: Linux 2.6.21)

[Adrian Bunk]

On Fri, Jun 15, 2007 at 04:20:57PM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 14 Jun 2007, Oleg Verych wrote:
> > 
> > I'm seeing this long (198) thread and just have no idea how it has
> > ended (wiki? hand-mailing?).
> 
> I'm hoping it's not "ended".
> 
> IOW, I really don't think we _resolved_ anything, although the work that 
> Adrian started is continuing through the wiki and other people trying to 
> track regressions, and that was obviously something good.
> 
> But I don't think we really know where we want to take this thing in the 
> long run. I think everybody wants a better bug-tracking system, but 
> whether something that makes people satisfied can even be built is open. 
> It sure doesn't seem to exist right now ;)

The problem is not the bug tracking system, be it manual tracking in a 
text file or a Wiki or be it in Bugzilla or any other bug tracking 
system.

One problem is the lack of experienced developers willing to debug bug 
reports.

But what really annoyed me was the missing integration of regression 
tracking into the release process, IOW how _you_ handled the regression 
lists.

If we want to offer something less of a disaster than 2.6.21, and if we 
want to encourage people to start and continue testing -rc kernels, we 
must try to fix as many reported regressions as reasonably possible.

This means going through every single point in the regression list 
asking "Have we tried everything possible to solve this regression?". 
There are very mysterious regressions and there are regressions that 
might simply be reported too late. But if at the time of the final 
release 3 week old regressions hadn't been debugged at all there's 
definitely room for improvement. And mere mortals like me reminding 
people is often not enough, sometimes an email by Linus Torvalds himself 
asking a patch author or maintainer to look after a regression might be 
required.

And a low hanging fruit to improve the release would be if you could 
release one last -rc, wait for 48 hours, and then release either this 
-rc unchanged as -final or another -rc (and wait another 48 hours).
There were at least two different regressions people ran into in 2.6.21 
who successfully tested -rc7.

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: regression tracking (Re: Linux 2.6.21)

[Oleg Verych]

On Sat, Jun 16, 2007 at 01:42:02AM +0200, Adrian Bunk wrote:
> On Fri, Jun 15, 2007 at 04:20:57PM -0700, Linus Torvalds wrote:
> > 
> > 
> > On Thu, 14 Jun 2007, Oleg Verych wrote:
> > > 
> > > I'm seeing this long (198) thread and just have no idea how it has
> > > ended (wiki? hand-mailing?).
> > 
> > I'm hoping it's not "ended".
> > 
> > IOW, I really don't think we _resolved_ anything, although the work that 
> > Adrian started is continuing through the wiki and other people trying to 
> > track regressions, and that was obviously something good.
> > 
> > But I don't think we really know where we want to take this thing in the 
> > long run. I think everybody wants a better bug-tracking system, but 
> > whether something that makes people satisfied can even be built is open. 
> > It sure doesn't seem to exist right now ;)
> 
> The problem is not the bug tracking system, be it manual tracking in a 
> text file or a Wiki or be it in Bugzilla or any other bug tracking 
> system.
> 
> One problem is the lack of experienced developers willing to debug bug 
> reports.

*debug*

I hope you saw what subject i've chosen to bring this discussion back.
Yes, "tracking", as the first brick for big wall.

Your arguments about developers and users, you've said already, but i've
asked different questions, have i?

Lets look on regular automatic report, like this one:

Message-ID: <E1Hz5HK-0007uB-MO@merkel.debian.org>
Xref: news.gmane.org gmane.linux.debian.devel.general:116248
Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.general/116248

And what we see? Basic packages, like ``dpkg'', ``grub'', ``mc'' are
in the list, requesting help. And as you can see for quite some time.
And it's *OK*, because distribution is working, development is going on.
Sometimes slowly, sometimes with delays.

> But what really annoyed me was the missing integration of regression 
> tracking into the release process, IOW how _you_ handled the regression 
> lists.

So, _tracking_ or _debugging_?

> If we want to offer something less of a disaster than 2.6.21, and if we 
> want to encourage people to start and continue testing -rc kernels, we 
> must try to fix as many reported regressions as reasonably possible.

*tracking*

Despite of tools, Debian have such thing as long release cycles, so
called ``Debian sickness''. And reason, i see, is what you've just
pointed out: less disaster, zer0 RC bugs. Plus everybody is volunteer,
big chunk of bureaucracy-based decisions. And all this for about
15000 packages.

* + reporting*

One side Linux is facing is hardware, and that kind of thing is very-very
diverse. LKML traffic is huge, yet there's no suitable tracking and
reporting *tool*.

> This means going through every single point in the regression list 
> asking "Have we tried everything possible to solve this regression?". 
> There are very mysterious regressions and there are regressions that 
> might simply be reported too late. But if at the time of the final 
> release 3 week old regressions hadn't been debugged at all there's 
> definitely room for improvement. And mere mortals like me reminding 
> people is often not enough, sometimes an email by Linus Torvalds himself 
> asking a patch author or maintainer to look after a regression might be 
> required.

*social* (first approximation)

That's a social problem, just like Debian loosing good kernel team members.

For example you feel, that you've wasted time. But after all, if you've
came up with some kind of tool, everybody else could take it. No
problems, useful ideas must and will evolve. But _ideally_ this must not be
from ground zero every time. _Ideally_ from technical, not personal
point of view ;).

That's why people in Debian have started *team* maintenance with alioth. 

Unfortunately problems with individuals in big machine with bad people,
got randomly elected, can't be solved (IMHO). Even LKML's rule "patches
are welcome", that is very technical, thus good, doesn't work there.

Finally, read the end of 2.6.21 release message ;)

> And a low hanging fruit to improve the release would be if you could 
> release one last -rc, wait for 48 hours, and then release either this 
> -rc unchanged as -final or another -rc (and wait another 48 hours).
> There were at least two different regressions people ran into in 2.6.21 
> who successfully tested -rc7.

What about Linus' tree is a development tree, Andrew's one is a "crazy
development one" (quoting Linus)?

What about open (web page still exists) position on bug manager in
Google?

What about *volunteers* working from both developer's and user's
sides? What about "release is a reward" for everybody?

Balanced eco-system will oscillate. Be it .19(---++), .20(-++++),
.21(----+) *relese*. That's natural, unless pushed to extremes.

I think, i can trust Linus in it, and you are welcome too.

*tools*

That's why i'm talking about tools, and started to discuss them.

My last try: reportbug (there's "-ng" one also), Debian BTS.

Adrian, what can/must be done to adopt them? If not, your experience may
provide information about "why?" (re-consider my first mail about
background, please).
____

Re: regression tracking (Re: Linux 2.6.21)

[Adrian Bunk]

On Sat, Jun 16, 2007 at 03:32:36AM +0200, Oleg Verych wrote:
> On Sat, Jun 16, 2007 at 01:42:02AM +0200, Adrian Bunk wrote:
> > On Fri, Jun 15, 2007 at 04:20:57PM -0700, Linus Torvalds wrote:
> > > 
> > > 
> > > On Thu, 14 Jun 2007, Oleg Verych wrote:
> > > > 
> > > > I'm seeing this long (198) thread and just have no idea how it has
> > > > ended (wiki? hand-mailing?).
> > > 
> > > I'm hoping it's not "ended".
> > > 
> > > IOW, I really don't think we _resolved_ anything, although the work that 
> > > Adrian started is continuing through the wiki and other people trying to 
> > > track regressions, and that was obviously something good.
> > > 
> > > But I don't think we really know where we want to take this thing in the 
> > > long run. I think everybody wants a better bug-tracking system, but 
> > > whether something that makes people satisfied can even be built is open. 
> > > It sure doesn't seem to exist right now ;)
> > 
> > The problem is not the bug tracking system, be it manual tracking in a 
> > text file or a Wiki or be it in Bugzilla or any other bug tracking 
> > system.
> > 
> > One problem is the lack of experienced developers willing to debug bug 
> > reports.
> 
> *debug*
> 
> I hope you saw what subject i've chosen to bring this discussion back.
> Yes, "tracking", as the first brick for big wall.

Tracking regressions is not a real problem.
Especially since it doesn't require much technical knowledge.

> Your arguments about developers and users, you've said already, but i've
> asked different questions, have i?
> 
> Lets look on regular automatic report, like this one:
> 
> Message-ID: <E1Hz5HK-0007uB-MO@merkel.debian.org>
> Xref: news.gmane.org gmane.linux.debian.devel.general:116248
> Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.general/116248
> 
> And what we see? Basic packages, like ``dpkg'', ``grub'', ``mc'' are
> in the list, requesting help. And as you can see for quite some time.
> And it's *OK*, because distribution is working, development is going on.
> Sometimes slowly, sometimes with delays.

I sent weekly regression reports.
Not automatically generated but manually - but that doesn't matter.

The problem is that sending reports itself does not fix anything.

> > But what really annoyed me was the missing integration of regression 
> > tracking into the release process, IOW how _you_ handled the regression 
> > lists.
> 
> So, _tracking_ or _debugging_?

_debugging_ (can be indirectly by poking other people to do debugging)

> > If we want to offer something less of a disaster than 2.6.21, and if we 
> > want to encourage people to start and continue testing -rc kernels, we 
> > must try to fix as many reported regressions as reasonably possible.
> 
> *tracking*

no, *debugging*

I tracked regressions for the 2.6.21 disaster, and the not debugged 
regressions that I had tracked are exactly where we should be better.

>...
> > This means going through every single point in the regression list 
> > asking "Have we tried everything possible to solve this regression?". 
> > There are very mysterious regressions and there are regressions that 
> > might simply be reported too late. But if at the time of the final 
> > release 3 week old regressions hadn't been debugged at all there's 
> > definitely room for improvement. And mere mortals like me reminding 
> > people is often not enough, sometimes an email by Linus Torvalds himself 
> > asking a patch author or maintainer to look after a regression might be 
> > required.
> 
> *social* (first approximation)

Yes.

> That's a social problem, just like Debian loosing good kernel team members.

Different social problem.

> For example you feel, that you've wasted time. But after all, if you've
> came up with some kind of tool, everybody else could take it. No
> problems, useful ideas must and will evolve. But _ideally_ this must not be
> from ground zero every time. _Ideally_ from technical, not personal
> point of view ;).

As I expected, someone else has picked up regression tracking.
And other from what you claim, this did not require any kind of tool.

> That's why people in Debian have started *team* maintenance with alioth. 

The problem for the Linux kernel is that for a better bug handling you'd 
need people willing to learn other people's code and to do the hard work 
of debugging bug reports. E.g. writing a new filesystem is simply much 
more fun than learning and debugging other people's code in some old 
filesystem.

Talking about "team maintenance" sounds nice, but the problem in the 
kernel starts with code that has zero maintainers. And if there's 
already a maintainer, it's unlikely that he'll not accept patches from 
some new person debugging bug reports. But how to find people who will 
become good maintainers?

> Unfortunately problems with individuals in big machine with bad people,
> got randomly elected, can't be solved (IMHO). Even LKML's rule "patches
> are welcome", that is very technical, thus good, doesn't work there.

Debian has it's own problems, Linux kernel development at least has a 
working structure for getting decisions and regular releases.

> Finally, read the end of 2.6.21 release message ;)
> 
> > And a low hanging fruit to improve the release would be if you could 
> > release one last -rc, wait for 48 hours, and then release either this 
> > -rc unchanged as -final or another -rc (and wait another 48 hours).
> > There were at least two different regressions people ran into in 2.6.21 
> > who successfully tested -rc7.
> 
> What about Linus' tree is a development tree, Andrew's one is a "crazy
> development one" (quoting Linus)?
> 
> What about open (web page still exists) position on bug manager in
> Google?
> 
> What about *volunteers* working from both developer's and user's
> sides? What about "release is a reward" for everybody?

People aren't that dumb that some empty words like "release is a reward"
would change anything.

> Balanced eco-system will oscillate. Be it .19(---++), .20(-++++),
> .21(----+) *relese*. That's natural, unless pushed to extremes.
> 
> I think, i can trust Linus in it, and you are welcome too.
> 
> *tools*
> 
> That's why i'm talking about tools, and started to discuss them.
> 
> My last try: reportbug (there's "-ng" one also), Debian BTS.
> 
> Adrian, what can/must be done to adopt them? If not, your experience may
> provide information about "why?" (re-consider my first mail about
> background, please).

Bug tracking for the kernel is more or less working.
The main problem is getting people to debug bug reports.

We need the main problem fixed, not a different tool in an area that is 
not the main problem.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: regression tracking (Re: Linux 2.6.21)

[Oleg Verych]

[I've added Herbert as former kernel team member in the debian(AFAIK),
 sorry, if i'm wrong and you have no opinion on that, Herbert.]

On Sat, Jun 16, 2007 at 04:55:16AM +0200, Adrian Bunk wrote:
> On Sat, Jun 16, 2007 at 03:32:36AM +0200, Oleg Verych wrote:
> > On Sat, Jun 16, 2007 at 01:42:02AM +0200, Adrian Bunk wrote:
> > > On Fri, Jun 15, 2007 at 04:20:57PM -0700, Linus Torvalds wrote:
> > > > 
> > > > 
> > > > On Thu, 14 Jun 2007, Oleg Verych wrote:
> > > > > 
> > > > > I'm seeing this long (198) thread and just have no idea how it has
> > > > > ended (wiki? hand-mailing?).
> > > > 
> > > > I'm hoping it's not "ended".
> > > > 
> > > > IOW, I really don't think we _resolved_ anything, although the work that 
> > > > Adrian started is continuing through the wiki and other people trying to 
> > > > track regressions, and that was obviously something good.
> > > > 
> > > > But I don't think we really know where we want to take this thing in the 
> > > > long run. I think everybody wants a better bug-tracking system, but 
> > > > whether something that makes people satisfied can even be built is open. 
> > > > It sure doesn't seem to exist right now ;)
> > > 
> > > The problem is not the bug tracking system, be it manual tracking in a 
> > > text file or a Wiki or be it in Bugzilla or any other bug tracking 
> > > system.
> > > 
> > > One problem is the lack of experienced developers willing to debug bug 
> > > reports.
> > 
> > *debug*
> > 
> > I hope you saw what subject i've chosen to bring this discussion back.
> > Yes, "tracking", as the first brick for big wall.
> 
> Tracking regressions is not a real problem.
> Especially since it doesn't require much technical knowledge.

I've tried to express different point of view. We have different ones {0}.
Thus, no comments.
 
> > Your arguments about developers and users, you've said already, but i've
> > asked different questions, have i?
> > 
> > Lets look on regular automatic report, like this one:
> > 
> > Message-ID: <E1Hz5HK-0007uB-MO@merkel.debian.org>
> > Xref: news.gmane.org gmane.linux.debian.devel.general:116248
> > Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.general/116248
> > 
> > And what we see? Basic packages, like ``dpkg'', ``grub'', ``mc'' are
> > in the list, requesting help. And as you can see for quite some time.
> > And it's *OK*, because distribution is working, development is going on.
> > Sometimes slowly, sometimes with delays.
> 
> I sent weekly regression reports.
> Not automatically generated but manually - but that doesn't matter.
> 
> The problem is that sending reports itself does not fix anything.

...{1}

> > > But what really annoyed me was the missing integration of regression 
> > > tracking into the release process, IOW how _you_ handled the regression 
> > > lists.
> > 
> > So, _tracking_ or _debugging_?
> 
> _debugging_ (can be indirectly by poking other people to do debugging)
> 
> > > If we want to offer something less of a disaster than 2.6.21, and if we 
> > > want to encourage people to start and continue testing -rc kernels, we 
> > > must try to fix as many reported regressions as reasonably possible.
> > 
> > *tracking*
> 
> no, *debugging*
> 
> I tracked regressions for the 2.6.21 disaster, and the not debugged 
> regressions that I had tracked are exactly where we should be better.

...{2}

> >...
> > > This means going through every single point in the regression list 
> > > asking "Have we tried everything possible to solve this regression?". 
> > > There are very mysterious regressions and there are regressions that 
> > > might simply be reported too late. But if at the time of the final 
> > > release 3 week old regressions hadn't been debugged at all there's 
> > > definitely room for improvement. And mere mortals like me reminding 
> > > people is often not enough, sometimes an email by Linus Torvalds himself 
> > > asking a patch author or maintainer to look after a regression might be 
> > > required.
> > 
> > *social* (first approximation)
> 
> Yes.
> 
> > That's a social problem, just like Debian loosing good kernel team members.
> 
> Different social problem.

The term ``like'' here means people are not able/willing to do work, they
might/can do. And cause of it is *social*, not technical. {1},{2} are
results of that problem/behavior. But according to {0}, you think,
differently.

> > For example you feel, that you've wasted time. But after all, if you've
> > came up with some kind of tool, everybody else could take it. No
> > problems, useful ideas must and will evolve. But _ideally_ this must not be
> > from ground zero every time. _Ideally_ from technical, not personal
> > point of view ;).
> 
> As I expected, someone else has picked up regression tracking.
> And other from what you claim, this did not require any kind of tool.

So you expected good, doing bad. ``Bad'' means bringing pointless flame
about what everybody should do, without constructive approach like: "OK,
i can't do it due to my POV{0}, useless manual work. Everybody willing to
bring another way of dealing with it is welcome."

Your first reply:

"And it's not that Linus started developing the Linux kernel with writing
git, the first 10 years of Linux development were without any SCM." {3}

to my note about, that you are not hurry anywhere, that after all that
years of Open Source and Free Software development you are not trying
to deal with such important thing like regression/bug tracking in
***organized way***, is rather pointless.

> > That's why people in Debian have started *team* maintenance with alioth. 
> 
> The problem for the Linux kernel is that for a better bug handling you'd 
> need people willing to learn other people's code and to do the hard work 
> of debugging bug reports.

... {0}++

Do you know, for example, why i'm not making my "hacker's career"
doing that?

1. because i ended up with lynx, slrn, mutt, emacs-nox. Including
   "zarro bogs found" kind of thing and other "userspace suck" problems.
2. i have no way to know if something *really* broken, unless it right
   on my hardware

This all unlike Debian BTS using reportbug, where you have basic
information, mbox format messages for easy "mutt -f", and other funny
things, real maintainers aware of (i'm trying to know, learn about).

Thus organized, non brain-damaged way of reporting and tracking is the
key issue.

> E.g. writing a new filesystem is simply much more fun than learning and
> debugging other people's code in some old filesystem.

It's like in {3} -- i don't like it (personally), so i'm going along.

That's wrong and counter productive, as i've tried to explain.

> Talking about "team maintenance" sounds nice, but the problem in the 
> kernel starts with code that has zero maintainers.

Floppy went pretty fine, before it was started to be maintained, and
you know that. But you also told that unmaintained and not-working are
different things.

Thus, if that just happen to break, well reports are welcome, and if
long time run will show, that user count is small, so let be it.
Nothing is long forever.

Positive side i think obvious, because Linus have found ext2 bug back
under the New Year tree (AFAIK), Thomas did his best on timers, etc.

For more productivity score system must be implemented and synchronized[0]
with distributions. Only after that *noise* filter, you may claim
importance of problems. Otherwise, you must know how noise affects human
beings.

(In the prev. e-mail i've mentioned such effort from one of the DDs:
[0] Message-ID: <handler.s.C.118074292516912.transcript@bugs.debian.org>
    Xref: news.gmane.org gmane.linux.debian.devel.kernel:29426
    Archived-At: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29426>
)

> And if there's already a maintainer, it's unlikely that he'll not
> accept patches from some new person debugging bug reports. But how to
> find people who will become good maintainers?

... {0}++

> > Unfortunately problems with individuals in big machine with bad people,
> > got randomly elected, can't be solved (IMHO). Even LKML's rule "patches
> > are welcome", that is very technical, thus good, doesn't work there.
> 
> Debian has it's own problems, Linux kernel development at least has a 
> working structure for getting decisions and regular releases.
> 
> > Finally, read the end of 2.6.21 release message ;)
> > 
> > > And a low hanging fruit to improve the release would be if you could 
> > > release one last -rc, wait for 48 hours, and then release either this 
> > > -rc unchanged as -final or another -rc (and wait another 48 hours).
> > > There were at least two different regressions people ran into in 2.6.21 
> > > who successfully tested -rc7.
> > 
> > What about Linus' tree is a development tree, Andrew's one is a "crazy
> > development one" (quoting Linus)?
> > 
> > What about open (web page still exists) position on bug manager in
> > Google?
> > 
> > What about *volunteers* working from both developer's and user's
> > sides? What about "release is a reward" for everybody?
> 
> People aren't that dumb that some empty words like "release is a reward"
> would change anything.

So, distro kernel teams, making .21 available to their user are ones?
That's simply pointless.

> > Balanced eco-system will oscillate. Be it .19(---++), .20(-++++),
> > .21(----+) *relese*. That's natural, unless pushed to extremes.
> > 
> > I think, i can trust Linus in it, and you are welcome too.
> > 
> > *tools*
> > 
> > That's why i'm talking about tools, and started to discuss them.
> > 
> > My last try: reportbug (there's "-ng" one also), Debian BTS.
> > 
> > Adrian, what can/must be done to adopt them? If not, your experience may
> > provide information about "why?" (re-consider my first mail about
> > background, please).
> 
> Bug tracking for the kernel is more or less working.
> The main problem is getting people to debug bug reports.
> 
> We need the main problem fixed, not a different tool in an area that is 
> not the main problem.

I see (this repetition). Maybe i've managed to express my POV, that it
can be seeing more cleanly.
____

Re: regression tracking (Re: Linux 2.6.21)

[Adrian Bunk]

On Sat, Jun 16, 2007 at 07:03:44AM +0200, Oleg Verych wrote:
>...
> On Sat, Jun 16, 2007 at 04:55:16AM +0200, Adrian Bunk wrote:
> > On Sat, Jun 16, 2007 at 03:32:36AM +0200, Oleg Verych wrote:
> > > On Sat, Jun 16, 2007 at 01:42:02AM +0200, Adrian Bunk wrote:
>...
> > > For example you feel, that you've wasted time. But after all, if you've
> > > came up with some kind of tool, everybody else could take it. No
> > > problems, useful ideas must and will evolve. But _ideally_ this must not be
> > > from ground zero every time. _Ideally_ from technical, not personal
> > > point of view ;).
> > 
> > As I expected, someone else has picked up regression tracking.
> > And other from what you claim, this did not require any kind of tool.
> 
> So you expected good, doing bad. ``Bad'' means bringing pointless flame
> about what everybody should do, without constructive approach like: "OK,
> i can't do it due to my POV{0}, useless manual work. Everybody willing to
> bring another way of dealing with it is welcome."
> 
> Your first reply:
> 
> "And it's not that Linus started developing the Linux kernel with writing
> git, the first 10 years of Linux development were without any SCM." {3}
> 
> to my note about, that you are not hurry anywhere, that after all that
> years of Open Source and Free Software development you are not trying
> to deal with such important thing like regression/bug tracking in
> ***organized way***, is rather pointless.

I am dealing with bug tracking in the kernel Bugzilla.
I did regression tracking for the kernel.
Michal is currently tracking regressions.
Andrew is doing an enormous amount of work in these areas.

You might not see it because you are not active in this area, but it is 
working in an organized way.

What is not working is getting all tracked bugs properly debugged.

> > > That's why people in Debian have started *team* maintenance with alioth. 
> > 
> > The problem for the Linux kernel is that for a better bug handling you'd 
> > need people willing to learn other people's code and to do the hard work 
> > of debugging bug reports.
> 
> ... {0}++
> 
> Do you know, for example, why i'm not making my "hacker's career"
> doing that?
> 
> 1. because i ended up with lynx, slrn, mutt, emacs-nox. Including
>    "zarro bogs found" kind of thing and other "userspace suck" problems.
> 2. i have no way to know if something *really* broken, unless it right
>    on my hardware
> 
> This all unlike Debian BTS using reportbug, where you have basic
> information, mbox format messages for easy "mutt -f", and other funny
> things, real maintainers aware of (i'm trying to know, learn about).
> 
> Thus organized, non brain-damaged way of reporting and tracking is the
> key issue.

Bugzilla is a usable tool for bug and regression reporting and tracking 
tracking.

I am using the Debian BTS since 8 years and I've used many Bugzillas.

Both are usable, and the real problem in kernel development is not 
really related to the question which tool to use for bug tracking.

>...
> > Talking about "team maintenance" sounds nice, but the problem in the 
> > kernel starts with code that has zero maintainers.
> 
> Floppy went pretty fine, before it was started to be maintained, and
> you know that. But you also told that unmaintained and not-working are
> different things.

unmaintained != unused
user != developer
worked != went pretty fine

Stuff can easily get broken and noone looks after bugs if there's no 
maintainer both knowing the code and willing to debug bugs.

The floppy driver is actually an example of code that has been broken 
quite often by patches simply because noone who completely understands 
this driver reviewed patches.

It somehow works and it might work for some years, but there is a 
problem.

>...
> I see (this repetition). Maybe i've managed to express my POV, that it
> can be seeing more cleanly.

IMHO your point of view is simply not related to the real current 
quality problems in kernel development.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: regression tracking (Re: Linux 2.6.21)

[Stefan Richter]

Oleg Verych wrote:
> On Sat, Jun 16, 2007 at 01:42:02AM +0200, Adrian Bunk wrote:
[...]
>> This means going through every single point in the regression list 
>> asking "Have we tried everything possible to solve this regression?". 
[...]
>> And a low hanging fruit to improve the release would be if you could 
>> release one last -rc, wait for 48 hours, and then release either this 
>> -rc unchanged as -final or another -rc (and wait another 48 hours).
>> There were at least two different regressions people ran into in 2.6.21 
>> who successfully tested -rc7.
> 
> What about Linus' tree is a development tree, Andrew's one is a "crazy
> development one" (quoting Linus)?
[...]

Linus also said that Andrew's tree is abused too often for broken stuff.

My goal for the little driver subsystem I'm maintaining is
  - everything that Andrew pulls from me builds and runs and doesn't
    introduce regressions to my and the submitters' knowledge.  I am
    slowly expanding my test procedures to catch things that fail that
    goal.
  - Everything that Linus pulls from me fulfills the above criteria
    and, in addition, had reasonable time and publication for test and
    review, depending on the kind of patch.

I had a few regressions in Linus' releases.  None of them were known
before release.  All of them were debugged and fixed rather soon after
report, AFAIR.

So what _I_ need is neither better regression tracking nor more manpower
for debugging of regression reports.  What I need is more own spare time
and equipment for tests, more own knowledge and experience, and more
people who run-time test -rc kernels or at least my subsystem updates on
top of older kernels.

    (Note, I'm talking only about regressions here, not old bugs.
    There my requirements are different; the by far most important
    one is more manpower for debugging and fixing.)

Well, if _other_ subsystems would get regressions in Linus' tree fixed
quicker, there might perhaps be more people who would consider to run
-rc kernels and would catch and report "my" regressions.

[Oleg, sorry that I too digressed from the subject of your thread, but
your remark about "[crazy] development tree"s caught my eye.  IMO people
should care for quality already in Andrew's tree --- more so than at the
moment.]

[Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too
few FireWire driver users run -rc kernels".]
-- 
Stefan Richter
-=====-=-=== -==- =----
http://arcgraph.de/sr/

Re: regression tracking (Re: Linux 2.6.21)

[Michal Piotrowski]

Hi Stefan,

On 16/06/07, Stefan Richter <stefanr@s5r6.in-berlin.de> wrote:
[..]
> Well, if _other_ subsystems would get regressions in Linus' tree fixed
> quicker, there might perhaps be more people who would consider to run
> -rc kernels and would catch and report "my" regressions.
[..]
>
> [Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too
> few FireWire driver users run -rc kernels".]

Rafael is working on translation of "Linux Kernel Tester's Guide"
(it's almost finished). I hope you will get more -rc testers.

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

Re: regression tracking (Re: Linux 2.6.21)

[Adrian Bunk]

On Sat, Jun 16, 2007 at 02:23:25PM +0200, Stefan Richter wrote:
>...
> [Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too
> few FireWire driver users run -rc kernels".]

Getting more people testing -rc kernels might be possible, and I don't 
think it would be too hard. And not only FireWire would benefit from 
this, remember e.g. that at least 2 out of the last 5 kernels Linus 
released contained filesystem corruption regressions.

The problem is that we aren't able to handle the many regression reports 
we get today, so asking for more testing and regression reports today 
would attack it at the wrong part of the chain.

Additionally, every reported and unhandled regression will frustrate the 
reporter - never forget that we have _many_ unhandled bug reports 
(including but not limited to regression reports) where the submitter 
spent much time and energy in writing a good bug report.

If we somehow gain the missing manpower for debugging regressions we can 
actively ask for more testing. Missing manpower (of people knowing some 
part of the kernel well) for debugging bug reports is IMHO the one big 
source of quality problems in the Linux kernel. If we get this solved, 
things like getting more testers for -rc kernels will become low hanging 
fruits.

> Stefan Richter

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

[PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Michal Piotrowski]

Hi all,

Adrian Bunk pisze:
> On Sat, Jun 16, 2007 at 02:23:25PM +0200, Stefan Richter wrote:
>> ...
>> [Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too
>> few FireWire driver users run -rc kernels".]
> 
> Getting more people testing -rc kernels might be possible, and I don't 
> think it would be too hard. And not only FireWire would benefit from 
> this, remember e.g. that at least 2 out of the last 5 kernels Linus 
> released contained filesystem corruption regressions.
> 
> The problem is that we aren't able to handle the many regression reports 
> we get today, so asking for more testing and regression reports today 
> would attack it at the wrong part of the chain.
> 
> Additionally, every reported and unhandled regression will frustrate the 
> reporter - never forget that we have _many_ unhandled bug reports 
> (including but not limited to regression reports) where the submitter 
> spent much time and energy in writing a good bug report.
> 
> If we somehow gain the missing manpower for debugging regressions we can 
> actively ask for more testing. Missing manpower (of people knowing some 
> part of the kernel well) for debugging bug reports is IMHO the one big 
> source of quality problems in the Linux kernel. If we get this solved, 
> things like getting more testers for -rc kernels will become low hanging 
> fruits.

Adrian, I agree with _all_ your points.

I bet that developers will hate me for this.

Please consider for 2.6.23

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

Signed-off-by: Michal Piotrowski <michal.k.k.piotrowski@gmail.com>

--- linux-work-clean/Documentation/SubmitChecklist	2007-06-17 11:18:37.000000000 +0200
+++ linux-work/Documentation/SubmitChecklist	2007-06-17 11:29:26.000000000 +0200
@@ -90,3 +90,8 @@ kernel patches.
     patch style checker prior to submission (scripts/checkpatch.pl).
     You should be able to justify all violations that remain in
     your patch.
+
+
+
+If the patch introduces a new regression and this regression was not fixed
+in seven days, then the patch will be reverted.

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Andrew Morton]

On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:

> +If the patch introduces a new regression and this regression was not fixed
> +in seven days, then the patch will be reverted.

Those regressions where we know which patch caused them are the easy ones.
Often we don't know which patch (or even which subsystem merge) is at
fault.

I think.  How many of the present 2.6.22-rc regressions which you're
presently tracking have such a well-identified cause?

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Michal Piotrowski]

On 17/06/07, Andrew Morton <akpm@linux-foundation.org> wrote:
> On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:
>
> > +If the patch introduces a new regression and this regression was not fixed
> > +in seven days, then the patch will be reverted.
>
> Those regressions where we know which patch caused them are the easy ones.
> Often we don't know which patch (or even which subsystem merge) is at
> fault.
>
> I think.  How many of the present 2.6.22-rc regressions which you're
> presently tracking have such a well-identified cause?
>

Here lays the problem.

git-bisect is a killer app, people should start using it.

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Oleg Verych]

On Sun, Jun 17, 2007 at 12:22:26PM +0200, Michal Piotrowski wrote:
> On 17/06/07, Andrew Morton <akpm@linux-foundation.org> wrote:
> >On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski 
> ><michal.k.k.piotrowski@gmail.com> wrote:
> >
> >> +If the patch introduces a new regression and this regression was not 
> >fixed
> >> +in seven days, then the patch will be reverted.
> >
> >Those regressions where we know which patch caused them are the easy ones.
> >Often we don't know which patch (or even which subsystem merge) is at
> >fault.
> >
> >I think.  How many of the present 2.6.22-rc regressions which you're
> >presently tracking have such a well-identified cause?
> >
> 
> Here lays the problem.
> 
> git-bisect is a killer app, people should start using it.

It's OK _only_ in case of unknown, hard to find *hardware* bugs.

If you think it's "a good thing" for bad, untested by developer
code, then something is completely wrong.

And if there's no debugger in the mainline kernel, which is developer's
tool, then why do you think testers must stick with git-bisect, as their
debugger-like tool (bandwidth in most and time consuming in some cases)?

That's wrong if developers are tending to reply only one thing --
git-bisect.

If things are going to be that bad, then better to start dealing with the
cause, not consequences. In this situation requesting test-cases is a
better way, as it's going to influence developer as cause of potential
problems. If tests will show *hardware* side of problem, then, well some
parts may be not obvious, thus bisecting is a way to continue.

Sorry if i'm from the abnormally different side yet one more time.
____

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Rafael J. Wysocki]

On Sunday, 17 June 2007 13:47, Oleg Verych wrote:
> On Sun, Jun 17, 2007 at 12:22:26PM +0200, Michal Piotrowski wrote:
> > On 17/06/07, Andrew Morton <akpm@linux-foundation.org> wrote:
> > >On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski 
> > ><michal.k.k.piotrowski@gmail.com> wrote:
> > >
> > >> +If the patch introduces a new regression and this regression was not 
> > >fixed
> > >> +in seven days, then the patch will be reverted.
> > >
> > >Those regressions where we know which patch caused them are the easy ones.
> > >Often we don't know which patch (or even which subsystem merge) is at
> > >fault.
> > >
> > >I think.  How many of the present 2.6.22-rc regressions which you're
> > >presently tracking have such a well-identified cause?
> > >
> > 
> > Here lays the problem.
> > 
> > git-bisect is a killer app, people should start using it.
> 
> It's OK _only_ in case of unknown, hard to find *hardware* bugs.
> 
> If you think it's "a good thing" for bad, untested by developer
> code, then something is completely wrong.

Oh, I've just fixed two purely software bugs pointed out by binary searching
in the code that I'm sure has been tested, not only by its developers, but the
bugs only showed up in my configuration (on one out of four test boxes).

There are so many different kernel configurations possible that there's no way
a developer can test them all.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Oleg Verych]

On Sun, Jun 17, 2007 at 02:13:39PM +0200, Rafael J. Wysocki wrote:
> On Sunday, 17 June 2007 13:47, Oleg Verych wrote:
[]
> > It's OK _only_ in case of unknown, hard to find *hardware* bugs.
> > 
> > If you think it's "a good thing" for bad, untested by developer
> > code, then something is completely wrong.
> 
> Oh, I've just fixed two purely software bugs pointed out by binary searching
> in the code that I'm sure has been tested, not only by its developers, but the
> bugs only showed up in my configuration (on one out of four test boxes).
> 
> There are so many different kernel configurations possible that there's no way
> a developer can test them all.

With current state of affairs it's not only hard for developers, but
and for users: <20070221220520.GA20659@artselect.com>,
              <20070429230037.95120@gmx.net>

I'm trying to re-do some kbuild stuff, but i'm getting rather offensive
answers :( <1182020654.8176.398.camel@chaos>

(Even if i'm academic with free Internet, i doubt i even tried to
think to improve something, if i didn't have one, because i wouldn't knew
huge lkml traffic, problems, etc.)

Maybe i'm wrong. But reducing amount of traffic/files and ease of
(re-)configuration are not last things to be done for better testing.
All for speed of getting and compiling kernel. Latter for avoiding
bugs and noise due to inconsistent build configuration.

Finally again, bug-reporting and tracking tools, i've tried to discuss
are major problems out there I think it's plain easy and deal with. One
more example:

    <handler.s.C.117647526113388.transcript@bugs.debian.org>
    Xref: news.gmane.org gmane.linux.debian.devel.kernel:28095
    <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/28095>
____

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Adrian Bunk]

On Sun, Jun 17, 2007 at 04:24:30PM +0200, Oleg Verych wrote:
> On Sun, Jun 17, 2007 at 02:13:39PM +0200, Rafael J. Wysocki wrote:
> > On Sunday, 17 June 2007 13:47, Oleg Verych wrote:
> []
> > > It's OK _only_ in case of unknown, hard to find *hardware* bugs.
> > > 
> > > If you think it's "a good thing" for bad, untested by developer
> > > code, then something is completely wrong.
> > 
> > Oh, I've just fixed two purely software bugs pointed out by binary searching
> > in the code that I'm sure has been tested, not only by its developers, but the
> > bugs only showed up in my configuration (on one out of four test boxes).
> > 
> > There are so many different kernel configurations possible that there's no way
> > a developer can test them all.
> 
> With current state of affairs it's not only hard for developers, but
> and for users: <20070221220520.GA20659@artselect.com>,
>               <20070429230037.95120@gmx.net>

Uwe has an attitude that made many people (including Linus himself) 
set their mail filters to deliver his emails directly to /dev/null.

Parts of the contents of his emails were usable including usable 
regression reports - but the way he treats people simply disqualified 
him.

> I'm trying to re-do some kbuild stuff, but i'm getting rather offensive
> answers :( <1182020654.8176.398.camel@chaos>
>...

I'm not seeing anything in Thomas' email that could be considered 
offensive. He told you in a technical way why he disagrees with you.

If you call this email "rather offensive", you should _really_ 
unsubscribe from lkml (or even any Debian mailing lists). And this is 
not meant against you, it's simply that for the standards of lkml there 
is nothing offensive in this email.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[david]

On Sun, 17 Jun 2007, Oleg Verych wrote:

> On Sun, Jun 17, 2007 at 12:22:26PM +0200, Michal Piotrowski wrote:
>> On 17/06/07, Andrew Morton <akpm@linux-foundation.org> wrote:
>>> On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski
>>> <michal.k.k.piotrowski@gmail.com> wrote:
>>>
>>>> +If the patch introduces a new regression and this regression was not
>>> fixed
>>>> +in seven days, then the patch will be reverted.
>>>
>>> Those regressions where we know which patch caused them are the easy ones.
>>> Often we don't know which patch (or even which subsystem merge) is at
>>> fault.
>>>
>>> I think.  How many of the present 2.6.22-rc regressions which you're
>>> presently tracking have such a well-identified cause?
>>>
>>
>> Here lays the problem.
>>
>> git-bisect is a killer app, people should start using it.
>
> It's OK _only_ in case of unknown, hard to find *hardware* bugs.
>
> If you think it's "a good thing" for bad, untested by developer
> code, then something is completely wrong.
>
> And if there's no debugger in the mainline kernel, which is developer's
> tool, then why do you think testers must stick with git-bisect, as their
> debugger-like tool (bandwidth in most and time consuming in some cases)?
>
> That's wrong if developers are tending to reply only one thing --
> git-bisect.
>
> If things are going to be that bad, then better to start dealing with the
> cause, not consequences. In this situation requesting test-cases is a
> better way, as it's going to influence developer as cause of potential
> problems. If tests will show *hardware* side of problem, then, well some
> parts may be not obvious, thus bisecting is a way to continue.

most people who report bugs don't know enough about what's actually going 
wrong to be able to write a test case (those that do can probably just 
write a patch to fix it). Along similar lines a debugger wouldn't be of 
much use either.

the fact that git-bisect doesn't require any knowledge other then 
knowledge the reporter has demonstrated that they already have (the 
ability to compile and install their own kernel) puts it within the reach 
of testers.

unfortunantly, as good as it is it can take a lot of effort, especially if 
the bug takes time to show up. it's not perfect, but it's a huge help.

and developers aren't always responding with 'do a bisect', sometimes they 
respond with 'yes, we know about that' or 'that sounds like X', so it's 
still worthwhile for people to report the problem first before going to 
the ffort of doing a bisect.

David Lang

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Oleg Verych]

On Sun, Jun 17, 2007 at 10:44:39AM -0700, david@lang.hm wrote:
> On Sun, 17 Jun 2007, Oleg Verych wrote:
[]
> >That's wrong if developers are tending to reply only one thing --
> >git-bisect.
> >
> >If things are going to be that bad, then better to start dealing with the
> >cause, not consequences. In this situation requesting test-cases is a
> >better way, as it's going to influence developer as cause of potential
> >problems. If tests will show *hardware* side of problem, then, well some
> >parts may be not obvious, thus bisecting is a way to continue.
> 
> most people who report bugs don't know enough about what's actually going 
> wrong to be able to write a test case (those that do can probably just 
> write a patch to fix it). Along similar lines a debugger wouldn't be of 
> much use either.

Sorry for my English. Requesting test cases from the developers of
course. Or at least results of some kind of testing, so people may run
and check them as well, if something is suspected. And this from my POV
leads again to organized way of filtering noise and collecting structured
information easily (yea, i think it's BTS with reportbug in Debian).{0}

> the fact that git-bisect doesn't require any knowledge other then 
> knowledge the reporter has demonstrated that they already have (the 
> ability to compile and install their own kernel) puts it within the reach 
> of testers.
> 
> unfortunantly, as good as it is it can take a lot of effort, especially if 
> the bug takes time to show up. it's not perfect, but it's a huge help.

I think, positive feedback from {0} to the LTP may improve that.
Especially, if things are in parts and easy to choose for testing.

> and developers aren't always responding with 'do a bisect', sometimes they 
> respond with 'yes, we know about that'

> or 'that sounds like X',

That two are _exactly_ what reportbug tool is doing. That's why i'm
talking about it. And i'm *no* wonder why developers are boring -- at
some points they might not handle the *noise*.

> so it's still worthwhile for people to report the problem first before
> going to the ffort of doing a bisect.
> 
> David Lang
__

Bits for Adrian.

*ML*

I *use* Gmane. I'm not subscribed (receiving e-mail to my mbox) to any ML,
except <kbuild@sf.net>.

Nearly every my e-mail here is with Gmane links. You seem ignored all of
them. As for me it's result of *your personal*, rather than technical
activity.

*offense*

I'm not talking about personal offense, you are seem thinking about, but
technical one. I.e. when possible benefit might be even more, than NOHZ
on x86 and a like[0], with much less effort. I still think, unless i will
develop or fail, that reducing traffic on one or two order of magnitude
is possible as well as improving kbuild/kconfig to reduce of the noise of
mis-configurations/tester's .config length. Discouraging that effort is
my source of offense.

(FYI Until Linus checked in my _RFT_ kbuild patches, i realized how
*many* people are willing to understand and try to test kbuild stuff.)

[0] I bet VGA, DRAM, HDD are far more power hungry room-heaters. Unless
    you can substantially lower frequency, you might have no benefit at
    all. Whana know how it's done in perfectly designed embedded
    MCU/CPU? Please, see for instance MSP430 from TI (i know, it uses
    SRAM, but i've asked to look on processor core design).
____

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Rafael J. Wysocki]

On Sunday, 17 June 2007 12:22, Michal Piotrowski wrote:
> On 17/06/07, Andrew Morton <akpm@linux-foundation.org> wrote:
> > On Sun, 17 Jun 2007 11:41:36 +0200 Michal Piotrowski <michal.k.k.piotrowski@gmail.com> wrote:
> >
> > > +If the patch introduces a new regression and this regression was not fixed
> > > +in seven days, then the patch will be reverted.
> >
> > Those regressions where we know which patch caused them are the easy ones.

Except when the bisection points us to a patch exposing a bug that is present
regardless (see http://lkml.org/lkml/2007/6/13/273 for example).

Besides, if a patch is merged before -rc1 as a bugfix, there are several
patches depending on it and only after -rc5 has been released we find out
that it breaks someone's system, then reverting it is not a solution, IMO.

> > Often we don't know which patch (or even which subsystem merge) is at
> > fault.
> >
> > I think.  How many of the present 2.6.22-rc regressions which you're
> > presently tracking have such a well-identified cause?
> >
> 
> Here lays the problem.
> 
> git-bisect is a killer app, people should start using it.

People should test _all_ of the -rc kernels and report problems.  Otherwise, we
may assume that there are no problems and go on.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Adrian Bunk]

On Sun, Jun 17, 2007 at 11:41:36AM +0200, Michal Piotrowski wrote:
> Hi all,
>
> Adrian Bunk pisze:
>> On Sat, Jun 16, 2007 at 02:23:25PM +0200, Stefan Richter wrote:
>>> ...
>>> [Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too
>>> few FireWire driver users run -rc kernels".]
>> Getting more people testing -rc kernels might be possible, and I don't 
>> think it would be too hard. And not only FireWire would benefit from this, 
>> remember e.g. that at least 2 out of the last 5 kernels Linus released 
>> contained filesystem corruption regressions.
>> The problem is that we aren't able to handle the many regression reports 
>> we get today, so asking for more testing and regression reports today 
>> would attack it at the wrong part of the chain.
>> Additionally, every reported and unhandled regression will frustrate the 
>> reporter - never forget that we have _many_ unhandled bug reports 
>> (including but not limited to regression reports) where the submitter 
>> spent much time and energy in writing a good bug report.
>> If we somehow gain the missing manpower for debugging regressions we can 
>> actively ask for more testing. Missing manpower (of people knowing some 
>> part of the kernel well) for debugging bug reports is IMHO the one big 
>> source of quality problems in the Linux kernel. If we get this solved, 
>> things like getting more testers for -rc kernels will become low hanging 
>> fruits.
>
> Adrian, I agree with _all_ your points.
>
> I bet that developers will hate me for this.
>
> Please consider for 2.6.23

Fine with me, but:

There are not so simple cases like big infrastructure patches with
20 other patches in the tree depending on it causing a regression, or 
even worse, a big infrastructure patch exposing a latent old bug in some 
completely different area of the kernel.

And we should be aware that reverting is only a workaround for the real 
problem which lies in our bug handling.

> Regards,
> Michal
>...

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Michal Piotrowski]

On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Sun, Jun 17, 2007 at 11:41:36AM +0200, Michal Piotrowski wrote:
> > Hi all,
> >
> > Adrian Bunk pisze:
> >> On Sat, Jun 16, 2007 at 02:23:25PM +0200, Stefan Richter wrote:
> >>> ...
> >>> [Adrian, I'm not saying "too few users run -rc kernels", I'm saying "too
> >>> few FireWire driver users run -rc kernels".]
> >> Getting more people testing -rc kernels might be possible, and I don't
> >> think it would be too hard. And not only FireWire would benefit from this,
> >> remember e.g. that at least 2 out of the last 5 kernels Linus released
> >> contained filesystem corruption regressions.
> >> The problem is that we aren't able to handle the many regression reports
> >> we get today, so asking for more testing and regression reports today
> >> would attack it at the wrong part of the chain.
> >> Additionally, every reported and unhandled regression will frustrate the
> >> reporter - never forget that we have _many_ unhandled bug reports
> >> (including but not limited to regression reports) where the submitter
> >> spent much time and energy in writing a good bug report.
> >> If we somehow gain the missing manpower for debugging regressions we can
> >> actively ask for more testing. Missing manpower (of people knowing some
> >> part of the kernel well) for debugging bug reports is IMHO the one big
> >> source of quality problems in the Linux kernel. If we get this solved,
> >> things like getting more testers for -rc kernels will become low hanging
> >> fruits.
> >
> > Adrian, I agree with _all_ your points.
> >
> > I bet that developers will hate me for this.
> >
> > Please consider for 2.6.23
>
> Fine with me, but:
>
> There are not so simple cases like big infrastructure patches with
> 20 other patches in the tree depending on it causing a regression, or
> even worse, a big infrastructure patch exposing a latent old bug in some
> completely different area of the kernel.

It is different case.

"If the patch introduces a new regression"

introduces != exposes an old bug

Removal of 20 patches will be painful, but sometimes you need to
"choose minor evil to prevent a greater one" [1].

> And we should be aware that reverting is only a workaround for the real
> problem which lies in our bug handling.
>
> > Regards,
> > Michal
> >...
>
> cu
> Adrian
>
> --
>
>        "Is there not promise of rain?" Ling Tan asked suddenly out
>         of the darkness. There had been need of rain for many days.
>        "Only a promise," Lao Er said.
>                                        Pearl S. Buck - Dragon Seed
>
>

Regards,
Michal

[1] the quote from "The Last Wish/Minor Evil" by Andrzej Sapkowski :)

-- 
LOG
http://www.stardust.webpages.pl/log/

Re: [PATCH] (Re: regression tracking (Re: Linux 2.6.21))

[Stefan Richter]

Michal Piotrowski wrote:
> "choose minor evil to prevent a greater one"

The measurement of "evil" is subjective.  That's why there are releases
with known regressions.
-- 
Stefan Richter
-=====-=-=== -==- =---=
http://arcgraph.de/sr/

How to improve the quality of the kernel?

[Adrian Bunk]

On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
>...
>> Fine with me, but:
>>
>> There are not so simple cases like big infrastructure patches with
>> 20 other patches in the tree depending on it causing a regression, or
>> even worse, a big infrastructure patch exposing a latent old bug in some
>> completely different area of the kernel.
>
> It is different case.
>
> "If the patch introduces a new regression"
>
> introduces != exposes an old bug

My remark was meant as a note "this sentence can't handle all 
regressions" (and for a user it doesn't matter whether a new 
regression is introduced or an old regression exposed).

It could be we simply agree on this one.  ;-)

> Removal of 20 patches will be painful, but sometimes you need to
> "choose minor evil to prevent a greater one" [1].
> 
>> And we should be aware that reverting is only a workaround for the real
>> problem which lies in our bug handling.
>...

And this is something I want to emphasize again.

How can we make any progress with the real problem and not only the 
symptoms?

There's now much money in the Linux market, and the kernel quality 
problems might result in real costs in the support of companies like
IBM, SGI, Redhat or Novell (plus it harms the Linux image which might 
result in lower revenues).

If [1] this is true, it might even pay pay off for them to each assign 
X man hours per month of experienced kernel developers to upstream 
kernel bug handling?

This is just a wild thought and it might be nonsense - better 
suggestions for solving our quality problems would be highly welcome...

cu
Adrian

[1] note that this is an "if"

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: How to improve the quality of the kernel?

[Michal Piotrowski]

On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> > On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> >...
> >> Fine with me, but:
> >>
> >> There are not so simple cases like big infrastructure patches with
> >> 20 other patches in the tree depending on it causing a regression, or
> >> even worse, a big infrastructure patch exposing a latent old bug in some
> >> completely different area of the kernel.
> >
> > It is different case.
> >
> > "If the patch introduces a new regression"
> >
> > introduces != exposes an old bug
>
> My remark was meant as a note "this sentence can't handle all
> regressions" (and for a user it doesn't matter whether a new
> regression is introduced or an old regression exposed).
>
> It could be we simply agree on this one.  ;-)
>
> > Removal of 20 patches will be painful, but sometimes you need to
> > "choose minor evil to prevent a greater one" [1].
> >
> >> And we should be aware that reverting is only a workaround for the real
> >> problem which lies in our bug handling.
> >...
>
> And this is something I want to emphasize again.
>
> How can we make any progress with the real problem and not only the
> symptoms?
>
> There's now much money in the Linux market, and the kernel quality
> problems might result in real costs in the support of companies like
> IBM, SGI, Redhat or Novell (plus it harms the Linux image which might
> result in lower revenues).
>
> If [1] this is true, it might even pay pay off for them to each assign
> X man hours per month of experienced kernel developers to upstream
> kernel bug handling?
>
> This is just a wild thought and it might be nonsense - better
> suggestions for solving our quality problems would be highly welcome...

Just one comment.

We don't try to recruit new skilled testers - it's a big problem.
Skilled tester can narrow down the problem, try to fix it etc. There
are too many "something between 2.6.10 and 2.6.21 broke my laptop"
reports...

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

Re: How to improve the quality of the kernel?

[Stefan Richter]

Adrian Bunk wrote:
>>> And we should be aware that reverting is only a workaround for the real
>>> problem which lies in our bug handling.
>> ...
> 
> And this is something I want to emphasize again.
> 
> How can we make any progress with the real problem and not only the 
> symptoms?
...

Perhaps make lists of

  - bug reports which never lead to any debug activity
    (no responsible person/team was found, or a seemingly person/team
    did not start to debug the report)

  - known regressions on release,

  - regressions that became known after release,

  - subsystems with notable backlogs of old bugs,

  - other categories?

Select typical cases from each categories, analyze what went wrong in
these cases, and try to identify practicable countermeasures.

Another approach:  Figure out areas where quality is exemplary and try
to draw conclusions for areas where quality is lacking.
-- 
Stefan Richter
-=====-=-=== -==- =---=
http://arcgraph.de/sr/

Re: How to improve the quality of the kernel?

[Michal Piotrowski]

On 17/06/07, Stefan Richter <stefanr@s5r6.in-berlin.de> wrote:
> Adrian Bunk wrote:
> >>> And we should be aware that reverting is only a workaround for the real
> >>> problem which lies in our bug handling.
> >> ...
> >
> > And this is something I want to emphasize again.
> >
> > How can we make any progress with the real problem and not only the
> > symptoms?
> ...
>
> Perhaps make lists of
>
>   - bug reports which never lead to any debug activity
>     (no responsible person/team was found, or a seemingly person/team
>     did not start to debug the report)
>
>   - known regressions on release,
>
>   - regressions that became known after release,
>
>   - subsystems with notable backlogs of old bugs,
>
>   - other categories?

It is unworkable in wiki.

There is a new regression field in bugzilla, but it is only the first
step to implement regression tracking feature.

>
> Select typical cases from each categories, analyze what went wrong in
> these cases, and try to identify practicable countermeasures.
>
> Another approach:  Figure out areas where quality is exemplary and try
> to draw conclusions for areas where quality is lacking.
> --
> Stefan Richter
> -=====-=-=== -==- =---=
> http://arcgraph.de/sr/
>

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

Re: How to improve the quality of the kernel?

[Adrian Bunk]

On Sun, Jun 17, 2007 at 06:26:55PM +0200, Stefan Richter wrote:
> Adrian Bunk wrote:
> >>> And we should be aware that reverting is only a workaround for the real
> >>> problem which lies in our bug handling.
> >> ...
> > 
> > And this is something I want to emphasize again.
> > 
> > How can we make any progress with the real problem and not only the 
> > symptoms?
> ...
> 
> Perhaps make lists of
> 
>   - bug reports which never lead to any debug activity
>     (no responsible person/team was found, or a seemingly person/team
>     did not start to debug the report)
> 
>   - known regressions on release,
> 
>   - regressions that became known after release,
> 
>   - subsystems with notable backlogs of old bugs,
> 
>   - other categories?
> 
> Select typical cases from each categories, analyze what went wrong in
> these cases, and try to identify practicable countermeasures.

No maintainer or no maintainer who is debugging bug reports is the 
major problem in all parts of your list.

> Another approach:  Figure out areas where quality is exemplary and try
> to draw conclusions for areas where quality is lacking.

ieee1394 has a maintainer who is looking after all bug reports he gets.

Conclusion: We need such maintainers for all parts of the kernel.

> Stefan Richter

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: How to improve the quality of the kernel?

[Stefan Richter]

Adrian Bunk wrote:
> On Sun, Jun 17, 2007 at 06:26:55PM +0200, Stefan Richter wrote:
>> Another approach:  Figure out areas where quality is exemplary and try
>> to draw conclusions for areas where quality is lacking.
> 
> ieee1394 has a maintainer who is looking after all bug reports he gets.

...but doesn't fix them all, and is usually slow with fixes.  He should
spend less time conversing on LKML.  :-)
-- 
Stefan Richter
-=====-=-=== -==- =---=
http://arcgraph.de/sr/

Re: How to improve the quality of the kernel?

[Natalie Protasevich]

On 6/17/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Sun, Jun 17, 2007 at 06:26:55PM +0200, Stefan Richter wrote:
> > Adrian Bunk wrote:
> > >>> And we should be aware that reverting is only a workaround for the real
> > >>> problem which lies in our bug handling.
> > >> ...
> > >
> > > And this is something I want to emphasize again.
> > >
> > > How can we make any progress with the real problem and not only the
> > > symptoms?
> > ...
> >
> > Perhaps make lists of
> >
> >   - bug reports which never lead to any debug activity
> >     (no responsible person/team was found, or a seemingly person/team
> >     did not start to debug the report)
> >
> >   - known regressions on release,
> >
> >   - regressions that became known after release,
> >
> >   - subsystems with notable backlogs of old bugs,
> >
> >   - other categories?
> >
> > Select typical cases from each categories, analyze what went wrong in
> > these cases, and try to identify practicable countermeasures.
>
> No maintainer or no maintainer who is debugging bug reports is the
> major problem in all parts of your list.
>
> > Another approach:  Figure out areas where quality is exemplary and try
> > to draw conclusions for areas where quality is lacking.
>
> ieee1394 has a maintainer who is looking after all bug reports he gets.
>
> Conclusion: We need such maintainers for all parts of the kernel.
>

I noticed some areas are well maintained because there is an awesome
maintainer, or good and well coordinated team - and this is mostly in
the "fun" areas ;) But there are "boring" areas that are about to be
deprecated or no new development expected etc. It will be hard to get
a dedicated person to take care of such. How about having people on
rotation, or jury duty so to speak - for a period of time (completely
voluntary!) Nice stats on the report about contributions in non-native
areas for a developer would be great accomplishment and also good
chance to look into other things! Besides, this way "old parts" will
get attention to be be revised and re-implemented sooner. And we can
post "Temp maintainer needed" list...

--Natalie

> > Stefan Richter
>
> cu
> Adrian

Re: How to improve the quality of the kernel?

[Markus Rechberger]

On 6/17/07, Natalie Protasevich <protasnb@gmail.com> wrote:
> On 6/17/07, Adrian Bunk <bunk@stusta.de> wrote:
> > On Sun, Jun 17, 2007 at 06:26:55PM +0200, Stefan Richter wrote:
> > > Adrian Bunk wrote:
> > > >>> And we should be aware that reverting is only a workaround for the
> real
> > > >>> problem which lies in our bug handling.
> > > >> ...
> > > >
> > > > And this is something I want to emphasize again.
> > > >
> > > > How can we make any progress with the real problem and not only the
> > > > symptoms?
> > > ...
> > >
> > > Perhaps make lists of
> > >
> > >   - bug reports which never lead to any debug activity
> > >     (no responsible person/team was found, or a seemingly person/team
> > >     did not start to debug the report)
> > >
> > >   - known regressions on release,
> > >
> > >   - regressions that became known after release,
> > >
> > >   - subsystems with notable backlogs of old bugs,
> > >
> > >   - other categories?
> > >
> > > Select typical cases from each categories, analyze what went wrong in
> > > these cases, and try to identify practicable countermeasures.
> >
> > No maintainer or no maintainer who is debugging bug reports is the
> > major problem in all parts of your list.
> >
> > > Another approach:  Figure out areas where quality is exemplary and try
> > > to draw conclusions for areas where quality is lacking.
> >
> > ieee1394 has a maintainer who is looking after all bug reports he gets.
> >
> > Conclusion: We need such maintainers for all parts of the kernel.
> >
>
> I noticed some areas are well maintained because there is an awesome
> maintainer, or good and well coordinated team - and this is mostly in
> the "fun" areas ;) But there are "boring" areas that are about to be
> deprecated or no new development expected etc. It will be hard to get
> a dedicated person to take care of such. How about having people on
> rotation, or jury duty so to speak - for a period of time (completely
> voluntary!) Nice stats on the report about contributions in non-native
> areas for a developer would be great accomplishment and also good
> chance to look into other things! Besides, this way "old parts" will
> get attention to be be revised and re-implemented sooner. And we can
> post "Temp maintainer needed" list...
>

I'd vote for that, I've seen alot very bad code already within some
subsystems and critical problems which just have been ignored by some
maintainers.
It mostly helps if some volunteers read through existing code and
state out their considerations about implementations which they don't
like.

I just grep'ed some examples I noticed (note I do not want to jump
onto someone's toe here, just give some examples):

(sn9c102_ov7660.c)
...
        err += sn9c102_i2c_write(cam, 0x12, 0x80);
        err += sn9c102_i2c_write(cam, 0x11, 0x09);
        err += sn9c102_i2c_write(cam, 0x00, 0x0A);
        err += sn9c102_i2c_write(cam, 0x01, 0x80);
        err += sn9c102_i2c_write(cam, 0x02, 0x80);
        err += sn9c102_i2c_write(cam, 0x03, 0x00);
... (around 150 lines directly after each other doing such writes and
adding error values to a variable, I don't understand why someone
should add the errors but continue with sending 150 more updates, how
about one write failed but others succeeded for any reason)

(tvp5150.c)
static int tvp5150_read(struct i2c_client *c, unsigned char addr)
{
        unsigned char buffer[1];
        int rc;

        buffer[0] = addr;
        if (1 != (rc = i2c_master_send(c, buffer, 1)))
                tvp5150_dbg(0, "i2c i/o error: rc == %d (should be 1)\n", rc);

        msleep(10);

        if (1 != (rc = i2c_master_recv(c, buffer, 1)))
                tvp5150_dbg(0, "i2c i/o error: rc == %d (should be 1)\n", rc);

        tvp5150_dbg(2, "tvp5150: read 0x%02x = 0x%02x\n", addr, buffer[0]);

        return (buffer[0]);
}

(i2c issues within some driver)
        /* This code detects calls by card attach_inform */
        if (NULL == t->i2c.dev.driver) {
                tuner_dbg ("tuner 0x%02x: called during i2c_client
register by adapter's attach_inform\n", c->addr);

                return;
        }
... that code doesn't even work anymore since the i2c.dev.driver is
always initialized.

just reading through it and cleaning up some code isn't so difficult
and can be done by many people - if they're allowed/wanted to do so.

Markus

Re: How to improve the quality of the kernel?

[Stefan Richter]

Markus Rechberger wrote:
> just reading through it and cleaning up some code isn't so difficult
> and can be done by many people -

Doing cleanups is a good way to get into the matter, to become able to
eventually fix bugs of the difficult type.

> if they're allowed/wanted to do so.

Everybody is allowed to submit.  But there is a certain degree of both
persistence and adaptability required to get one's first submissions
upstream.  However, these qualities are also required to fix difficult bugs.
-- 
Stefan Richter
-=====-=-=== -==- =-==-
http://arcgraph.de/sr/

Re: How to improve the quality of the kernel?

[Oleg Verych]

On Fri, Jun 22, 2007 at 04:19:34PM +0200, Stefan Richter wrote:
> Markus Rechberger wrote:
> > just reading through it and cleaning up some code isn't so difficult
> > and can be done by many people -
> 
> Doing cleanups is a good way to get into the matter, to become able to
> eventually fix bugs of the difficult type.
> 
> > if they're allowed/wanted to do so.
> 
> Everybody is allowed to submit.  But there is a certain degree of both
> persistence and adaptability required to get one's first submissions
> upstream.  However, these qualities are also required to fix difficult bugs.

Deja-kernel.

Just two messages:
<http://permalink.gmane.org/gmane.linux.debian.devel.general/116453>
<http://permalink.gmane.org/gmane.linux.debian.devel.general/116463>

Tell me, i'm wrong, if similar thing cannot be implemented here.
Again, key word is _tracking_ system...

Just trying attract attention, that time of ignorance and manual work
must be ended. There must be new time, time of *tracking*, *counting*
opinions and any kernel work anybody want to contribute. I just got bored
after repeatings like, not funny work, code, etc. The manager, who will
do that not funny work is automated tracking system. Based on e-mail,
with additional tools, like

* ``reportbug''-- reporting (imroved REPORTING-BUGS,
  EVERY-WORK-IS-APPRECIATED-THANK-YOU)

* ``bts''-- command line interface, etc.

I want to change it, and i will try to work on that. Important thing is
-- to be in the corner *alone*, even with good, open source example system 
as Debian BTS is not gonna work.

WRT this, opinions and doings of people in this thread, who spend in
Linux development much more time, than i, just counter productive (fine,
fine but i have a right to have different, wrong opinion on that :).

--
  Frenzy
-o--=O`C
 #oo'L O
<___=E M

Re: How to improve the quality of the kernel?

[Rafael J. Wysocki]

On Sunday, 17 June 2007 16:29, Adrian Bunk wrote:
> On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> > On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> >...
> >> Fine with me, but:
> >>
> >> There are not so simple cases like big infrastructure patches with
> >> 20 other patches in the tree depending on it causing a regression, or
> >> even worse, a big infrastructure patch exposing a latent old bug in some
> >> completely different area of the kernel.
> >
> > It is different case.
> >
> > "If the patch introduces a new regression"
> >
> > introduces != exposes an old bug
> 
> My remark was meant as a note "this sentence can't handle all 
> regressions" (and for a user it doesn't matter whether a new 
> regression is introduced or an old regression exposed).
> 
> It could be we simply agree on this one.  ;-)
> 
> > Removal of 20 patches will be painful, but sometimes you need to
> > "choose minor evil to prevent a greater one" [1].
> > 
> >> And we should be aware that reverting is only a workaround for the real
> >> problem which lies in our bug handling.
> >...
> 
> And this is something I want to emphasize again.
> 
> How can we make any progress with the real problem and not only the 
> symptoms?

I think that we can handle bug reports like we handle modifications of code.

Namely, for each subsystem there can be a person (or a team) responsible
for handling bugs, by which I don't mean fixing them, but directing bug reports
at the right developers or subsystem maintainers, following the history of each
bug report etc.  [Of course, these people can choose to use the bugzilla or any
other bug tracking system they want, as long as it works for them.]

The email addresses of these people should be known (and even documented),
so that everyone can notify them if need be and so that it's clear who should
handle given bug reports.

Just an idea. :-)

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: How to improve the quality of the kernel?

[Natalie Protasevich]

On 6/17/07, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> On Sunday, 17 June 2007 16:29, Adrian Bunk wrote:
> > On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> > > On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> > >...
> > >> Fine with me, but:
> > >>
> > >> There are not so simple cases like big infrastructure patches with
> > >> 20 other patches in the tree depending on it causing a regression, or
> > >> even worse, a big infrastructure patch exposing a latent old bug in some
> > >> completely different area of the kernel.
> > >
> > > It is different case.
> > >
> > > "If the patch introduces a new regression"
> > >
> > > introduces != exposes an old bug
> >
> > My remark was meant as a note "this sentence can't handle all
> > regressions" (and for a user it doesn't matter whether a new
> > regression is introduced or an old regression exposed).
> >
> > It could be we simply agree on this one.  ;-)
> >
> > > Removal of 20 patches will be painful, but sometimes you need to
> > > "choose minor evil to prevent a greater one" [1].
> > >
> > >> And we should be aware that reverting is only a workaround for the real
> > >> problem which lies in our bug handling.
> > >...
> >
> > And this is something I want to emphasize again.
> >
> > How can we make any progress with the real problem and not only the
> > symptoms?
>
> I think that we can handle bug reports like we handle modifications of code.
>
> Namely, for each subsystem there can be a person (or a team) responsible
> for handling bugs, by which I don't mean fixing them, but directing bug reports
> at the right developers or subsystem maintainers, following the history of each
> bug report etc.  [Of course, these people can choose to use the bugzilla or any
> other bug tracking system they want, as long as it works for them.]
>
> The email addresses of these people should be known (and even documented),
> so that everyone can notify them if need be and so that it's clear who should
> handle given bug reports.
>
> Just an idea. :-)
>

Those are very good ideas indeed. The whole development process came
to the point when all realize that something needs to be done for the
team to balance out new development and old and recent unresolved
issues that are piling up...

I've looked through a number of bugzillas recently and here is my
scoop on shortcomings and some ideas. I am not sure how realistic they
are, probably might fall into "wishful thinking" category.

The way bugs get tracked and resolved is definitely a "no guarantee",
and main reasons are:
  not enough time for a maintainer to attend to them all
  nobody else (except at best very few busy people) knows about
majority of the problems. Andrew and Adrian and Michal post the most
pressing ones. But there are many many smaller ones that are not
assessed and not being taken care of.
  many problems are not easily reproducible and not easy to verify
because there is no identical system, motherboard, application, etc.
in case if reporter doesn't stick around until the end of the bug's
life.

Maybe along with bugzilla there should be another tracking tool - for
resources and systems that are available to individual developers.
Someone might have same or similar system to verify fixes in case if
the reporter disappears or "the system is gone now". Requests for
specific hardware can be automatically generated by the bugzilla say.
Those can be posted once in a while for everyone to see and chip in
and acknowledge if they happen to have such hardware and able to run a
quick test to at least verify the patch. Statistically, such need
doesn't happen often for each type of hardware, so it shouldn't be a
big burden for owners.

Besides, the database and resources can be useful for developers who
want to test their new patches on variety of hardware. This might
prevent future regressions which often caused by lack of testing as we
all know.

There are problems that require more research and thinking such as
implementing new features or redesigning old ones. Those should be
posted as a wish list I think as invitation for constructive
discussion and as possible project for takers. They also can be
extracted from bugzilla, I ran into several ones in intermission state
like that.

And finally, the most wishful would probably be collecting test tools
that are written by and can be reused by and available to developers.
It's normally possible to find something on the Internet or write a
quick test program - and probably lots of people end up writing little
programs to allocate shared memory and exercise it in certain way or
some affinity tool etc. But sometimes people come up with pretty
elaborate ones - why won't we attempt to sort out those test programs,
have them contributed (and maybe not code reviewed! - just as is, take
it or leave it :) and have them handy for better and faster bug
fixing/testing. And again - there are times we wish for such tool or
emulator and don't have spare cycles, so those type of requests for
custom test scripts and programs can also be posted.

I also had on mind what to do about maintainers and project teams and
alternative contacts who can handle issues on a particular module or
subsystem. Probably list or database of volunteers can be arranged,
this is something that is really needed. I can relate after trying to
get hold of alternative people myself...

Regards,
--Natalie

Re: How to improve the quality of the kernel?

[Rafael J. Wysocki]

On Sunday, 17 June 2007 19:42, Natalie Protasevich wrote:
> On 6/17/07, Rafael J. Wysocki <rjw@sisk.pl> wrote:
> > On Sunday, 17 June 2007 16:29, Adrian Bunk wrote:
> > > On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> > > > On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> > > >...
> > > >> Fine with me, but:
> > > >>
> > > >> There are not so simple cases like big infrastructure patches with
> > > >> 20 other patches in the tree depending on it causing a regression, or
> > > >> even worse, a big infrastructure patch exposing a latent old bug in some
> > > >> completely different area of the kernel.
> > > >
> > > > It is different case.
> > > >
> > > > "If the patch introduces a new regression"
> > > >
> > > > introduces != exposes an old bug
> > >
> > > My remark was meant as a note "this sentence can't handle all
> > > regressions" (and for a user it doesn't matter whether a new
> > > regression is introduced or an old regression exposed).
> > >
> > > It could be we simply agree on this one.  ;-)
> > >
> > > > Removal of 20 patches will be painful, but sometimes you need to
> > > > "choose minor evil to prevent a greater one" [1].
> > > >
> > > >> And we should be aware that reverting is only a workaround for the real
> > > >> problem which lies in our bug handling.
> > > >...
> > >
> > > And this is something I want to emphasize again.
> > >
> > > How can we make any progress with the real problem and not only the
> > > symptoms?
> >
> > I think that we can handle bug reports like we handle modifications of code.
> >
> > Namely, for each subsystem there can be a person (or a team) responsible
> > for handling bugs, by which I don't mean fixing them, but directing bug reports
> > at the right developers or subsystem maintainers, following the history of each
> > bug report etc.  [Of course, these people can choose to use the bugzilla or any
> > other bug tracking system they want, as long as it works for them.]
> >
> > The email addresses of these people should be known (and even documented),
> > so that everyone can notify them if need be and so that it's clear who should
> > handle given bug reports.
> >
> > Just an idea. :-)
> >
> 
> Those are very good ideas indeed. The whole development process came
> to the point when all realize that something needs to be done for the
> team to balance out new development and old and recent unresolved
> issues that are piling up...
> 
> I've looked through a number of bugzillas recently and here is my
> scoop on shortcomings and some ideas. I am not sure how realistic they
> are, probably might fall into "wishful thinking" category.
> 
> The way bugs get tracked and resolved is definitely a "no guarantee",
> and main reasons are:
>   not enough time for a maintainer to attend to them all
>   nobody else (except at best very few busy people) knows about
> majority of the problems. Andrew and Adrian and Michal post the most
> pressing ones. But there are many many smaller ones that are not
> assessed and not being taken care of.
>   many problems are not easily reproducible and not easy to verify
> because there is no identical system, motherboard, application, etc.
> in case if reporter doesn't stick around until the end of the bug's
> life.

I agree.  In addition, there is only a limited time window in which it makes
sense to debug given problem before the kernel changes too much (that of
course depends on the subsystem in question).

> Maybe along with bugzilla there should be another tracking tool - for
> resources and systems that are available to individual developers.

Yes, that would be very nice to have.

> Someone might have same or similar system to verify fixes in case if
> the reporter disappears or "the system is gone now". Requests for
> specific hardware can be automatically generated by the bugzilla say.
> Those can be posted once in a while for everyone to see and chip in
> and acknowledge if they happen to have such hardware and able to run a
> quick test to at least verify the patch. Statistically, such need
> doesn't happen often for each type of hardware, so it shouldn't be a
> big burden for owners.
> 
> Besides, the database and resources can be useful for developers who
> want to test their new patches on variety of hardware. This might
> prevent future regressions which often caused by lack of testing as we
> all know.

For that, I think, some "professional testers" would be needed ...

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: How to improve the quality of the kernel?

[Adrian Bunk]

On Sun, Jun 17, 2007 at 07:31:01PM +0200, Rafael J. Wysocki wrote:
> On Sunday, 17 June 2007 16:29, Adrian Bunk wrote:
> > On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> > > On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> > >...
> > >> Fine with me, but:
> > >>
> > >> There are not so simple cases like big infrastructure patches with
> > >> 20 other patches in the tree depending on it causing a regression, or
> > >> even worse, a big infrastructure patch exposing a latent old bug in some
> > >> completely different area of the kernel.
> > >
> > > It is different case.
> > >
> > > "If the patch introduces a new regression"
> > >
> > > introduces != exposes an old bug
> > 
> > My remark was meant as a note "this sentence can't handle all 
> > regressions" (and for a user it doesn't matter whether a new 
> > regression is introduced or an old regression exposed).
> > 
> > It could be we simply agree on this one.  ;-)
> > 
> > > Removal of 20 patches will be painful, but sometimes you need to
> > > "choose minor evil to prevent a greater one" [1].
> > > 
> > >> And we should be aware that reverting is only a workaround for the real
> > >> problem which lies in our bug handling.
> > >...
> > 
> > And this is something I want to emphasize again.
> > 
> > How can we make any progress with the real problem and not only the 
> > symptoms?
> 
> I think that we can handle bug reports like we handle modifications of code.
> 
> Namely, for each subsystem there can be a person (or a team) responsible
> for handling bugs, by which I don't mean fixing them, but directing bug reports
> at the right developers or subsystem maintainers, following the history of each
> bug report etc.  [Of course, these people can choose to use the bugzilla or any
> other bug tracking system they want, as long as it works for them.]
> 
> The email addresses of these people should be known (and even documented),
> so that everyone can notify them if need be and so that it's clear who should
> handle given bug reports.

Currently, these people are "Andrew Morton" and the addresses are
linux-kernel@vger.kernel.org and http://bugzilla.kernel.org/ - and this 
part is working.

Although there is room for improvement in this area, the problem in the 
pipeline is really to find developers who know the code in question and 
who are willing to debug bug reports.

There are unmaintained parts of the kernel.

And there are parts of the kernel where the maintainers are developing 
code, reviewing code and handling patches but are not willing or simply 
not capable of looking at bug reports. That's not against these people 
and they might do great work, but then there's simply an additional 
person missing who would be willing to learn the subsystem in question 
and handle bug reports.

All bug handling becomes moot and every request for more information 
from the submitter a waste of time if there's noone available for 
looking deeper into a bug.

> Just an idea. :-)
> 
> Greetings,
> Rafael

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: How to improve the quality of the kernel?

[Bartlomiej Zolnierkiewicz]

Hi,

On Sunday 17 June 2007, Adrian Bunk wrote:
> On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> > On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> >...
> >> Fine with me, but:
> >>
> >> There are not so simple cases like big infrastructure patches with
> >> 20 other patches in the tree depending on it causing a regression, or
> >> even worse, a big infrastructure patch exposing a latent old bug in some
> >> completely different area of the kernel.
> >
> > It is different case.
> >
> > "If the patch introduces a new regression"
> >
> > introduces != exposes an old bug
> 
> My remark was meant as a note "this sentence can't handle all 
> regressions" (and for a user it doesn't matter whether a new 
> regression is introduced or an old regression exposed).
> 
> It could be we simply agree on this one.  ;-)
> 
> > Removal of 20 patches will be painful, but sometimes you need to
> > "choose minor evil to prevent a greater one" [1].
> > 
> >> And we should be aware that reverting is only a workaround for the real
> >> problem which lies in our bug handling.
> >...
> 
> And this is something I want to emphasize again.
> 
> How can we make any progress with the real problem and not only the 
> symptoms?
> 
> There's now much money in the Linux market, and the kernel quality 
> problems might result in real costs in the support of companies like
> IBM, SGI, Redhat or Novell (plus it harms the Linux image which might 
> result in lower revenues).
> 
> If [1] this is true, it might even pay pay off for them to each assign 
> X man hours per month of experienced kernel developers to upstream 
> kernel bug handling?
> 
> This is just a wild thought and it might be nonsense - better 
> suggestions for solving our quality problems would be highly welcome...

IMO we should concentrate more on preventing regressions than on fixing them.
In the long-term preventing bugs is cheaper than fixing them afterwards.

First let me tell you all a little story...

Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs
in it (in other words I potentially prevented three regressions).  I also
asked for more thorough verification of the patch as I suspected that it may
have more problems.  The author fixed the issues and replied that he hasn't
done the full verification yet but he doesn't suspect any problems...

Fast forward...

Year later I discover that the final version of the patch hit the mainline.
I don't remember ever seeing the final version in my mailbox (there are no
cc: lines in the patch description) and I saw that I'm not credited in the
patch description.  However the worse part is that it seems that the full
verification has never been done.  The result?   Regression in the release
kernel (exactly the issue that I was worried about) which required three
patches and over a month to be fixed completely.  It seems that a year
was not enough to get this ~70k _cleanup_ patch fully verified and tested
(it hit -mm soon before being merged)...

>From reviewer's POV: I have invested my time into review, discovered real
issues and as a reward I got no credit et all and extra frustration from the
fact that part of my review was forgotten/ignored (the part which resulted in
real regression in the release kernel)...  Oh and in the past the said
developer has already been asked (politely in private message) to pay more
attention to his changes (after I silently fixed some other regression caused
by his other patch).

But wait there is more, I happend to be the maintainer of the subsystem which
got directly hit by the issue and I was getting bugreports from the users about
the problem...  :-)

It wasn't my first/last bad experience as a reviewer... finally I just gave up
on reviewing other people patches unless they are stricly for IDE subsystem.

The moral of the story is that currently it just doesn't pay off to do
code reviews.  From personal POV it pays much more to wait until buggy patch
hits the mainline and then fix the issues yourself (at least you will get
some credit).  To change this we should put more ephasize on the importance
of code reviews by "rewarding" people investing their time into reviews
and "rewarding" developers/maintainers taking reviews seriously.

We should credit reviewers more, sometimes it takes more time/knowledge to
review the patch than to make it so getting near to zero credit for review
doesn't sound too attractive.  Hmm, wait it can be worse - your review
may be ignored... ;-)

>From my side I think I'll start adding less formal "Reviewed-by" to IDE
patches even if the review resulted in no issues being found (in additon to
explicit "Acked-by" tags and crediting people for finding real issues - which
I currently always do as a way for showing my appreciation for their work).

I also encourage other maintainers/developers to pay more attention to
adding "Acked-by"/"Reviewed-by" tags and crediting reviewers.  I hope
that maintainers will promote changes that have been reviewed by others
by giving them priority over other ones (if the changes are on more-or-less
the same importance level of course, you get the idea).

Now what to do with people who ignore reviews and/or have rather high
regressions/patches ratio?

I think that we should have info about regressions integrated into SCM,
i.e. in git we should have optional "fixes-commit" tag and we should be
able to do some reverse data colletion.   This feature combined with
"Author:" info after some time should give us some very interesting
statistics (Top Ten "Regressors").  It wouldn't be ideal (ie. we need some
patches threshold to filter out people with 1 patch and >= 1 regression(s),
we need to remember that some code areas are more difficult than the others
and that patches are not equal per se etc.) however I believe than making it
into Top Ten "Regressors" should give the winners some motivation to improve
their work ethic.  Well, in the worst case we would just get some extra
trivial/documentation patches. ;-)

Sorry for a bit chaotic mail but I hope that message is clear.

Thanks,
Bart

Re: How to improve the quality of the kernel?

[Andrew Morton]

On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> wrote:

> 
> 
> IMO we should concentrate more on preventing regressions than on fixing them.
> In the long-term preventing bugs is cheaper than fixing them afterwards.
> 
> First let me tell you all a little story...
> 
> Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs
> in it (in other words I potentially prevented three regressions).  I also
> asked for more thorough verification of the patch as I suspected that it may
> have more problems.  The author fixed the issues and replied that he hasn't
> done the full verification yet but he doesn't suspect any problems...
> 
> Fast forward...
> 
> Year later I discover that the final version of the patch hit the mainline.
> I don't remember ever seeing the final version in my mailbox (there are no
> cc: lines in the patch description) and I saw that I'm not credited in the
> patch description.  However the worse part is that it seems that the full
> verification has never been done.  The result?   Regression in the release
> kernel (exactly the issue that I was worried about) which required three
> patches and over a month to be fixed completely.  It seems that a year
> was not enough to get this ~70k _cleanup_ patch fully verified and tested
> (it hit -mm soon before being merged)...

crap.  Commit ID, please ;)

> >From reviewer's POV: I have invested my time into review, discovered real
> issues and as a reward I got no credit et all and extra frustration from the
> fact that part of my review was forgotten/ignored (the part which resulted in
> real regression in the release kernel)...  Oh and in the past the said
> developer has already been asked (politely in private message) to pay more
> attention to his changes (after I silently fixed some other regression caused
> by his other patch).
> 
> But wait there is more, I happend to be the maintainer of the subsystem which
> got directly hit by the issue and I was getting bugreports from the users about
> the problem...  :-)
> 
> It wasn't my first/last bad experience as a reviewer... finally I just gave up
> on reviewing other people patches unless they are stricly for IDE subsystem.
> 
> The moral of the story is that currently it just doesn't pay off to do
> code reviews.

I dunno.  I suspect (hope) that this was an exceptional case, hence one
should not draw general conclusions from it.  It certainly sounds very bad.

>  From personal POV it pays much more to wait until buggy patch
> hits the mainline and then fix the issues yourself (at least you will get
> some credit).  To change this we should put more ephasize on the importance
> of code reviews by "rewarding" people investing their time into reviews
> and "rewarding" developers/maintainers taking reviews seriously.
> 
> We should credit reviewers more, sometimes it takes more time/knowledge to
> review the patch than to make it so getting near to zero credit for review
> doesn't sound too attractive.  Hmm, wait it can be worse - your review
> may be ignored... ;-)
> 
> >From my side I think I'll start adding less formal "Reviewed-by" to IDE
> patches even if the review resulted in no issues being found (in additon to
> explicit "Acked-by" tags and crediting people for finding real issues - which
> I currently always do as a way for showing my appreciation for their work).

yup, Reviewed-by: is good and I do think we should start adopting it,
although I haven't thought through exactly how.

On my darker days I consider treating a Reviewed-by: as a prerequisite for
merging.  I suspect that would really get the feathers flying.

> I also encourage other maintainers/developers to pay more attention to
> adding "Acked-by"/"Reviewed-by" tags and crediting reviewers.  I hope
> that maintainers will promote changes that have been reviewed by others
> by giving them priority over other ones (if the changes are on more-or-less
> the same importance level of course, you get the idea).
> 
> Now what to do with people who ignore reviews and/or have rather high
> regressions/patches ratio?

Ignoring a review would be a wildly wrong thing to do.  It's so unusual
that I'd be suspecting a lost email or an i-sent-the-wrong-patch.

As for high regressions/patches ratio: that'll be hard to calculate and
tends to be dependent upon the code which is being altered rather than who
is doing the altering: some stuff is just fragile, for various reasons.

One ratio which we might want to have a think about is the patches-sent
versus reviews-done ratio ;)

> I think that we should have info about regressions integrated into SCM,
> i.e. in git we should have optional "fixes-commit" tag and we should be
> able to do some reverse data colletion.   This feature combined with
> "Author:" info after some time should give us some very interesting
> statistics (Top Ten "Regressors").  It wouldn't be ideal (ie. we need some
> patches threshold to filter out people with 1 patch and >= 1 regression(s),
> we need to remember that some code areas are more difficult than the others
> and that patches are not equal per se etc.) however I believe than making it
> into Top Ten "Regressors" should give the winners some motivation to improve
> their work ethic.  Well, in the worst case we would just get some extra
> trivial/documentation patches. ;-)

We of course do want to minimise the amount of overhead for each developer. 
I'm a strong believer in specialisation: rather than requiring that *every*
developer/maintainer integrate new steps in their processes it would be
better to allow them to proceed in a close-to-usual fashion and to provide
for a specialist person (or team) to do the sorts of things which you're
thinking about.

Re: How to improve the quality of the kernel?

[Rafael J. Wysocki]

On Sunday, 17 June 2007 20:52, Andrew Morton wrote:
> On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> wrote:
> 
> > 
> > 
> > IMO we should concentrate more on preventing regressions than on fixing them.
> > In the long-term preventing bugs is cheaper than fixing them afterwards.
> > 
> > First let me tell you all a little story...
> > 
> > Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs
> > in it (in other words I potentially prevented three regressions).  I also
> > asked for more thorough verification of the patch as I suspected that it may
> > have more problems.  The author fixed the issues and replied that he hasn't
> > done the full verification yet but he doesn't suspect any problems...
> > 
> > Fast forward...
> > 
> > Year later I discover that the final version of the patch hit the mainline.
> > I don't remember ever seeing the final version in my mailbox (there are no
> > cc: lines in the patch description) and I saw that I'm not credited in the
> > patch description.  However the worse part is that it seems that the full
> > verification has never been done.  The result?   Regression in the release
> > kernel (exactly the issue that I was worried about) which required three
> > patches and over a month to be fixed completely.  It seems that a year
> > was not enough to get this ~70k _cleanup_ patch fully verified and tested
> > (it hit -mm soon before being merged)...
> 
> crap.  Commit ID, please ;)
> 
> > >From reviewer's POV: I have invested my time into review, discovered real
> > issues and as a reward I got no credit et all and extra frustration from the
> > fact that part of my review was forgotten/ignored (the part which resulted in
> > real regression in the release kernel)...  Oh and in the past the said
> > developer has already been asked (politely in private message) to pay more
> > attention to his changes (after I silently fixed some other regression caused
> > by his other patch).
> > 
> > But wait there is more, I happend to be the maintainer of the subsystem which
> > got directly hit by the issue and I was getting bugreports from the users about
> > the problem...  :-)
> > 
> > It wasn't my first/last bad experience as a reviewer... finally I just gave up
> > on reviewing other people patches unless they are stricly for IDE subsystem.
> > 
> > The moral of the story is that currently it just doesn't pay off to do
> > code reviews.
> 
> I dunno.  I suspect (hope) that this was an exceptional case, hence one
> should not draw general conclusions from it.  It certainly sounds very bad.
> 
> >  From personal POV it pays much more to wait until buggy patch
> > hits the mainline and then fix the issues yourself (at least you will get
> > some credit).  To change this we should put more ephasize on the importance
> > of code reviews by "rewarding" people investing their time into reviews
> > and "rewarding" developers/maintainers taking reviews seriously.
> > 
> > We should credit reviewers more, sometimes it takes more time/knowledge to
> > review the patch than to make it so getting near to zero credit for review
> > doesn't sound too attractive.  Hmm, wait it can be worse - your review
> > may be ignored... ;-)
> > 
> > >From my side I think I'll start adding less formal "Reviewed-by" to IDE
> > patches even if the review resulted in no issues being found (in additon to
> > explicit "Acked-by" tags and crediting people for finding real issues - which
> > I currently always do as a way for showing my appreciation for their work).
> 
> yup, Reviewed-by: is good and I do think we should start adopting it,
> although I haven't thought through exactly how.
> 
> On my darker days I consider treating a Reviewed-by: as a prerequisite for
> merging.  I suspect that would really get the feathers flying.

How about the following "algorithm":

* Step 1: Send a patch as an RFC to the relevant lists/people and only if there
  are no negative comments within at least n days, you are allowed to proceed
  to the next step.  If anyone has reviewed/acked the patch, add their names
  and email addresses as "Reviewed-by"/"Acked-by" to the patch in the next
  step.
* Step 2: Send the patch as an RC to the relevant lists/people _and_ LKML and
  if there are no negative comments within at least n days, you can proceed to
  the next step.  If anyone has reviewed/acked the patch, add their names
  and email addresses as "Reviewed-by"/"Acked-by" to the patch in the next
  step.
* Step 3: Submit the patch for merging to the right maintainer (keeping the
  previous CC list).

where n is a number that needs to be determined (I think that n could be 3).
Well, "negative comments" should also be defined more precisely. ;-)

> > I also encourage other maintainers/developers to pay more attention to
> > adding "Acked-by"/"Reviewed-by" tags and crediting reviewers.  I hope
> > that maintainers will promote changes that have been reviewed by others
> > by giving them priority over other ones (if the changes are on more-or-less
> > the same importance level of course, you get the idea).
> > 
> > Now what to do with people who ignore reviews and/or have rather high
> > regressions/patches ratio?
> 
> Ignoring a review would be a wildly wrong thing to do.  It's so unusual
> that I'd be suspecting a lost email or an i-sent-the-wrong-patch.
> 
> As for high regressions/patches ratio: that'll be hard to calculate and
> tends to be dependent upon the code which is being altered rather than who
> is doing the altering: some stuff is just fragile, for various reasons.
> 
> One ratio which we might want to have a think about is the patches-sent
> versus reviews-done ratio ;)
> 
> > I think that we should have info about regressions integrated into SCM,
> > i.e. in git we should have optional "fixes-commit" tag and we should be
> > able to do some reverse data colletion.   This feature combined with
> > "Author:" info after some time should give us some very interesting
> > statistics (Top Ten "Regressors").  It wouldn't be ideal (ie. we need some
> > patches threshold to filter out people with 1 patch and >= 1 regression(s),
> > we need to remember that some code areas are more difficult than the others
> > and that patches are not equal per se etc.) however I believe than making it
> > into Top Ten "Regressors" should give the winners some motivation to improve
> > their work ethic.  Well, in the worst case we would just get some extra
> > trivial/documentation patches. ;-)
> 
> We of course do want to minimise the amount of overhead for each developer. 
> I'm a strong believer in specialisation: rather than requiring that *every*
> developer/maintainer integrate new steps in their processes it would be
> better to allow them to proceed in a close-to-usual fashion and to provide
> for a specialist person (or team) to do the sorts of things which you're
> thinking about.

Still, even very experienced developers make trivial mistakes, so there should
be a way to catch such things before they hit -rc or even -mm kernels

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: How to improve the quality of the kernel?

[Carlo Wood]

On Sun, Jun 17, 2007 at 09:18:17PM +0200, Rafael J. Wysocki wrote:
> where n is a number that needs to be determined (I think that n could be 3).
> Well, "negative comments" should also be defined more precisely. ;-)

I think that n should be a function of the number of accepted patches
that this person sent in before, and the number of regressions he
caused in the past.

Ie, new developers have to wait a considerable amount of time - while
experienced developers who never caused a regression should be able
to write patches that are immediately applied. Also, if anyone causes
a regression - that would lead to them having to wait longer the
next time before they can apply the patch - a good reason for a
developer to put extra time into making sure there are no regressions.

-- 
Carlo Wood <carlo@alinoe.com>

Re: How to improve the quality of the kernel?

[Stefan Richter]

Carlo Wood wrote:
> On Sun, Jun 17, 2007 at 09:18:17PM +0200, Rafael J. Wysocki wrote:
>> where n is a number that needs to be determined (I think that n could be 3).
>> Well, "negative comments" should also be defined more precisely. ;-)
> 
> I think that n should be a function of the number of accepted patches
> that this person sent in before, and the number of regressions he
> caused in the past.

The character of the patch (potential impacts, size...) and availability
of reviewers and testers influence the required review time so much that
other factors, like reputation of the submitter, hardly matter.
-- 
Stefan Richter
-=====-=-=== -==- =---=
http://arcgraph.de/sr/

Re: How to improve the quality of the kernel?

[Michal Piotrowski]

On 17/06/07, Stefan Richter <stefanr@s5r6.in-berlin.de> wrote:
> Carlo Wood wrote:
> > On Sun, Jun 17, 2007 at 09:18:17PM +0200, Rafael J. Wysocki wrote:
> >> where n is a number that needs to be determined (I think that n could be 3).
> >> Well, "negative comments" should also be defined more precisely. ;-)
> >
> > I think that n should be a function of the number of accepted patches
> > that this person sent in before, and the number of regressions he
> > caused in the past.
>
> The character of the patch (potential impacts, size...) and availability
> of reviewers and testers influence the required review time so much that
> other factors, like reputation of the submitter, hardly matter.

So we need a bug/regression/patch tracking system based on MMORPG game ;)

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

Re: How to improve the quality of the kernel?

[Bartlomiej Zolnierkiewicz]

On Sunday 17 June 2007, Andrew Morton wrote:
> On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> wrote:
> 
> > 
> > 
> > IMO we should concentrate more on preventing regressions than on fixing them.
> > In the long-term preventing bugs is cheaper than fixing them afterwards.
> > 
> > First let me tell you all a little story...
> > 
> > Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs
> > in it (in other words I potentially prevented three regressions).  I also
> > asked for more thorough verification of the patch as I suspected that it may
> > have more problems.  The author fixed the issues and replied that he hasn't
> > done the full verification yet but he doesn't suspect any problems...
> > 
> > Fast forward...
> > 
> > Year later I discover that the final version of the patch hit the mainline.
> > I don't remember ever seeing the final version in my mailbox (there are no
> > cc: lines in the patch description) and I saw that I'm not credited in the
> > patch description.  However the worse part is that it seems that the full
> > verification has never been done.  The result?   Regression in the release
> > kernel (exactly the issue that I was worried about) which required three
> > patches and over a month to be fixed completely.  It seems that a year
> > was not enough to get this ~70k _cleanup_ patch fully verified and tested
> > (it hit -mm soon before being merged)...
> 
> crap.  Commit ID, please ;)

Will send in pm.

I don't want to reveal the "guilty" person identify in public.

> > >From reviewer's POV: I have invested my time into review, discovered real
> > issues and as a reward I got no credit et all and extra frustration from the
> > fact that part of my review was forgotten/ignored (the part which resulted in
> > real regression in the release kernel)...  Oh and in the past the said
> > developer has already been asked (politely in private message) to pay more
> > attention to his changes (after I silently fixed some other regression caused
> > by his other patch).
> > 
> > But wait there is more, I happend to be the maintainer of the subsystem which
> > got directly hit by the issue and I was getting bugreports from the users about
> > the problem...  :-)
> > 
> > It wasn't my first/last bad experience as a reviewer... finally I just gave up
> > on reviewing other people patches unless they are stricly for IDE subsystem.
> > 
> > The moral of the story is that currently it just doesn't pay off to do
> > code reviews.
> 
> I dunno.  I suspect (hope) that this was an exceptional case, hence one
> should not draw general conclusions from it.  It certainly sounds very bad.

I've been too long around to not learn a few things...

rule #3 of successful kernel developer

Ignore reviewers - fix the bugs but don't credit reviewers (crediting them
makes your patch and you look less perfect), if they are asking question
requiring you to do the work (verification of taken assumptions etc.) do not
check anything - answer in a misleading way and present the assumptions you've
taken as a truth written in the stone - eventually they will do verification
themselves.

I really shouldn't be giving these rules out (at least for free 8) so this
time only #3 but there are much more rules and they are as dead serious as
Linus' advices on Linux kernel management style...

> >  From personal POV it pays much more to wait until buggy patch
> > hits the mainline and then fix the issues yourself (at least you will get
> > some credit).  To change this we should put more ephasize on the importance
> > of code reviews by "rewarding" people investing their time into reviews
> > and "rewarding" developers/maintainers taking reviews seriously.
> > 
> > We should credit reviewers more, sometimes it takes more time/knowledge to
> > review the patch than to make it so getting near to zero credit for review
> > doesn't sound too attractive.  Hmm, wait it can be worse - your review
> > may be ignored... ;-)
> > 
> > >From my side I think I'll start adding less formal "Reviewed-by" to IDE
> > patches even if the review resulted in no issues being found (in additon to
> > explicit "Acked-by" tags and crediting people for finding real issues - which
> > I currently always do as a way for showing my appreciation for their work).
> 
> yup, Reviewed-by: is good and I do think we should start adopting it,
> although I haven't thought through exactly how.

Adding Reviewed-by for reviews which highlighted real issues is obvious
(with more detailed credits for noticed problems in the patch description).

Also when somebody reviewed your patch but the discussions it turned out
that the patch is valid - the review itself was still valuable so it would
be appropriate to credit the reviewer by adding Reviewed-by:.

> On my darker days I consider treating a Reviewed-by: as a prerequisite for
> merging.  I suspect that would really get the feathers flying.

Easy to workaround by a friendly mine "Reviewed-by:" for yours "Reviewed-by:"
deals (without any _proper_ review being done in reality)... ;)

> > I also encourage other maintainers/developers to pay more attention to
> > adding "Acked-by"/"Reviewed-by" tags and crediting reviewers.  I hope
> > that maintainers will promote changes that have been reviewed by others
> > by giving them priority over other ones (if the changes are on more-or-less
> > the same importance level of course, you get the idea).
> > 
> > Now what to do with people who ignore reviews and/or have rather high
> > regressions/patches ratio?
> 
> Ignoring a review would be a wildly wrong thing to do.  It's so unusual
> that I'd be suspecting a lost email or an i-sent-the-wrong-patch.

It is not unusual et all.  I mean patches which affect code in such way
that it is difficult to prove it's (in)correctness without doing time
consuming audit.

ie. lets imagine doing a small patch affecting many drivers - you've tested
it quickly on your driver/hardware, then you skip the part of verifying
correctness of new code in other drivers and just push the patch

As a patch author you can either assume "works for me" and push the patch
or do the audit (requires good understanding of the changed code and could
be time consuming).  It is usually quite easy to find out which approach
the author has choosen - the very sparse patch description combined with
the changes in code behavior not mentioned in the patch description should
raise the red flag. :)

As a reviewer having enough knowledge in the area of code affected by patch
you can see the potential problems but you can't prove them without doing
the time consuming part.  You may try to NACK the patch if you have enough
power but you will end up being bypassed by not proving incorrectness of
the patch (not to mention that developer will feel bad about you NACKing
his patch).  Now the funny thing is that despite the fact that audit takes
more time/knowledge then making the patch you will end up with zero credit
if patch turns out to be (luckily) correct.  Even if you find out issues
and report them you are still on mercy of author for being credited so
from personal POV you are much better to wait and fix issues after they
hit mainline kernel.  You have to choose between being a good citizen and
preventing kernel regressions or being bastard and getting the credit. ;)

If you happen to be maintainer of the affected code the choice is similar
with more pros for letting the patch in especially if you can't afford the
time to do audit (and by being maintainer you are guaranteed to be heavily
time constrained).

I hope this makes people see the importance of proper review and proper
recognition of reviewers in preventing kernel regressions.

> As for high regressions/patches ratio: that'll be hard to calculate and
> tends to be dependent upon the code which is being altered rather than who
> is doing the altering: some stuff is just fragile, for various reasons.
> 
> One ratio which we might want to have a think about is the patches-sent
> versus reviews-done ratio ;)

Sounds like a good idea.

> > I think that we should have info about regressions integrated into SCM,
> > i.e. in git we should have optional "fixes-commit" tag and we should be
> > able to do some reverse data colletion.   This feature combined with
> > "Author:" info after some time should give us some very interesting
> > statistics (Top Ten "Regressors").  It wouldn't be ideal (ie. we need some
> > patches threshold to filter out people with 1 patch and >= 1 regression(s),
> > we need to remember that some code areas are more difficult than the others
> > and that patches are not equal per se etc.) however I believe than making it
> > into Top Ten "Regressors" should give the winners some motivation to improve
> > their work ethic.  Well, in the worst case we would just get some extra
> > trivial/documentation patches. ;-)
> 
> We of course do want to minimise the amount of overhead for each developer. 
> I'm a strong believer in specialisation: rather than requiring that *every*
> developer/maintainer integrate new steps in their processes it would be
> better to allow them to proceed in a close-to-usual fashion and to provide
> for a specialist person (or team) to do the sorts of things which you're
> thinking about.

Makes sense... however we need to educate each and every developer about
importance of the code review and proper recognition of reviewers.

Thanks,
Bart

Re: How to improve the quality of the kernel?

[Stefan Richter]

Bartlomiej Zolnierkiewicz wrote:
> despite the fact that audit takes
> more time/knowledge then making the patch you will end up with zero credit
> if patch turns out to be (luckily) correct.  Even if you find out issues
> and report them you are still on mercy of author for being credited

If we introduce a "Reviewed-by" with reasonably clear semantics
(different from Signed-off-by; e.g. the reviewer is not a middle-man in
patch forwarding; the reviewer might have had remaining reservations...
very similar to but not entirely the same as "Acked-by" as currently
defined in -mm) --- and also make the already somewhat established
"Tested-by" more official, --- then the maintainers could start to make
it a habit to add Reviewed-by and Tested-by.

Plus, reviewers and testers could formally reply with Reviewed-by and
Tested-by lines to patch postings and even could explicitly ask the
maintainer to add these lines.

> so from personal POV you are much better to wait and fix issues after they
> hit mainline kernel.  You have to choose between being a good citizen and
> preventing kernel regressions or being bastard and getting the credit. ;)
> 
> If you happen to be maintainer of the affected code the choice is similar
> with more pros for letting the patch in especially if you can't afford the
> time to do audit (and by being maintainer you are guaranteed to be heavily
> time constrained).

I don't think that a maintainer (who signs off on patches after all) can
easily afford to take the "bastard approach".  I may be naive.
-- 
Stefan Richter
-=====-=-=== -==- =--=-
http://arcgraph.de/sr/

Re: How to improve the quality of the kernel?

[Bartlomiej Zolnierkiewicz]

On Monday 18 June 2007, Stefan Richter wrote:
> Bartlomiej Zolnierkiewicz wrote:
> > despite the fact that audit takes
> > more time/knowledge then making the patch you will end up with zero credit
> > if patch turns out to be (luckily) correct.  Even if you find out issues
> > and report them you are still on mercy of author for being credited
> 
> If we introduce a "Reviewed-by" with reasonably clear semantics
> (different from Signed-off-by; e.g. the reviewer is not a middle-man in
> patch forwarding; the reviewer might have had remaining reservations...
> very similar to but not entirely the same as "Acked-by" as currently
> defined in -mm) --- and also make the already somewhat established
> "Tested-by" more official, --- then the maintainers could start to make
> it a habit to add Reviewed-by and Tested-by.
> 
> Plus, reviewers and testers could formally reply with Reviewed-by and
> Tested-by lines to patch postings and even could explicitly ask the
> maintainer to add these lines.

Sounds great.

> > so from personal POV you are much better to wait and fix issues after they
> > hit mainline kernel.  You have to choose between being a good citizen and
> > preventing kernel regressions or being bastard and getting the credit. ;)
> > 
> > If you happen to be maintainer of the affected code the choice is similar
> > with more pros for letting the patch in especially if you can't afford the
> > time to do audit (and by being maintainer you are guaranteed to be heavily
> > time constrained).
> 
> I don't think that a maintainer (who signs off on patches after all) can
> easily afford to take the "bastard approach".  I may be naive.

Well, I'm not doing it myself but I find it tempting... ;)

In case of being maintainer "bastard approach" is more about not discouraging
developers by holding patches for too long than about getting credit.

Bart

Re: How to improve the quality of the kernel?

[Stefan Richter]

Bartlomiej Zolnierkiewicz wrote:
> In case of being maintainer "bastard approach" is more about not discouraging
> developers by holding patches for too long than about getting credit.

The maintainer who is about to suffocate in newly contributed code is
actually a lucky guy:  He can ask his eager contributors to also help
with cross-reviewing and bug fixing, otherwise all the fine work will be
stuck in the clogged pipeline.  (E.g. post a subsystem todo-list now and
then, as a subtle hint.)
-- 
Stefan Richter
-=====-=-=== -==- =--=-
http://arcgraph.de/sr/

Re: How to improve the quality of the kernel?

[Andrew Morton]

On Mon, 18 Jun 2007 01:15:15 +0200 Stefan Richter <stefanr@s5r6.in-berlin.de> wrote:

> Tested-by

Tested-by would be good too.  Because over time, we will generate a list of
people who own the relevant hardware and who are prepared to test changes. 

So if you make changes to random-driver.c you can do `git-log
random-driver.c|grep Tested-by" to find people who can test your changes
for you.

Not that many people are likely to bother.  The consequences of being
slack are negligible, hence there is little incentive to do the extra
work.

RE: How to improve the quality of the kernel?

[Fortier,Vincent [Montreal]]

> -----Message d'origine-----
> De : linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] De la part de 
> Andrew Morton
> 
> On Mon, 18 Jun 2007 01:15:15 +0200 Stefan Richter 
> <stefanr@s5r6.in-berlin.de> wrote:
> 
> > Tested-by
> 
> Tested-by would be good too.  Because over time, we will 
> generate a list of people who own the relevant hardware and 
> who are prepared to test changes. 

Why not include a user-space tool that, when invoked, if you agree to
send personnal info, sends your hardware vs driver info to a web
database + your email address (maybie even you .config, etc..) ... In
case of help for testing new patches/finding a bug/etc.. your email
could be used by maintainers to ask for help...

> So if you make changes to random-driver.c you can do `git-log 
> random-driver.c|grep Tested-by" to find people who can test 
> your changes for you.

You would'nt even need to search in GIT.  Maybie even when ever a
patchset is being proposed a mail could be sent to appropriate
hardware/or feature pseudo-auto-generated mailing-list?

On lkml I mostly try to follow patches/bugs associated with hardware I
use.  Why not try to automate the process and get more testers in?

- vin

Re: How to improve the quality of the kernel?

[Natalie Protasevich]

On 6/18/07, Fortier,Vincent [Montreal] <Vincent.Fortier1@ec.gc.ca> wrote:
> > -----Message d'origine-----
> > De : linux-kernel-owner@vger.kernel.org
> > [mailto:linux-kernel-owner@vger.kernel.org] De la part de
> > Andrew Morton
> >
> > On Mon, 18 Jun 2007 01:15:15 +0200 Stefan Richter
> > <stefanr@s5r6.in-berlin.de> wrote:
> >
> > > Tested-by
> >
> > Tested-by would be good too.  Because over time, we will
> > generate a list of people who own the relevant hardware and
> > who are prepared to test changes.
>
> Why not include a user-space tool that, when invoked, if you agree to
> send personnal info, sends your hardware vs driver info to a web
> database + your email address (maybie even you .config, etc..) ... In
> case of help for testing new patches/finding a bug/etc.. your email
> could be used by maintainers to ask for help...
>
> > So if you make changes to random-driver.c you can do `git-log
> > random-driver.c|grep Tested-by" to find people who can test
> > your changes for you.
>
> You would'nt even need to search in GIT.  Maybie even when ever a
> patchset is being proposed a mail could be sent to appropriate
> hardware/or feature pseudo-auto-generated mailing-list?
>
> On lkml I mostly try to follow patches/bugs associated with hardware I
> use.  Why not try to automate the process and get more testers in?
>

I think this is an excellent point. One data point could be a field in
bugzilla to input the hardware information. Simple query can select
common hardware and platform. So far it's not working when hardware is
just mentioned in the text part.

--Natalie

> - vin
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: How to improve the quality of the kernel?

[Martin Bligh]

>> > So if you make changes to random-driver.c you can do `git-log
>> > random-driver.c|grep Tested-by" to find people who can test
>> > your changes for you.
>>
>> You would'nt even need to search in GIT.  Maybie even when ever a
>> patchset is being proposed a mail could be sent to appropriate
>> hardware/or feature pseudo-auto-generated mailing-list?
>>
>> On lkml I mostly try to follow patches/bugs associated with hardware I
>> use.  Why not try to automate the process and get more testers in?
>>
> 
> I think this is an excellent point. One data point could be a field in
> bugzilla to input the hardware information. Simple query can select
> common hardware and platform. So far it's not working when hardware is
> just mentioned in the text part.

if it's free text it'll be useless for search ... I suppose we could
do drop-downs for architecture at least? Not sure much beyond that
would work ... *possibly* the common drivers, but I don't think
we'd get enough coverage for it to be of use.

M.

Re: How to improve the quality of the kernel?

[Natalie Protasevich]

On 6/18/07, Martin Bligh <mbligh@mbligh.org> wrote:
> >> > So if you make changes to random-driver.c you can do `git-log
> >> > random-driver.c|grep Tested-by" to find people who can test
> >> > your changes for you.
> >>
> >> You would'nt even need to search in GIT.  Maybie even when ever a
> >> patchset is being proposed a mail could be sent to appropriate
> >> hardware/or feature pseudo-auto-generated mailing-list?
> >>
> >> On lkml I mostly try to follow patches/bugs associated with hardware I
> >> use.  Why not try to automate the process and get more testers in?
> >>
> >
> > I think this is an excellent point. One data point could be a field in
> > bugzilla to input the hardware information. Simple query can select
> > common hardware and platform. So far it's not working when hardware is
> > just mentioned in the text part.
>
> if it's free text it'll be useless for search ... I suppose we could
> do drop-downs for architecture at least? Not sure much beyond that
> would work ... *possibly* the common drivers, but I don't think
> we'd get enough coverage for it to be of use.
>
How about several buckets for model/BIOS version/chipset etc., at
least optional, and some will be relevant some not for particular
cases. But at least people will make an attempt to collect such data
from their system, boards, etc.

--Natalie

Re: How to improve the quality of the kernel?

[Martin Bligh]

Natalie Protasevich wrote:
> On 6/18/07, Martin Bligh <mbligh@mbligh.org> wrote:
>> >> > So if you make changes to random-driver.c you can do `git-log
>> >> > random-driver.c|grep Tested-by" to find people who can test
>> >> > your changes for you.
>> >>
>> >> You would'nt even need to search in GIT.  Maybie even when ever a
>> >> patchset is being proposed a mail could be sent to appropriate
>> >> hardware/or feature pseudo-auto-generated mailing-list?
>> >>
>> >> On lkml I mostly try to follow patches/bugs associated with hardware I
>> >> use.  Why not try to automate the process and get more testers in?
>> >>
>> >
>> > I think this is an excellent point. One data point could be a field in
>> > bugzilla to input the hardware information. Simple query can select
>> > common hardware and platform. So far it's not working when hardware is
>> > just mentioned in the text part.
>>
>> if it's free text it'll be useless for search ... I suppose we could
>> do drop-downs for architecture at least? Not sure much beyond that
>> would work ... *possibly* the common drivers, but I don't think
>> we'd get enough coverage for it to be of use.
>
> How about several buckets for model/BIOS version/chipset etc., at
> least optional, and some will be relevant some not for particular
> cases. But at least people will make an attempt to collect such data
> from their system, boards, etc.

Mmm. the problem is that either they're:

1. free text, in which case they're useless, as everyone types
mis-spelled random crud into them. However, free-text search
through the comment fields might work out.

2. Drop downs, in which case someone has to manage the lists
etc, they're horribly crowded with lots of options. trying to
do that for model/BIOS version/chipset would be a nightmare.

If they're mandatory, they're a pain in the butt, and often
irrelevant ... if they're optional, nobody will fill them in.
Either way, they clutter the interface ;-(

Sorry to be a wet blanket, but I've seen those sort of things
before, and they just don't seem to work, especially in the
environment we're in with such a massive diversity of hardware.

If we can come up with some very clear, tightly constrained
choices, that's a decent possibility. Nothing other than
kernel architecture (i386 / x86_64 / ia64) or whatever springs
to mind, but perhaps I'm being unimaginative.

Nothing complicated ever seems to work ... even the simple
stuff is difficult ;-(

M.

Re: How to improve the quality of the kernel?

[Linus Torvalds]

On Mon, 18 Jun 2007, Martin Bligh wrote:
> 
> Sorry to be a wet blanket, but I've seen those sort of things
> before, and they just don't seem to work, especially in the
> environment we're in with such a massive diversity of hardware.

I do agree. It _sounds_ like a great idea to try to control the flow of 
patches better, but in the end, it needs to also be easy and painfree to 
the people involved, and also make sure that any added workflow doesn't 
require even *more* load and expertise on the already often overworked 
maintainers..

In many cases, I think it tends to *sound* great to try to avoid 
regressions in the first place - but it also sounds like one of those "I 
wish the world didn't work the way it did" kind of things. A worthy goal, 
but not necessarily one that is compatible with reality.

			Linus

Re: How to improve the quality of the kernel?

[Natalie Protasevich]

On 6/18/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Mon, 18 Jun 2007, Martin Bligh wrote:
> >
> > Sorry to be a wet blanket, but I've seen those sort of things
> > before, and they just don't seem to work, especially in the
> > environment we're in with such a massive diversity of hardware.
>
> I do agree. It _sounds_ like a great idea to try to control the flow of
> patches better, but in the end, it needs to also be easy and painfree to
> the people involved, and also make sure that any added workflow doesn't
> require even *more* load and expertise on the already often overworked
> maintainers..
>
> In many cases, I think it tends to *sound* great to try to avoid
> regressions in the first place - but it also sounds like one of those "I
> wish the world didn't work the way it did" kind of things. A worthy goal,
> but not necessarily one that is compatible with reality.
>
>                         Linus
>

Sure, simplicity is a key - but most of reporters on bugs are pretty
professional folks (or very well rounded amateurs :) We can try still
why not? the worst that can happen will be empty fields.

Maybe searching free text fields can then be implemented.  Then every
message exchange in bugzilla can be used for extracting such info -
questions about HW specifics are asked a lot, almost in every one.
It's a shame we cant' use this information. I was once searching for
"VIA" and got "zero bugs found", but in reality there are hundreds!
Probably something that makes sense to bring up with bugzilla project?

However, I've been working with other bugzillas (have to admit they
were mostly company/corporate), where this was a required field that
didn't seem to cause difficulties. I am planning to do some more
research and get some more ideas from other bugzillas. I suppose we
can have them discussed and revised sometime.

--Natalie

Re: How to improve the quality of the kernel?

[Martin Bligh]

> Sure, simplicity is a key - but most of reporters on bugs are pretty
> professional folks (or very well rounded amateurs :) We can try still
> why not? the worst that can happen will be empty fields.

mmm. added complexity and interface clutter for little or no benefit
is what I'm trying to avoid - they did that in the IBM bugzilla and
it turned into a big ugly unusable monster. You can call me either
"experienced" or "bitter" depending what mood you're in ;-)

Not sure I'd agree that most of the bug submitters are all that
professional, it's a very mixed bag.

> Maybe searching free text fields can then be implemented.  Then every
> message exchange in bugzilla can be used for extracting such info -
> questions about HW specifics are asked a lot, almost in every one.
> It's a shame we cant' use this information. I was once searching for
> "VIA" and got "zero bugs found", but in reality there are hundreds!
> Probably something that makes sense to bring up with bugzilla project?

That should work now ... seems to for me.

http://bugzilla.kernel.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=substring&long_desc=VIA&kernel_version_type=allwordssubstr&kernel_version=&bug_status=NEW&bug_status=REOPENED&bug_status=ASSIGNED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=REJECTED&bug_status=DEFERRED&bug_status=NEEDINFO&bug_status=CLOSED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&regression=both&cmdtype=doit&order=Bug+Number&field0-0-0=noop&type0-0-0=noop&value0-0-0=

Produces a metric-buttload of results. Go to the advanced search option
and do "A Comment" contains the string "VIA". By default "Status" is
only set to do open bugs, which you might want to change to all types.

> However, I've been working with other bugzillas (have to admit they
> were mostly company/corporate), where this was a required field that
> didn't seem to cause difficulties. I am planning to do some more
> research and get some more ideas from other bugzillas. I suppose we
> can have them discussed and revised sometime.

Would be good, thanks. I tend to favour keeping things as simple as
possible though, we have very little control over our users, and they're
a very broad base. Making the barrier to entry for use as low as
possible is the design we've been pursuing.

M.

Re: How to improve the quality of the kernel?

[Natalie Protasevich]

On 6/18/07, Martin Bligh <mbligh@mbligh.org> wrote:
> > Sure, simplicity is a key - but most of reporters on bugs are pretty
> > professional folks (or very well rounded amateurs :) We can try still
> > why not? the worst that can happen will be empty fields.
>
> mmm. added complexity and interface clutter for little or no benefit
> is what I'm trying to avoid - they did that in the IBM bugzilla and
> it turned into a big ugly unusable monster. You can call me either
> "experienced" or "bitter" depending what mood you're in ;-)
>
> Not sure I'd agree that most of the bug submitters are all that
> professional, it's a very mixed bag.
>
> > Maybe searching free text fields can then be implemented.  Then every
> > message exchange in bugzilla can be used for extracting such info -
> > questions about HW specifics are asked a lot, almost in every one.
> > It's a shame we cant' use this information. I was once searching for
> > "VIA" and got "zero bugs found", but in reality there are hundreds!
> > Probably something that makes sense to bring up with bugzilla project?
>
> That should work now ... seems to for me.
>
> http://bugzilla.kernel.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=substring&long_desc=VIA&kernel_version_type=allwordssubstr&kernel_version=&bug_status=NEW&bug_status=REOPENED&bug_status=ASSIGNED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=REJECTED&bug_status=DEFERRED&bug_status=NEEDINFO&bug_status=CLOSED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&regression=both&cmdtype=doit&order=Bug+Number&field0-0-0=noop&type0-0-0=noop&value0-0-0=
>
> Produces a metric-buttload of results. Go to the advanced search option
> and do "A Comment" contains the string "VIA". By default "Status" is
> only set to do open bugs, which you might want to change to all types.

Yes it works great! Thanks... I'd say this should be really a default
search, because first search screen is misleading - it promises find
all for any "words" :)

>
> > However, I've been working with other bugzillas (have to admit they
> > were mostly company/corporate), where this was a required field that
> > didn't seem to cause difficulties. I am planning to do some more
> > research and get some more ideas from other bugzillas. I suppose we
> > can have them discussed and revised sometime.
>
> Would be good, thanks. I tend to favour keeping things as simple as
> possible though, we have very little control over our users, and they're
> a very broad base. Making the barrier to entry for use as low as
> possible is the design we've been pursuing.

Actually, as long as search above is possible - it is going to work.

I must say the new bugzilla interface is very nice in general, and
well designed and easy to use.

--Natalie

Re: How to improve the quality of the kernel?

[Martin Bligh]

>> > Maybe searching free text fields can then be implemented.  Then every
>> > message exchange in bugzilla can be used for extracting such info -
>> > questions about HW specifics are asked a lot, almost in every one.
>> > It's a shame we cant' use this information. I was once searching for
>> > "VIA" and got "zero bugs found", but in reality there are hundreds!
>> > Probably something that makes sense to bring up with bugzilla project?
>>
>> That should work now ... seems to for me.
>>
>> http://bugzilla.kernel.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=substring&long_desc=VIA&kernel_version_type=allwordssubstr&kernel_version=&bug_status=NEW&bug_status=REOPENED&bug_status=ASSIGNED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=REJECTED&bug_status=DEFERRED&bug_status=NEEDINFO&bug_status=CLOSED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&regression=both&cmdtype=doit&order=Bug+Number&field0-0-0=noop&type0-0-0=noop&value0-0-0= 
>>
>>
>> Produces a metric-buttload of results. Go to the advanced search option
>> and do "A Comment" contains the string "VIA". By default "Status" is
>> only set to do open bugs, which you might want to change to all types.
> 
> Yes it works great! Thanks... I'd say this should be really a default
> search, because first search screen is misleading - it promises find
> all for any "words" :)

OK, or at the very least we can fix the text at least to indicate it'll
only search summaries (and likely only of open bugs at that ...)

M.

This is [Re:] How to improve the quality of the kernel[?].

[Oleg Verych]

[Dear Debbug developers, i wish your ideas will be useful.]

* From: Linus Torvalds
* Newsgroups: gmane.linux.kernel
* Date: Mon, 18 Jun 2007 17:09:37 -0700 (PDT)
>
> On Mon, 18 Jun 2007, Martin Bligh wrote:
>> 
>> Sorry to be a wet blanket, but I've seen those sort of things
>> before, and they just don't seem to work, especially in the
>> environment we're in with such a massive diversity of hardware.
>
> I do agree. It _sounds_ like a great idea to try to control the flow of 
> patches better,

There were some ideas, i will try to summarize:

* New Patches (or sets) need tracking, review, testing

  Zero (tracking) done by sending (To, or bcc) [RFC] patch to something
  like submit@pts.e-mail.example.com (like BTS now). Notifications will
  be sent to intrested maintainers (if meta-information is ok) or testers
  (see below)

  First is mostly done by maintainers or interested individuals.
  Result is "Acked-by" and "Cc" entries in the next patch sent. Due to
  lack of tracking this things are done manually, are generally in
  trusted manner. But bad like <200706172053.41806.bzolnier@gmail.com>
  sometimes happens.

  When patch in sent to this PTS, your lovely
  checkpatch/check-whatever-crap-has-being-sent tools can be set up as
  gatekeepers, thus making impossible stupid errors with MIME coding,
  line wrapping, whatever style you've came up with now in checking
  sent crap.

* Tracking results of review (Acked-by).

  This can be mostly e-mail exchange with comments and agreements.
  "Acked-by" semantic may be implemented in form of contlol message to
  tracking system, and this system will generate e-mail confirmation
  to the patch author in form of
  "Acked-by: Developer's Name <message-id of e-mail with acke-by>"

  Thus, next patch will have this entry. And if testing of this
  version ir regression happens, there's info about who is/was
  interested/involved.

* Testing.

  Mainly same for "Tested-by"
  (newly suggested by Stefan <4675C083.6080409@s5r6.in-berlin.de>)

|-*- Feature Needed -*-
  Addition, needed is hardware user tested have/had/used. Currently
  ``reportbug'' tool includes packed specific and system specific
  additions automaticly gathered and inserted to e-mail sent to BTS.
  (e.g. <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29740>)

  Formats of that hardware profile(as system information in reportbug)
  . arch
  . chipset
  . hdd
  . vga
  ...
  in meaningful fields, and not just lspci -v[vv]. If additional info
  (-vvv) or something required, profile can be exteded.

  For kernel's sub-system information(as packed info):
  . subsystem/driver/kernel version (or similar)
  . maintainers must know what they need to see more here

|-*- back to patches -*-

  Last and not least tast cases, that everyone might came up with.

  All formats this can be agreed (or implemented and updated latter)
  and inserted automaticly.

* Commit.
  Id is recorded, patch archived. But any additions are welcome,
  regressions will pop up this patch again (reopen in BTS).

> but in the end, it needs to also be easy and painfree to the people
> involved, and also make sure that any added workflow doesn't require
> even *more* load and expertise on the already often overworked 
> maintainers..

Experienced BTS users and developers. Please, correct me if i'm wrong.
At least e-mail part of Debian's BTS and whole idea of it is *exactly*
what is needed. Bugzilla fans, you can still use you useless pet,
because Debian developers have done things, to track and e-mail states
of bugs: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29736>

> In many cases, I think it tends to *sound* great to try to avoid 
> regressions in the first place - but it also sounds like one of those "I 
> wish the world didn't work the way it did" kind of things. A worthy goal, 
> but not necessarily one that is compatible with reality.

I wish perl hackers out there will join this yet-new effort. I know
there many of them out there, writing kilobytes of checkfile and
checkpatch (i've wrote in few lines of ``sed'').

BTS is written on perl, but any interoperability interface, like
stdin/stdout for python or shell hackers is worth of thinking about.

Please, see more and make useful follows ups: http://bugs.debian.org/

Please, do not (<46772321.9080602@mbligh.org>)
""" I know you hate bugzilla ... but at least I can try to make that bit
    of the process work better.
""" [here's you fancy checkbox...]

Thanks.
____

Re: This is [Re:] How to improve the quality of the kernel[?].

[Adrian Bunk]

On Tue, Jun 19, 2007 at 06:06:47AM +0200, Oleg Verych wrote:
> [Dear Debbug developers, i wish your ideas will be useful.]
> 
> * From: Linus Torvalds
> * Newsgroups: gmane.linux.kernel
> * Date: Mon, 18 Jun 2007 17:09:37 -0700 (PDT)
> >
> > On Mon, 18 Jun 2007, Martin Bligh wrote:
> >> 
> >> Sorry to be a wet blanket, but I've seen those sort of things
> >> before, and they just don't seem to work, especially in the
> >> environment we're in with such a massive diversity of hardware.
> >
> > I do agree. It _sounds_ like a great idea to try to control the flow of 
> > patches better,
> 
> There were some ideas, i will try to summarize:
> 
> * New Patches (or sets) need tracking, review, testing
> 
>   Zero (tracking) done by sending (To, or bcc) [RFC] patch to something
>   like submit@pts.e-mail.example.com (like BTS now). Notifications will
>   be sent to intrested maintainers (if meta-information is ok) or testers
>   (see below)
> 
>   First is mostly done by maintainers or interested individuals.
>   Result is "Acked-by" and "Cc" entries in the next patch sent. Due to
>   lack of tracking this things are done manually, are generally in
>   trusted manner. But bad like <200706172053.41806.bzolnier@gmail.com>
>   sometimes happens.

The goal is to get all patches for a maintained subsystem submitted to 
Linus by the maintainer.

>   When patch in sent to this PTS, your lovely
>   checkpatch/check-whatever-crap-has-being-sent tools can be set up as
>   gatekeepers, thus making impossible stupid errors with MIME coding,
>   line wrapping, whatever style you've came up with now in checking
>   sent crap.

The -mm kernel already implements what your proposed PTS would do.

Plus it gives testers more or less all patches currently pending 
inclusion into Linus' tree in one kernel they can test.

The problem are more social problems like patches Andrew has never heard 
of before getting into Linus' tree during the merge window.

>...
> |-*- Feature Needed -*-
>   Addition, needed is hardware user tested have/had/used. Currently
>   ``reportbug'' tool includes packed specific and system specific
>   additions automaticly gathered and inserted to e-mail sent to BTS.
>   (e.g. <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29740>)

The problem is that most problems don't occur on one well-defined 
kind of hardware - patches often break in exactly the areas the patch
author expected no problems in.

And in many cases a patch for a device driver was written due to a bug 
report - in such cases a tester with the hardware in question is already 
available.

>...
> > but in the end, it needs to also be easy and painfree to the people
> > involved, and also make sure that any added workflow doesn't require
> > even *more* load and expertise on the already often overworked 
> > maintainers..
> 
> Experienced BTS users and developers. Please, correct me if i'm wrong.
> At least e-mail part of Debian's BTS and whole idea of it is *exactly*
> what is needed. Bugzilla fans, you can still use you useless pet,
> because Debian developers have done things, to track and e-mail states
> of bugs: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29736>
>...

"useless pet"?
Be serious.
How many open source projects use Bugzilla and how many use the Debian BTS?
And then start thinking about why the "useless pet" has so many more 
user...

The Debian BTS requires you to either write emails with control messages 
or generating control messages with external tools.

In Bugzilla the same works through a web interface.

I know both the Debian BTS and Bugzilla and although they are quite 
different they both are reasonable tools for their purpose.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: This is [Re:] How to improve the quality of the kernel[?].

[Oleg Verych]

[Dropping noise for Debbugs, because interested people may join via Gmane]

On Tue, Jun 19, 2007 at 02:48:55PM +0200, Adrian Bunk wrote:
> On Tue, Jun 19, 2007 at 06:06:47AM +0200, Oleg Verych wrote:
> > [Dear Debbug developers, i wish your ideas will be useful.]
> > 
> > * From: Linus Torvalds
> > * Newsgroups: gmane.linux.kernel
> > * Date: Mon, 18 Jun 2007 17:09:37 -0700 (PDT)
> > >
> > > On Mon, 18 Jun 2007, Martin Bligh wrote:
> > >> 
> > >> Sorry to be a wet blanket, but I've seen those sort of things
> > >> before, and they just don't seem to work, especially in the
> > >> environment we're in with such a massive diversity of hardware.
> > >
> > > I do agree. It _sounds_ like a great idea to try to control the flow of 
> > > patches better,
> > 
> > There were some ideas, i will try to summarize:
> > 
> > * New Patches (or sets) need tracking, review, testing
> > 
> >   Zero (tracking) done by sending (To, or bcc) [RFC] patch to something
> >   like submit@pts.e-mail.example.com (like BTS now). Notifications will
> >   be sent to intrested maintainers (if meta-information is ok) or testers
> >   (see below)
> > 
> >   First is mostly done by maintainers or interested individuals.
> >   Result is "Acked-by" and "Cc" entries in the next patch sent. Due to
> >   lack of tracking this things are done manually, are generally in
> >   trusted manner. But bad like <200706172053.41806.bzolnier@gmail.com>
> >   sometimes happens.
> 
> The goal is to get all patches for a maintained subsystem submitted to 
> Linus by the maintainer.
> 
> >   When patch in sent to this PTS, your lovely
> >   checkpatch/check-whatever-crap-has-being-sent tools can be set up as
> >   gatekeepers, thus making impossible stupid errors with MIME coding,
> >   line wrapping, whatever style you've came up with now in checking
> >   sent crap.
> 
> The -mm kernel already implements what your proposed PTS would do.

Having all-in-one patchset, like -mm is easy thing to provide
interested parties with "you know what you have -- crazy development"

However [P]TS is tracking, recording, organizing tool. {1} Andrew's cron
daemon easily can run script to check status of particular patch (cc,
tested-by, acked-by). If patch have no TS ID, Andrew's watchdog is
barking back to patch originator (with polite asking to send patch as:

* TS as "To:" target
* patch author as "Cc:" target, this is useful to require:
  . author can check that copy himself with text-only pager program
    (to see any MIME coding crap)
  . preventing SPAM
* maybe somebody else in Cc or Bcc.)

> Plus it gives testers more or less all patches currently pending 
> inclusion into Linus' tree in one kernel they can test.

Crazy development{0}. Somebody knows, that comprehensively testing
hibernation is their thing. I don't care about it, i care about foo, bar.
Thus i can apply for example lguest patches and implement and test new
asm-offset replacement, *easily*. Somebody, as you know, likes new fancy
file system, and no-way other. Let them be happy testing that thing
*easily*. Because another fancy NO_MHz will hang their testing bench
right after best ever speed results were recorded.

> The problem are more social problems like patches Andrew has never heard 
> of before getting into Linus' tree during the merge window.

Linus' watchdog, as well, asking for valid patch id, or just doesn't
care (in similar manner Linus does :).

So far no human is involved in social things. Do you agree?

Human power is worth and needed in particular patch discussion and
testing under the participation (by Cc, acking, test-ok *e-mails*) of
tracking system.

> >...
> > |-*- Feature Needed -*-
> >   Addition, needed is hardware user tested have/had/used. Currently
> >   ``reportbug'' tool includes packed specific and system specific
> >   additions automaticly gathered and inserted to e-mail sent to BTS.
> >   (e.g. <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29740>)
> 
> The problem is that most problems don't occur on one well-defined 
> kind of hardware - patches often break in exactly the areas the patch
> author expected no problems in.

I tried to test that new fancy FS, and couldn't boot because of
yet-another ACPI crap. See theme{0}?

Overall testing, like Andrew does, is doubtless brave thing, but he have
more time after {1}, isn't it?

> And in many cases a patch for a device driver was written due to a bug 
> report - in such cases a tester with the hardware in question is already 
> available.

Tracking all possible testers (for next driver update, for example) is
in question.

> 
> >...
> > > but in the end, it needs to also be easy and painfree to the people
> > > involved, and also make sure that any added workflow doesn't require
> > > even *more* load and expertise on the already often overworked 
> > > maintainers..
> > 
> > Experienced BTS users and developers. Please, correct me if i'm wrong.
> > At least e-mail part of Debian's BTS and whole idea of it is *exactly*
> > what is needed. Bugzilla fans, you can still use you useless pet,
> > because Debian developers have done things, to track and e-mail states
> > of bugs: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29736>
> >...
> 
> "useless pet"?
> Be serious.
> How many open source projects use Bugzilla and how many use the Debian BTS?
> And then start thinking about why the "useless pet" has so many more 
> user...

I might be stupid, but i faced it. On my amd64 512M laptop i *cannot* run
mozillka any more, for example! And i don't care, because Linus said his
opinion and i fully agree with him.

> The Debian BTS requires you to either write emails with control messages 
> or generating control messages with external tools.

Awesome!!! Are you wrote this reply to me by 

> In Bugzilla the same works through a web interface.

web interface? If you did .........</dev/random dd bs=1 count=13.....
Actually you didn't ;)

> I know both the Debian BTS and Bugzilla and although they are quite 
> different they both are reasonable tools for their purpose.

As you just might have seen here, i was talking about organizing,
tracking, hopefully saving and redirecting useful main power. And i don't
bother search e-mails i saw about Bugzilla's BD from many other prominent
developers. I just know that, not from my dream or physical vacuum.

Basic concept of Debian BTS is what i've discovered after many useless
hours i spent in Bugzilla. And this is mainly because of one basic
important thing, that nobody acknowledged (for newbies, like me):

* E-Mail with useful MUAs, after it got reliable delivery MTAs with qmail
  (or exim) is the main communication toolset.

Can't you see that from Linux's patch sending policy?

I also what to reply to myself about why LKML was established and
USENET (news) was abandoned.

To control and to keep running *your* _main communication toolset_
(read as "your user,developer feedback").

I just couldn't realize that, because i grew up in free web e-mail, after
having set up my own server with MTA and real e-mail and after
discovering Gmane (really mind-blowing evolution of the e-mail system!)

> cu
> Adrian
> 
> -- 
____

Re: This is [Re:] How to improve the quality of the kernel[?].

[Stefan Richter]

On 6/19/2007 4:05 PM, Oleg Verych wrote:
> On Tue, Jun 19, 2007 at 02:48:55PM +0200, Adrian Bunk wrote:
>> The Debian BTS requires you to either write emails with control messages 
>> or generating control messages with external tools.
...
>> In Bugzilla the same works through a web interface.
...
> Basic concept of Debian BTS is what i've discovered after many useless
> hours i spent in Bugzilla. And this is mainly because of one basic
> important thing, that nobody acknowledged (for newbies, like me):
> 
> * E-Mail with useful MUAs, after it got reliable delivery MTAs with qmail
>   (or exim) is the main communication toolset.
> 
> Can't you see that from Linux's patch sending policy?

That's for developers, not for users.

There are different people involved in
  - patch handling,
  - bug handling (bugs are reported by end-users),
therefore don't forget that PTS and BTS have different requirements.
-- 
Stefan Richter
-=====-=-=== -==- =--==
http://arcgraph.de/sr/

Re: This is [Re:] How to improve the quality of the kernel[?].

[Oleg Verych]

On Tue, Jun 19, 2007 at 04:27:15PM +0200, Stefan Richter wrote:
> On 6/19/2007 4:05 PM, Oleg Verych wrote:
> > On Tue, Jun 19, 2007 at 02:48:55PM +0200, Adrian Bunk wrote:
> >> The Debian BTS requires you to either write emails with control messages 
> >> or generating control messages with external tools.
> ...
> >> In Bugzilla the same works through a web interface.
> ...
> > Basic concept of Debian BTS is what i've discovered after many useless
> > hours i spent in Bugzilla. And this is mainly because of one basic
> > important thing, that nobody acknowledged (for newbies, like me):
> > 
> > * E-Mail with useful MUAs, after it got reliable delivery MTAs with qmail
> >   (or exim) is the main communication toolset.
> > 
> > Can't you see that from Linux's patch sending policy?
> 
> That's for developers, not for users.
> 
> There are different people involved in
>   - patch handling,
>   - bug handling (bugs are reported by end-users),
> therefore don't forget that PTS and BTS have different requirements.

Sure. But if tracking was done, possible bugs where killed, user's bug
report seems to depend on that patch (bisecting), why not to have a
linkage here? Usefulness for a developer (in sub-system association),
next time to see what went wrong, check test-cases, users might be
interested to have them run too before crying (again) about broken
system. Bug report can become part of (reopened) patch discussion (as
i've wrote). Until that, as bug-candidate without identified patch it
can be associated to some particular sub-system or abstract one
bug-category {1}.

Reversed time. As "do-bisection" shows, problems are not happening
just simply because of something abstract. If problem worth of solving
it, eventually there will be patch trying solve that, in both cases:

* when breaking patch (bisection) actually correct, but hardware
  (or similar independent) problem arise.
* something different, like feature request or something.

So, this guys are candidate for patch, and can have ID numerically from
the same domain as patch ID, but with different prefix, like "i'm just
candidate for patch". Bugs {1}, are obviously in this category.

Current identification of problems and patch association
have completely zero level of tracking or automation, while Bugzilla is
believed by somebody to have positive efficiency in bug tracking.

That two (patch/bug tracking) aren't that perpendicular to each other at
all.

Eventually it might be that perfect unification, that bug-tracking can be
obsolete, because of good tracking of patches/features-added and what
they did/do.

In any case, i would like to ask mentors to write at least something
similar to technical task, if that, what i'm saying is accessible for
you. Because your experience is treasure, that must be preserved and
possibly automated/organized.
____

Re: This is [Re:] How to improve the quality of the kernel[?].

[Stefan Richter]

Oleg Verych wrote:
> On Tue, Jun 19, 2007 at 04:27:15PM +0200, Stefan Richter wrote:
>> There are different people involved in
>>   - patch handling,
>>   - bug handling (bugs are reported by end-users),
>> therefore don't forget that PTS and BTS have different requirements.
> 
> Sure. But if tracking was done, possible bugs where killed, user's bug
> report seems to depend on that patch (bisecting), why not to have a
> linkage here?

Of course there are certain links between bugs and patches, and thus
there are certain links between bug tracking and patch tracking.

[...]
> Current identification of problems and patch association
> have completely zero level of tracking or automation, while Bugzilla is
> believed by somebody to have positive efficiency in bug tracking.

I, as maintainer of a small subsystem, can personally track bug--patch
relationships with bugzilla just fine, on its near-zero level of
automation and integration.

Nevertheless, would a more integrated bug/patch tracking system help me
improve quality of my output? ---
a) Would it save me more time than it costs me to fit into the system
   (time that can be invested in actual debugging)?
   This can only be answered after trying it.
b) Would it help me to spot mistakes in patches before I submit?
   No.
c) Would I get quicker feedback from testers?
   That depends on whether such a system attracts testers and helps
   testers to work efficiently.  This is also something that can only be
   speculated about without trying it.

The potential testers that I deal with are mostly either very
non-technical persons, or persons which are experienced in their
hardware/application area but *not* in kernel internals and kernel
development procedures.
-- 
Stefan Richter
-=====-=-=== -==- =--==
http://arcgraph.de/sr/

Re: This is [Re:] How to improve the quality of the kernel[?].

[Oleg Verych]

* Date: Tue, 19 Jun 2007 19:50:48 +0200
> 
> [...]
>> Current identification of problems and patch association
>> have completely zero level of tracking or automation, while Bugzilla is
>> believed by somebody to have positive efficiency in bug tracking.
>
> I, as maintainer of a small subsystem, can personally track bug--patch
> relationships with bugzilla just fine, on its near-zero level of
> automation and integration.
>
> Nevertheless, would a more integrated bug/patch tracking system help me
> improve quality of my output? ---
> a) Would it save me more time than it costs me to fit into the system
>    (time that can be invested in actual debugging)?
>    This can only be answered after trying it.

I'm not a wizard, if i will answer now: "No." [1:]

[1:] Your User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.4) Gecko/20070509 SeaMonkey/1.1.2

> b) Would it help me to spot mistakes in patches before I submit?
>    No.

If you ever tried to report bug with reportbug tool in Debian, you may
understand what i meant. Nothing can substitute intelligence. Something
can reduce impact of laziness (of searching relevant bugreports).

> c) Would I get quicker feedback from testers?
>    That depends on whether such a system attracts testers and helps
>    testers to work efficiently.  This is also something that can only be
>    speculated about without trying it.
>
> The potential testers that I deal with are mostly either very
> non-technical persons, or persons which are experienced in their
> hardware/application area but *not* in kernel internals and kernel
> development procedures.

They also don't bother subscribing to mailing lists and like to write
blogs. I'm not sure about hw databases you talked about, i will talk
about gathering information from testers.

Debian have experimental and unstable branches, people willing to have
new stuff are likely to have this, and not testing or stable. BTS just
collects bugreports <http://bugs.debian.org/>. If kernel team uploads new
kernel (release or even rc recently), interested people will use it after
next upgrade. Bug reports get collected, but main answer will be, try
reproduce on most recent kernel.org's one. Here, what i have proposed,
may play role you expect. Mis-configuration/malfunctioning, programmer's
error (Linus noted) in organized manner may easily join reporting person
to kernel.org's testing. On driver or small sub-system level this may
work. Again it's all about information, not intelligence.
____

Re: This is [Re:] How to improve the quality of the kernel[?].

[Stefan Richter]

Oleg Verych wrote:
[I wrote]
>> a) Would it save me more time than it costs me to fit into the system
>>    (time that can be invested in actual debugging)?
>>    This can only be answered after trying it.
> 
> I'm not a wizard, if i will answer now: "No." [1:]
> 
> [1:] Your User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.4) Gecko/20070509 SeaMonkey/1.1.2

Seamonkey isn't interoperable with Debian's BTS?
Lucky me that I frequently use other MUAs too.
-- 
Stefan Richter
-=====-=-=== -==- =--==
http://arcgraph.de/sr/

Re: This is [Re:] How to improve the quality of the kernel[?].

[Adrian Bunk]

On Tue, Jun 19, 2007 at 04:05:12PM +0200, Oleg Verych wrote:
>...
> On Tue, Jun 19, 2007 at 02:48:55PM +0200, Adrian Bunk wrote:
> > On Tue, Jun 19, 2007 at 06:06:47AM +0200, Oleg Verych wrote:
>...
> > >   When patch in sent to this PTS, your lovely
> > >   checkpatch/check-whatever-crap-has-being-sent tools can be set up as
> > >   gatekeepers, thus making impossible stupid errors with MIME coding,
> > >   line wrapping, whatever style you've came up with now in checking
> > >   sent crap.
> > 
> > The -mm kernel already implements what your proposed PTS would do.
> 
> Having all-in-one patchset, like -mm is easy thing to provide
> interested parties with "you know what you have -- crazy development"
> 
> However [P]TS is tracking, recording, organizing tool. {1} Andrew's cron
> daemon easily can run script to check status of particular patch (cc,
> tested-by, acked-by). If patch have no TS ID, Andrew's watchdog is
> barking back to patch originator (with polite asking to send patch as:
> 
> * TS as "To:" target
> * patch author as "Cc:" target, this is useful to require:
>   . author can check that copy himself with text-only pager program
>     (to see any MIME coding crap)
>   . preventing SPAM
> * maybe somebody else in Cc or Bcc.)

Quite a big part of -mm are git trees of maintainers.
Where are they in your tool?

And I still don't think your tool would make sense.
But hey, simply try it - that's the only way for you to prove me wrong.
People said similar things about the 2.6.16 kernel or my regression 
tracking, and I simply did it.

> > Plus it gives testers more or less all patches currently pending 
> > inclusion into Linus' tree in one kernel they can test.
> 
> Crazy development{0}. Somebody knows, that comprehensively testing
> hibernation is their thing. I don't care about it, i care about foo, bar.
> Thus i can apply for example lguest patches and implement and test new
> asm-offset replacement, *easily*. Somebody, as you know, likes new fancy
> file system, and no-way other. Let them be happy testing that thing
> *easily*. Because another fancy NO_MHz will hang their testing bench
> right after best ever speed results were recorded.

Patch dependencies and patch conflicts will be the interesting parts 
when you will implement this.

E.g. new fancy filesystem patch in -mm might depend on some VFS change 
that requires changes to all other filesystems.

I'm really looking forward to see how you will implement this for 
something like -mm with > 1000 patches (many of them git trees that 
themselves contain many different patches) without offloading all the 
additional work to the kernel developers.

> > The problem are more social problems like patches Andrew has never heard 
> > of before getting into Linus' tree during the merge window.
> 
> Linus' watchdog, as well, asking for valid patch id, or just doesn't
> care (in similar manner Linus does :).
> 
> So far no human is involved in social things. Do you agree?

No.

Forcing people to use some tool (no matter whether it's Bugzilla or
the PTS you want to implement) is 100% a social problem involving humans.

> Human power is worth and needed in particular patch discussion and
> testing under the participation (by Cc, acking, test-ok *e-mails*) of
> tracking system.

For getting people to use your tool, you will have to convince them that 
using your tool will bring them real benefits.

> > >...
> > > |-*- Feature Needed -*-
> > >   Addition, needed is hardware user tested have/had/used. Currently
> > >   ``reportbug'' tool includes packed specific and system specific
> > >   additions automaticly gathered and inserted to e-mail sent to BTS.
> > >   (e.g. <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29740>)
> > 
> > The problem is that most problems don't occur on one well-defined 
> > kind of hardware - patches often break in exactly the areas the patch
> > author expected no problems in.
> 
> I tried to test that new fancy FS, and couldn't boot because of
> yet-another ACPI crap. See theme{0}?
> 
> Overall testing, like Andrew does, is doubtless brave thing, but he have
> more time after {1}, isn't it?

I doubt the placing of some Acked-By- tags in patches is really what 
is killing Andrews time.

How does Andrew check the status of 1500 patches in -mm in your PTS?

And how do you implement the use case that Andrew forwards a batch of
200 patches to Linus? How does the information from your tool come into git?

But hey, write your tool and convince Andrew of it's advantages if you 
don't believe me.

> > And in many cases a patch for a device driver was written due to a bug 
> > report - in such cases a tester with the hardware in question is already 
> > available.
> 
> Tracking all possible testers (for next driver update, for example) is
> in question.

Spamming people who have some hardware with information about patches 
won't bring you anything. You need people willing to test patches that 
won't bring them any benefit - and if you have such people they are 
usually as well willing to simply regularly test -rc kernels.

> > >...
> > > > but in the end, it needs to also be easy and painfree to the people
> > > > involved, and also make sure that any added workflow doesn't require
> > > > even *more* load and expertise on the already often overworked 
> > > > maintainers..
> > > 
> > > Experienced BTS users and developers. Please, correct me if i'm wrong.
> > > At least e-mail part of Debian's BTS and whole idea of it is *exactly*
> > > what is needed. Bugzilla fans, you can still use you useless pet,
> > > because Debian developers have done things, to track and e-mail states
> > > of bugs: <http://permalink.gmane.org/gmane.linux.debian.devel.kernel/29736>
> > >...
> > 
> > "useless pet"?
> > Be serious.
> > How many open source projects use Bugzilla and how many use the Debian BTS?
> > And then start thinking about why the "useless pet" has so many more 
> > user...
> 
> I might be stupid, but i faced it. On my amd64 512M laptop i *cannot* run
> mozillka any more, for example!

Why not?

> And i don't care, because Linus said his
> opinion and i fully agree with him.

Did Linus state he would actually actively use a Debian BTS?
If not, then there's no advantage.

> > The Debian BTS requires you to either write emails with control messages 
> > or generating control messages with external tools.
> 
> Awesome!!! Are you wrote this reply to me by 
> 
> > In Bugzilla the same works through a web interface.
> 
> web interface? If you did .........</dev/random dd bs=1 count=13.....
> Actually you didn't ;)

There's a difference between a discussion email and a control message in 
a fixed format.

> > I know both the Debian BTS and Bugzilla and although they are quite 
> > different they both are reasonable tools for their purpose.
> 
> As you just might have seen here, i was talking about organizing,
> tracking, hopefully saving and redirecting useful main power. And i don't
> bother search e-mails i saw about Bugzilla's BD from many other prominent
> developers. I just know that, not from my dream or physical vacuum.
> 
> Basic concept of Debian BTS is what i've discovered after many useless
> hours i spent in Bugzilla. And this is mainly because of one basic
> important thing, that nobody acknowledged (for newbies, like me):
> 
> * E-Mail with useful MUAs, after it got reliable delivery MTAs with qmail
>   (or exim) is the main communication toolset.
>...

How do people sell and buy goods at eBay?
eBay has a "do everything through the web interface plus notification 
emails" quite similar to Bugzilla.

Or Wikis?
Or Blogs?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: This is [Re:] How to improve the quality of the kernel[?].

[Stefan Richter]

Oleg Verych wrote:
> On Tue, Jun 19, 2007 at 02:48:55PM +0200, Adrian Bunk wrote:
>> The -mm kernel already implements what your proposed PTS would do.
...
>> Plus it gives testers more or less all patches currently pending 
>> inclusion into Linus' tree in one kernel they can test.
> 
> Crazy development{0}. Somebody knows, that comprehensively testing
> hibernation is their thing. I don't care about it, i care about foo, bar.
> Thus i can apply for example lguest patches and implement and test new
> asm-offset replacement, *easily*.

That's right.  But the production of subsystem test patchkits is
volunteer work which will be hard to unify.

I'm not saying it's impossible to reach some degree of organized
production of test patchkits; after all we already have some
standardization regarding patch submission which is volunteer work too.
-- 
Stefan Richter
-=====-=-=== -==- =--==
http://arcgraph.de/sr/

Re: This is [Re:] How to improve the quality of the kernel[?].

[Oleg Verych]

* Date: Tue, 19 Jun 2007 17:08:13 +0200
>
>> Crazy development{0}. Somebody knows, that comprehensively testing
>> hibernation is their thing. I don't care about it, i care about foo, bar.
>> Thus i can apply for example lguest patches and implement and test new
>> asm-offset replacement, *easily*.
>
> That's right.  But the production of subsystem test patchkits is
> volunteer work which will be hard to unify.
>
> I'm not saying it's impossible to reach some degree of organized
> production of test patchkits; after all we already have some
> standardization regarding patch submission which is volunteer work too.

But still there's no one opinion about against what tree to base the
patch. For somebody it's Linus's mainline, for somebody it's bleeding
edge -mm. And there will be no one.

Thus, particular patch entry might have as -mm as Linus's re-based
versions or (as Adrian noted) VFS.asof02-07-2007 FANCYFS. For example,
Rusty did that, after somebody asked him to have not only -mm lguest
version. So, for really intrusive feature/patch (and not
in-middle-development, Adrian) author can have a version (with git
branch, patch directory or something).

Counter-example: Scheduler patches are extraordinary with large
threads or replies, but that is (one of) classical release-early and
often. Proposed bureaucracy doesn't apply ;)
____

Re: This is [Re:] How to improve the quality of the kernel[?].

[Linus Torvalds]

On Tue, 19 Jun 2007, Adrian Bunk wrote:
> 
> The goal is to get all patches for a maintained subsystem submitted to 
> Linus by the maintainer.

Well, to be honest, I've actually over the years tried to have a policy of 
*never* really having black-and-white policies.

The fact is, some maintainers are excellent. All the relevant patches 
*already* effectively go through them.

But at the same time, other maintainers are less than active, and some 
areas aren't clearly maintained at all. 

Also, being a maintainer often means that you are busy and spend a lot of 
time talking to *people* - it doesn't necessarily mean that you actually 
have the hardware and can test things, nor does it necessarily mean that 
you know every detail. 

So I point out in Documentation/ManagementStyle (which is written very 
much tongue-in-cheek, but at the same time it's really *true*) that 
maintainership is often about recognizing people who just know *better* 
than you!

> The -mm kernel already implements what your proposed PTS would do.
> 
> Plus it gives testers more or less all patches currently pending 
> inclusion into Linus' tree in one kernel they can test.
> 
> The problem are more social problems like patches Andrew has never heard 
> of before getting into Linus' tree during the merge window.

Not really. The "problem" boils down to this:

	[torvalds@woody linux]$ git-rev-list --all --since=100.days.ago | wc -l
	7147
	[torvalds@woody linux]$ git-rev-list --no-merges --all --since=100.days.ago | wc -l
	6768

ie over the last hundred days, we have averaged over 70 changes per day, 
and even ignoring merges and only looking at "pure patches" we have more 
than an average of 65 patches per day. Every day. Day in and day out.

That translates to five hundred commits a week, two _thousand_ commits per 
month, and 25 thousand commits per year. As a fairly constant stream.

Will mistakes happen? Hell *yes*. 

And I'd argue that any flow that tries to "guarantee" that mistakes don't 
happen is broken. It's a sure-fire way to just frustrate people, simply 
because it assumes a level of perfection in maintainers and developers 
that isn't possible.

The accepted industry standard for bug counts is basically one bug per a 
thousand lines of code. And that's for released, *debugged* code. 

Yes, we should aim higher. Obviously. Let's say that we aim for 0.1 bugs 
per KLOC, and that we actually aim for that not just in _released_ code, 
but in patches.

What does that mean?

Do the math:

	git log -M -p --all --since=100.days.ago | grep '^+' | wc -l

That basically takes the last one hundred days of development, shows it 
all as patches, and just counts the "new" lines. It takes about ten 
seconds to run, and returns 517252 for me right now.

That's *over*half*a*million* lines added or changed!

And even with the expectation that we do ten times better than what is 
often quoted as an industry average, and even with the expectation that 
this is already fully debugged code, that's at least 50 bugs in the last 
one hundred days.

Yeah, we can be even more stringent, and actually subtract the number of 
lines _removed_ (274930), and assume that only *new* code contains bugs, 
and that's still just under a quarter million purely *added* lines, and 
maybe we'd expect just new 24 bugs in the last 100 days.

[ Argument: some of the old code also contained bugs, so the lines added 
  to replace it balance out. Counter-argument: new code is less well 
  tested by *definition* than old code, so.. Counter-counter-argument: the 
  new code was often added to _fix_ a bug, so the code removed had an even 
  _higher_ bug rate than normal code.. 

  End result? We don't know. This is all just food for thought. ]

So here's the deal: even by the most *stringent* reasonable rules, we add 
a new bug every four days. That's just something that people need to 
accept. The people who say "we must never introduce a regression" aren't 
living on planet earth, they are living in some wonderful world of 
Blarney, where mistakes don't happen, developers are perfect, hardware is 
perfect, and maintainers always catch things.

> The problem is that most problems don't occur on one well-defined 
> kind of hardware - patches often break in exactly the areas the patch
> author expected no problems in.

Note that the industry-standard 1-bug-per-kloc thing has nothing to do 
with hardware. Somebody earlier in this thread (or one of the related 
ones) said that "git bisect is only valid for bugs that happen due to 
hardware issues", which is just totally *ludicrous*.

Yes, hardware makes it harder to test, but even *without* any hardware- 
specific issues, bugs happen. The developer just didn't happen to trigger 
the condition, or didn't happen to notice it when he *did* trigger it.

So don't go overboard about "hardware". Yes, hardware-specific issues have 
their own set of problems, and yes, drivers have a much higher incidence 
of bugs per KLOC, but in the end, even *without* that, you'd still have to 
face the music. Even for stuff that isn't drivers.

So this whole *notion* that you can get it right the first time is 
*insane*.

We should aim for doing well, yes.

But quite frankly, anybody who aims for "perfect" without taking reality 
into account is just not realistic. And if that's part of the goal of some 
"new process", then I'm not even interested in listening to people discuss 
it.

If this plan cannot take reality into account, please stop Cc'ing me. I'm 
simply not interested.

Any process that tries to "guarantee" that regressions don't happen is 
crap. Any process that tries to "guarantee" that we release only kernels 
without bugs can go screw itself. There's one thing I _can_ guarantee, and 
that's as long as we add a quarter million new lines per 100 days (and 
change another quarter million lines), we will have new bugs.

No ifs, buts or maybe's about it.

The process should aim for making them *fewer*. But any process that aims 
for total eradication of new bugs will result in one thing, and one thign 
only: we won't be getting any actual work done.

The only way to guarantee no regressions is to make no progress.

		Linus

Re: This is [Re:] How to improve the quality of the kernel[?].

[Oleg Verych]

Linus,

On Tue, Jun 19, 2007 at 08:01:19AM -0700, Linus Torvalds wrote:
> 
> 
> On Tue, 19 Jun 2007, Adrian Bunk wrote:
> > 
> > The goal is to get all patches for a maintained subsystem submitted to 
> > Linus by the maintainer.

Nice quote. I'm trying to make proposition/convince Adrian, who is in
opposition, but whole thread gets just like obeying his extreme POV...
 
> But quite frankly, anybody who aims for "perfect" without taking reality 
> into account is just not realistic. And if that's part of the goal of some 
> "new process", then I'm not even interested in listening to people discuss 
> it.

I'm proposing kind of smart tracking, summarized before. I'm not an
idealist, doing manual work. Making tools -- is what i've picked up from
one of your mails. Thus hope of having more opinions on that.

> If this plan cannot take reality into account, please stop Cc'ing me. I'm 
> simply not interested.

This one is last at least from me. Sorry for taking you time.
____

Re: This is [Re:] How to improve the quality of the kernel[?].

[Linus Torvalds]

On Tue, 19 Jun 2007, Oleg Verych wrote:
> 
> I'm proposing kind of smart tracking, summarized before. I'm not an
> idealist, doing manual work. Making tools -- is what i've picked up from
> one of your mails. Thus hope of having more opinions on that.

Don't get me wrong, I wasn't actually responing to you personally, I was 
actually responding mostly to the tone of this thread.

So I was responding to things like the example from Bartlomiej about 
missed opportunity for taking developer review into account (and btw, I 
think a little public shaming might not be a bad idea - I believe more in 
*social* rules than in *technical* rules), and I'm responding to some of 
the commentary by Adrian and others about "no regressions *ever*".

These are things we can *wish* for. But the fact that we migth wish for 
them doesn't actually mean that they are really good ideas to aim for in 
practice. 

Let me put it another way: a few weeks ago there was this big news story 
in the New York Times about how "forgetting" is a very essential part 
about remembering, and people passed this around as if it was a big 
revelation. People think that people with good memories have a "good 
thing".

And personally, I was like "Duh". 

Good memory is not about remembering everything. Good memory is about 
forgetting the irrelevant, so that the important stuff stands out and you 
*can* remember it. But the big deal is that yes, you have to forget stuff, 
and that means that you *will* miss details - but you'll hopefully miss 
the stuff you don't care for. The keyword being "hopefully". It works most 
of the time, but we all know we've sometimes been able to forget a detail 
that turned out to be crucial after all.

So the *stupid* response to that is "we should remember everything". It 
misses the point. Yes, we sometimes forget even important details, but 
it's *so* important to forget details, that the fact that our brains 
occasionally forget things we later ended up needing is still *much* 
preferable to trying to remember everything.

The same tends to be true of bug hunting, and regression tracking. 

There's a lot of "noise" there. We'll never get perfect, and I'll argue 
that if we don't have a system that tries to actively *remove* noise, 
we'll just be overwhelmed. But that _inevitably_ means that sometimes 
there was actually a signal in the noise that we ended up removing, 
because nobody saw it as anything but noise. 

So I think people should concentrate on turning "noise" into "clear 
signal", but at the same time remember that that inevitably is a "lossy" 
transformation, and just accept the fact that it will mean that we 
occasionally make "mistakes". 

This is why I've been advocating bugzilla "forget" stuff, for example. I 
tend to see bugzilla as a place where noise accumulates, rather than a 
place where noise is made into a signal. 

Which gets my to the real issue I have: the notion of having a process for 
_tracking_ all the information is actually totally counter-productive, if 
a big part of the process isn't also about throwing noise away.

We don't want to "save" all the crud. I don't want "smart tracking" to 
keep track of everything. I want "smart forgetting", so that we are only 
left with the major signal - the stuff that matters. 

		Linus

Re: This is [Re:] How to improve the quality of the kernel[?].

[Natalie Protasevich]

On 6/19/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Tue, 19 Jun 2007, Oleg Verych wrote:
> >
> > I'm proposing kind of smart tracking, summarized before. I'm not an
> > idealist, doing manual work. Making tools -- is what i've picked up from
> > one of your mails. Thus hope of having more opinions on that.
>
> Don't get me wrong, I wasn't actually responing to you personally, I was
> actually responding mostly to the tone of this thread.
>
> So I was responding to things like the example from Bartlomiej about
> missed opportunity for taking developer review into account (and btw, I
> think a little public shaming might not be a bad idea - I believe more in
> *social* rules than in *technical* rules), and I'm responding to some of
> the commentary by Adrian and others about "no regressions *ever*".
>
> These are things we can *wish* for. But the fact that we migth wish for
> them doesn't actually mean that they are really good ideas to aim for in
> practice.
>
> Let me put it another way: a few weeks ago there was this big news story
> in the New York Times about how "forgetting" is a very essential part
> about remembering, and people passed this around as if it was a big
> revelation. People think that people with good memories have a "good
> thing".
>
> And personally, I was like "Duh".
>
> Good memory is not about remembering everything. Good memory is about
> forgetting the irrelevant, so that the important stuff stands out and you
> *can* remember it. But the big deal is that yes, you have to forget stuff,
> and that means that you *will* miss details - but you'll hopefully miss
> the stuff you don't care for. The keyword being "hopefully". It works most
> of the time, but we all know we've sometimes been able to forget a detail
> that turned out to be crucial after all.
>
> So the *stupid* response to that is "we should remember everything". It
> misses the point. Yes, we sometimes forget even important details, but
> it's *so* important to forget details, that the fact that our brains
> occasionally forget things we later ended up needing is still *much*
> preferable to trying to remember everything.
>
> The same tends to be true of bug hunting, and regression tracking.
>
> There's a lot of "noise" there. We'll never get perfect, and I'll argue
> that if we don't have a system that tries to actively *remove* noise,
> we'll just be overwhelmed. But that _inevitably_ means that sometimes
> there was actually a signal in the noise that we ended up removing,
> because nobody saw it as anything but noise.
>
> So I think people should concentrate on turning "noise" into "clear
> signal", but at the same time remember that that inevitably is a "lossy"
> transformation, and just accept the fact that it will mean that we
> occasionally make "mistakes".

This is the most crucial point so far in my opinion.
Well, not only people who report bugs are smart - they are curious,
enthusiastic, and passionate about their system, and job, hobby -
whatever linux means to them. They often do own investigations, give
lots of detail, and often others jump in with "me too" and give even
more detail (and more noise) But real detail that would help in bug
assessment is not there, and needs to be requested in lengthy
exchanges (time wise, since every request takes  hours, days,
months...)
I think  would help to make some attempt to lead them on to giving out
what's important. Cold and impersonal upfront fields and drop-down
menus are taking a lot of noise and heat off the actual report.
Another observation - things like "me too" should be encouraged to
become separate reports because generally only maintainer and people
who work directly on the module can sort out if this is same problem,
and in fact real problems get lost and not accounted for when getting
in wrong buckets this way.
--Natalie
>
> This is why I've been advocating bugzilla "forget" stuff, for example. I
> tend to see bugzilla as a place where noise accumulates, rather than a
> place where noise is made into a signal.
>
> Which gets my to the real issue I have: the notion of having a process for
> _tracking_ all the information is actually totally counter-productive, if
> a big part of the process isn't also about throwing noise away.
>
> We don't want to "save" all the crud. I don't want "smart tracking" to
> keep track of everything. I want "smart forgetting", so that we are only
> left with the major signal - the stuff that matters.
>
>                 Linus
>

Re: This is [Re:] How to improve the quality of the kernel[?].

[Oleg Verych]

* Date: Tue, 19 Jun 2007 10:04:58 -0700 (PDT)
> 
> On Tue, 19 Jun 2007, Oleg Verych wrote:
>> 
>> I'm proposing kind of smart tracking, summarized before. I'm not an
>> idealist, doing manual work. Making tools -- is what i've picked up from
>> one of your mails. Thus hope of having more opinions on that.
>
> Don't get me wrong, I wasn't actually responing to you personally, I was 
> actually responding mostly to the tone of this thread.

By reading only known persons[1]? Fine, it is OK.

But i hope, i did useful statements. In fact, noise reduction stuff WRT
bug reports was before in my analysis of Adrian's POV here (reportbug
tool). Also it showed again, when i've wrote about traces, where testers
(bug reporters) can find test cases, before they will cry (again) about
some issues. I see this, example is bugzilla @ mozilla -- known history.

[1] Noise filtering -- that's obvious for me, after all :)

By not flaming further, i'm just going to try to implement something.
Hopefully my next patch will be usefully smart tracked.

Thanks!
____

Re: This is [Re:] How to improve the quality of the kernel[?].

[Adrian Bunk]

On Tue, Jun 19, 2007 at 10:04:58AM -0700, Linus Torvalds wrote:
>...
> This is why I've been advocating bugzilla "forget" stuff, for example. I 
> tend to see bugzilla as a place where noise accumulates, rather than a 
> place where noise is made into a signal. 
> 
> Which gets my to the real issue I have: the notion of having a process for 
> _tracking_ all the information is actually totally counter-productive, if 
> a big part of the process isn't also about throwing noise away.
> 
> We don't want to "save" all the crud. I don't want "smart tracking" to 
> keep track of everything. I want "smart forgetting", so that we are only 
> left with the major signal - the stuff that matters. 

Even generating the perfect signal is a complete waste of time if 
there's no recipient for the signal...

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: This is [Re:] How to improve the quality of the kernel[?].

[Linus Torvalds]

On Fri, 22 Jun 2007, Adrian Bunk wrote:
> On Tue, Jun 19, 2007 at 10:04:58AM -0700, Linus Torvalds wrote:
> >...
> > This is why I've been advocating bugzilla "forget" stuff, for example. I 
> > tend to see bugzilla as a place where noise accumulates, rather than a 
> > place where noise is made into a signal. 
> > 
> > Which gets my to the real issue I have: the notion of having a process for 
> > _tracking_ all the information is actually totally counter-productive, if 
> > a big part of the process isn't also about throwing noise away.
> > 
> > We don't want to "save" all the crud. I don't want "smart tracking" to 
> > keep track of everything. I want "smart forgetting", so that we are only 
> > left with the major signal - the stuff that matters. 
> 
> Even generating the perfect signal is a complete waste of time if 
> there's no recipient for the signal...

My argument is that *if* we had "more signal, less noise", we'd probably 
get more people looking at it.

In fact, I guarantee that's the case. You may  not be 100% happy with the 
regression list, but every single maintainer/developer I've talked to has 
said they appreciated it and it made it easier (and thus more likely) for 
them to actually look at what the outstanding issues were.

		Linus

Re: This is [Re:] How to improve the quality of the kernel[?].

[Adrian Bunk]

On Thu, Jun 21, 2007 at 04:59:39PM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 22 Jun 2007, Adrian Bunk wrote:
> > On Tue, Jun 19, 2007 at 10:04:58AM -0700, Linus Torvalds wrote:
> > >...
> > > This is why I've been advocating bugzilla "forget" stuff, for example. I 
> > > tend to see bugzilla as a place where noise accumulates, rather than a 
> > > place where noise is made into a signal. 
> > > 
> > > Which gets my to the real issue I have: the notion of having a process for 
> > > _tracking_ all the information is actually totally counter-productive, if 
> > > a big part of the process isn't also about throwing noise away.
> > > 
> > > We don't want to "save" all the crud. I don't want "smart tracking" to 
> > > keep track of everything. I want "smart forgetting", so that we are only 
> > > left with the major signal - the stuff that matters. 
> > 
> > Even generating the perfect signal is a complete waste of time if 
> > there's no recipient for the signal...
> 
> My argument is that *if* we had "more signal, less noise", we'd probably 
> get more people looking at it.
> 
> In fact, I guarantee that's the case. You may  not be 100% happy with the 
> regression list, but every single maintainer/developer I've talked to has 
> said they appreciated it and it made it easier (and thus more likely) for 
> them to actually look at what the outstanding issues were.


The problems are the parts of the kernel without maintainer or with a 
maintainer who is for whatever reason not able to look after bug 
reports.

And you often need someone with a good knowledge of a specific area of 
the kernel for getting a bug fixed.


Let me make an example:

During 2.6.16-rc, I reported a bug (not a regression) in CIFS where I 
had reproducible during big writes to a Samba server after some 100 MBs 
(not a fixed amount of data, but 100% reproducible when transferring 1 GB)
a complete freeze of my computer (no SysRq possible). And there is
nothing more I (or any other submitter) could have given as information -
in fact it even took me several days to isolate CIFS as the source of 
these freezes.

Steve French and Dave Kleikamp told me to try some mount option.

With this option, I got an Oops instead of a freeze.

After they fixed the Oops, it turned out the patch also fixed the 
freeze. The patch went into 2.6.16, and it was therefore fixed
in 2.6.16.

That's one important value of maintainers.

In many other parts of the kernel, my bug report wouldn't have had any 
effect.


We need more maintaners who look after bugs - but where to find them, 
they don't seem to grow on trees?


> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: This is [Re:] How to improve the quality of the kernel[?].

[Adrian Bunk]

On Tue, Jun 19, 2007 at 08:01:19AM -0700, Linus Torvalds wrote:
> On Tue, 19 Jun 2007, Adrian Bunk wrote:
>...
> > The -mm kernel already implements what your proposed PTS would do.
> > 
> > Plus it gives testers more or less all patches currently pending 
> > inclusion into Linus' tree in one kernel they can test.
> > 
> > The problem are more social problems like patches Andrew has never heard 
> > of before getting into Linus' tree during the merge window.
> 
> Not really. The "problem" boils down to this:
> 
> 	[torvalds@woody linux]$ git-rev-list --all --since=100.days.ago | wc -l
> 	7147
> 	[torvalds@woody linux]$ git-rev-list --no-merges --all --since=100.days.ago | wc -l
> 	6768
> 
> ie over the last hundred days, we have averaged over 70 changes per day, 
> and even ignoring merges and only looking at "pure patches" we have more 
> than an average of 65 patches per day. Every day. Day in and day out.
> 
> That translates to five hundred commits a week, two _thousand_ commits per 
> month, and 25 thousand commits per year. As a fairly constant stream.
> 
> Will mistakes happen? Hell *yes*. 
> 
> And I'd argue that any flow that tries to "guarantee" that mistakes don't 
> happen is broken. It's a sure-fire way to just frustrate people, simply 
> because it assumes a level of perfection in maintainers and developers 
> that isn't possible.
> 
> The accepted industry standard for bug counts is basically one bug per a 
> thousand lines of code. And that's for released, *debugged* code. 
> 
> Yes, we should aim higher. Obviously. Let's say that we aim for 0.1 bugs 
> per KLOC, and that we actually aim for that not just in _released_ code, 
> but in patches.
> 
> What does that mean?
> 
> Do the math:
> 
> 	git log -M -p --all --since=100.days.ago | grep '^+' | wc -l
> 
> That basically takes the last one hundred days of development, shows it 
> all as patches, and just counts the "new" lines. It takes about ten 
> seconds to run, and returns 517252 for me right now.
> 
> That's *over*half*a*million* lines added or changed!
> 
> And even with the expectation that we do ten times better than what is 
> often quoted as an industry average, and even with the expectation that 
> this is already fully debugged code, that's at least 50 bugs in the last 
> one hundred days.
> 
> Yeah, we can be even more stringent, and actually subtract the number of 
> lines _removed_ (274930), and assume that only *new* code contains bugs, 
> and that's still just under a quarter million purely *added* lines, and 
> maybe we'd expect just new 24 bugs in the last 100 days.
> 
> [ Argument: some of the old code also contained bugs, so the lines added 
>   to replace it balance out. Counter-argument: new code is less well 
>   tested by *definition* than old code, so.. Counter-counter-argument: the 
>   new code was often added to _fix_ a bug, so the code removed had an even 
>   _higher_ bug rate than normal code.. 
> 
>   End result? We don't know. This is all just food for thought. ]
> 
> So here's the deal: even by the most *stringent* reasonable rules, we add 
> a new bug every four days. That's just something that people need to 
> accept. The people who say "we must never introduce a regression" aren't 
> living on planet earth, they are living in some wonderful world of 
> Blarney, where mistakes don't happen, developers are perfect, hardware is 
> perfect, and maintainers always catch things.
>...

Exactly: We cannot get a regression free or even bug free kernel.
But we could handle the reported regressions (or even the reported bugs) 
better than we do.

Lesson #6:
Get the data.

Some real life numbers from 2.6.21 development:
- 80 days between 2.6.20 and 2.6.21
- 98 post-2.6.20 regression have been reported before 2.6.21 was released
- 15 open post-2.6.20 regression reports at the time of the 2.6.21 release
- 8 open post-2.6.20 regression reports at the time of the 2.6.21 release
    that were reported at least 3 weeks before the 2.6.21 release

This:
- only includes regressions with reasonably usable reports [1] and
- confirmed to be regressions and
- reported by the relatively small number (compared to the complete
  number of Linux users) of -rc testers and
- reported before the release of 2.6.21.

We weren't even able to handle all reported recent regressions in 
2.6.21, and for other bugs our numbers won't be better.

When Dave Jones says that for a kernel for a new RHEL release that is 
based on a "stable" upstream kernel they spend 3 months only for shaking 
out bugs in the kernel that's IMHO a good description of our "stable" 
kernels.

I'm not claiming the kernel could become bug-free, but aiming at being 
able to handle all incoming bug reports is IMHO a worthwhile and not 
completely unrealistic goal with benefits for all Linux users (and the 
overall image of Linux).

Currently, we are light years away from this goal.

> 		Linus

cu
Adrian

[1] submitter has given all information requested

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: This is [Re:] How to improve the quality of the kernel[?].

[Don Armstrong]

On Tue, 19 Jun 2007, Oleg Verych wrote:
> * From: Linus Torvalds
> * Newsgroups: gmane.linux.kernel
> * Date: Mon, 18 Jun 2007 17:09:37 -0700 (PDT)
>
> > I do agree. It _sounds_ like a great idea to try to control the
> > flow of patches better,
> 
> There were some ideas, i will try to summarize:
> 
> * New Patches (or sets) need tracking, review, testing
> 
>   Zero (tracking) done by sending (To, or bcc) [RFC] patch to something
>   like submit@pts.e-mail.example.com (like BTS now). Notifications will
>   be sent to intrested maintainers (if meta-information is ok) or testers
>   (see below)

The BTS, while fairly good at tracking issues for distributions made
up of thousands of packages (like Debian), is rather suboptimal for
dealing with the workflow of a single (relatively) monolithic entity
like the linux kernel.

Since the ultimate goal is presumably to apply a patch to a git tree,
some sort of system which is built directly on top of git (or
intimately intertwined) is probably required. Some of the metrics that
the BTS uses, like the easy ability to use mail to control bugs may be
useful to incorporate, but I'd be rather surprised if it could be made
to work with the kernel developer's workflow as it exists now.

It may be useful for whoever ends up designing the patch system to
take a glimpse at how it's done in debbugs, but since I don't know how
the workflow works now, and how people want to have it work in the
end, I can't tell you what features from debbugs would be useful to
use.

Finally, at the end of the day, my own time and effort (and the
primary direction of debbugs development) is aimed at supporting the
primary user of debbugs, the Debian project. People who understand (or
want to understand) the linux kernel team's workflow are the ones who
are going to need to do the heavy lifting here.


Don Armstrong
 
-- 
N: Why should I believe that?"
B: Because it's a fact."
N: Fact?"
B: F, A, C, T... fact"
N: So you're saying that I should believe it because it's true. 
   That's your argument?
B: It IS true.
-- "Ploy" http://www.mediacampaign.org/multimedia/Ploy.MPG

http://www.donarmstrong.com              http://rzlab.ucr.edu

RE: How to improve the quality of the kernel?

[Fortier,Vincent [Montreal]]

> -----Message d'origine-----
> De : Natalie Protasevich [mailto:protasnb@gmail.com] 
> Envoyé : 18 juin 2007 18:56
> 
> On 6/18/07, Martin Bligh <mbligh@mbligh.org> wrote:
> > >> > So if you make changes to random-driver.c you can do `git-log 
> > >> > random-driver.c|grep Tested-by" to find people who can test your 
> > >> > changes for you.
> > >>
> > >> You would'nt even need to search in GIT.  Maybie even when ever a 
> > >> patchset is being proposed a mail could be sent to appropriate 
> > >> hardware/or feature pseudo-auto-generated mailing-list?
> > >>
> > >> On lkml I mostly try to follow patches/bugs associated with 
> > >> hardware I use.  Why not try to automate the process and get more testers in?
> > >>
> > >
> > > I think this is an excellent point. One data point could be a field 
> > > in bugzilla to input the hardware information. Simple query can 
> > > select common hardware and platform. So far it's not working when 
> > > hardware is just mentioned in the text part.
> >
> > if it's free text it'll be useless for search ... I suppose we could 
> > do drop-downs for architecture at least? Not sure much beyond that 
> > would work ... *possibly* the common drivers, but I don't think we'd 
> > get enough coverage for it to be of use.
> >
> How about several buckets for model/BIOS version/chipset 
> etc., at least optional, and some will be relevant some not 
> for particular cases. But at least people will make an 
> attempt to collect such data from their system, boards, etc.

How about an easy way to send multiple hardware profiles to your bugzilla user account simultaniously linked to an online pciutils database and/or an hardware list database similar to overclocking web sites and why not even with a link to the git repository when possible?

A some sort of really usefull "send your profile" of RHN that would link the driver with the discovered hardware and add you to appropriate mailing lists to test patches/help reproducing & solving problems/etc.

In the end plenty of statistics and hardware compatibility list could be made.  For example, that would make my life easier knowing what level of compatibility Linux can offer for old HP9000 K-boxes that we still have running at the office and presumably get people to contact to get help?

- vin

Re: How to improve the quality of the kernel?

[Natalie Protasevich]

On 6/18/07, Fortier,Vincent [Montreal] <Vincent.Fortier1@ec.gc.ca> wrote:
> > -----Message d'origine-----
> > De : Natalie Protasevich [mailto:protasnb@gmail.com]
> > Envoyé : 18 juin 2007 18:56
> >
> > On 6/18/07, Martin Bligh <mbligh@mbligh.org> wrote:
> > > >> > So if you make changes to random-driver.c you can do `git-log
> > > >> > random-driver.c|grep Tested-by" to find people who can test your
> > > >> > changes for you.
> > > >>
> > > >> You would'nt even need to search in GIT.  Maybie even when ever a
> > > >> patchset is being proposed a mail could be sent to appropriate
> > > >> hardware/or feature pseudo-auto-generated mailing-list?
> > > >>
> > > >> On lkml I mostly try to follow patches/bugs associated with
> > > >> hardware I use.  Why not try to automate the process and get more testers in?
> > > >>
> > > >
> > > > I think this is an excellent point. One data point could be a field
> > > > in bugzilla to input the hardware information. Simple query can
> > > > select common hardware and platform. So far it's not working when
> > > > hardware is just mentioned in the text part.
> > >
> > > if it's free text it'll be useless for search ... I suppose we could
> > > do drop-downs for architecture at least? Not sure much beyond that
> > > would work ... *possibly* the common drivers, but I don't think we'd
> > > get enough coverage for it to be of use.
> > >
> > How about several buckets for model/BIOS version/chipset
> > etc., at least optional, and some will be relevant some not
> > for particular cases. But at least people will make an
> > attempt to collect such data from their system, boards, etc.
>
> How about an easy way to send multiple hardware profiles to your bugzilla user account simultaniously linked to an online pciutils database and/or an hardware list database similar to overclocking web sites and why not even with a link to the git repository when possible?
>
> A some sort of really usefull "send your profile" of RHN that would link the driver with the discovered hardware and add you to appropriate mailing lists to test patches/help reproducing & solving problems/etc.
>
> In the end plenty of statistics and hardware compatibility list could be made.  For example, that would make my life easier knowing what level of compatibility Linux can offer for old HP9000 K-boxes that we still have running at the office and presumably get people to contact to get help?

This is definitely something that can be done (and should) - well,
especially having ability search by certain criteria - then all sorts
of statistics and databases can be created.
Everything that helps  to find a way to work on a patch and to test
easier should be done to make bug fixing easier and even possible.
Often times the most knowledgeable people are not able to make quick
fix just because there is no way to reproduce the case or get access
to HW.

--Natalie

Re: How to improve the quality of the kernel?

[Stefan Richter]

Natalie Protasevich wrote:
> On 6/18/07, Fortier,Vincent [Montreal] <Vincent.Fortier1@ec.gc.ca> wrote:
[...]
>> In the end plenty of statistics and hardware compatibility list
>> could be made.  For example, that would make my life easier knowing
>> what level of compatibility Linux can offer for old HP9000 K-boxes
>> that we still have running at the office and presumably get people
>> to contact to get help?
> 
> This is definitely something that can be done (and should) - well,
> especially having ability search by certain criteria - then all sorts
> of statistics and databases can be created.

Hardware Compatibility Lists/ Databases already exist, for driver
subsystems, for distributions...

Some issues with those databases are:

  - Users typically can only test one specific combination of a
    hardware collection and software collection, at one or a few points
    in time.

  - Users have difficulties or don't have the means to identify chip
    revisions, used protocols etc.

  - The databases are typically not conceived to serve additional
    purposes like bidirectional contact between developer and user.

These issues notwithstanding, these databases are already highly useful
both for endusers and for developers.  That's why they exist.

> Everything that helps  to find a way to work on a patch and to test
> easier should be done to make bug fixing easier and even possible.
> Often times the most knowledgeable people are not able to make quick
> fix just because there is no way to reproduce the case or get access
> to HW.

As has been mentioned elsewhere in the thread,

  - bug---hardware associations are sometimes difficult or impossible
    to make.  For example, the x86-64 platform maintainers are bothered
    with "x86-64 bugs" which turn out to be driver bugs on all
    platforms.

    (We want details descriptions of the hardware environment in a bug
    report, but this means we must be able to handle the flood of
    false positives in bug---hardware associations, i.e. successively
    narrow down which parts of the hardware/software combo are actually
    affected, and what other combinations could be affected too.)

  - Patch---hardware associations, especially for preemptive regression
    tests, are virtually impossible to make.  Murphy says that the
    regression will hit hardware which the patch submitter or forwarder
    thought could never be affected by the patch.

Of course, /sensible/ patch---hardware associations are (1) to try out
fixes for known issues with a specific hardware, (2) to test that a
cleanup patch or refactoring patch or API changing patch to a driver of
very specific hardware ( = a single type or few types with little
variance) does not introduce regressions for this hardware.
-- 
Stefan Richter
-=====-=-=== -==- =--==
http://arcgraph.de/sr/

Re: How to improve the quality of the kernel?

[Rafael J. Wysocki]

On Sunday, 17 June 2007 23:49, Bartlomiej Zolnierkiewicz wrote:
> On Sunday 17 June 2007, Andrew Morton wrote:
> > On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> wrote:
[--snip--]
> > 
> > yup, Reviewed-by: is good and I do think we should start adopting it,
> > although I haven't thought through exactly how.
> 
> Adding Reviewed-by for reviews which highlighted real issues is obvious
> (with more detailed credits for noticed problems in the patch description).

Suppose you have modified the patch as a result of a review and you post the
modified version.  Is that still right to put "Reviewed-by" into it?
Personally, I don't think so, because that suggests that this particular
version of the patch has been reviewed and not the previous one.

> Also when somebody reviewed your patch but the discussions it turned out
> that the patch is valid - the review itself was still valuable so it would
> be appropriate to credit the reviewer by adding Reviewed-by:.

Yes, IMO in such a case it would be appropriate to do that.

Also, the review need not lead to any negative comments from the reviewer,
but in that case it's also appropriate to add a "Reviewed-by" to the patch.

Generally, if someone comments my patches, I add his/her address to the next
version's CC list, which sort of documents that the reviewer was involved.
Then, if the reviewer ACKs the patch, that will be recorded.

I think that for "Reviewed-by" to work correctly, we ought to have a two-stage
process of accepting patches, where in the first stage the patch is reviewed
and if there are no objections, the "Reviewed-by" (or "Acked-by") records are
added to it in the next stage (the patch itself remains unmodified).

> > On my darker days I consider treating a Reviewed-by: as a prerequisite for
> > merging.  I suspect that would really get the feathers flying.
> 
> Easy to workaround by a friendly mine "Reviewed-by:" for yours "Reviewed-by:"
> deals (without any _proper_ review being done in reality)... ;)
> 
> > > I also encourage other maintainers/developers to pay more attention to
> > > adding "Acked-by"/"Reviewed-by" tags and crediting reviewers.  I hope
> > > that maintainers will promote changes that have been reviewed by others
> > > by giving them priority over other ones (if the changes are on more-or-less
> > > the same importance level of course, you get the idea).
> > > 
> > > Now what to do with people who ignore reviews and/or have rather high
> > > regressions/patches ratio?
> > 
> > Ignoring a review would be a wildly wrong thing to do.  It's so unusual
> > that I'd be suspecting a lost email or an i-sent-the-wrong-patch.
> 
> It is not unusual et all.  I mean patches which affect code in such way
> that it is difficult to prove it's (in)correctness without doing time
> consuming audit.
> 
> ie. lets imagine doing a small patch affecting many drivers - you've tested
> it quickly on your driver/hardware, then you skip the part of verifying
> correctness of new code in other drivers and just push the patch
> 
> As a patch author you can either assume "works for me" and push the patch
> or do the audit (requires good understanding of the changed code and could
> be time consuming).  It is usually quite easy to find out which approach
> the author has choosen - the very sparse patch description combined with
> the changes in code behavior not mentioned in the patch description should
> raise the red flag. :)

First of all, the author should have a good understanding of what he's doing
and why.  If there are any doubts with respect to that, the patch is likely to
introduce bugs.

This also depends on who will be handling the bug reports related to the patch.
If that will be the patch author, then so be it. ;-)

> As a reviewer having enough knowledge in the area of code affected by patch
> you can see the potential problems but you can't prove them without doing
> the time consuming part.  You may try to NACK the patch if you have enough
> power but you will end up being bypassed by not proving incorrectness of
> the patch (not to mention that developer will feel bad about you NACKing
> his patch).

Well, IMHO, the author of the patch should convince _you_ that the patch is
correct, not the other way around.  If you have doubts and make him think
twice of the code and he still can't prove his point, this means that he
doesn't understand what he's doing well enough.

> Now the funny thing is that despite the fact that audit takes 
> more time/knowledge then making the patch you will end up with zero credit
> if patch turns out to be (luckily) correct.  Even if you find out issues
> and report them you are still on mercy of author for being credited so
> from personal POV you are much better to wait and fix issues after they
> hit mainline kernel.  You have to choose between being a good citizen and
> preventing kernel regressions or being bastard and getting the credit. ;)

Unless you are the poor soul having to handle bug reports related to the
problem.
 
> If you happen to be maintainer of the affected code the choice is similar
> with more pros for letting the patch in especially if you can't afford the
> time to do audit (and by being maintainer you are guaranteed to be heavily
> time constrained).
> 
> I hope this makes people see the importance of proper review and proper
> recognition of reviewers in preventing kernel regressions.
> 
> > As for high regressions/patches ratio: that'll be hard to calculate and
> > tends to be dependent upon the code which is being altered rather than who
> > is doing the altering: some stuff is just fragile, for various reasons.
> > 
> > One ratio which we might want to have a think about is the patches-sent
> > versus reviews-done ratio ;)
> 
> Sounds like a good idea.
> 
> > > I think that we should have info about regressions integrated into SCM,
> > > i.e. in git we should have optional "fixes-commit" tag and we should be
> > > able to do some reverse data colletion.   This feature combined with
> > > "Author:" info after some time should give us some very interesting
> > > statistics (Top Ten "Regressors").  It wouldn't be ideal (ie. we need some
> > > patches threshold to filter out people with 1 patch and >= 1 regression(s),
> > > we need to remember that some code areas are more difficult than the others
> > > and that patches are not equal per se etc.) however I believe than making it
> > > into Top Ten "Regressors" should give the winners some motivation to improve
> > > their work ethic.  Well, in the worst case we would just get some extra
> > > trivial/documentation patches. ;-)
> > 
> > We of course do want to minimise the amount of overhead for each developer. 
> > I'm a strong believer in specialisation: rather than requiring that *every*
> > developer/maintainer integrate new steps in their processes it would be
> > better to allow them to proceed in a close-to-usual fashion and to provide
> > for a specialist person (or team) to do the sorts of things which you're
> > thinking about.
> 
> Makes sense... however we need to educate each and every developer about
> importance of the code review and proper recognition of reviewers.

I don't think that the education alone will be enough.  IMO we need to have a
system that promotes the reviewing of code.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: How to improve the quality of the kernel?

[Bartlomiej Zolnierkiewicz]

On Monday 18 June 2007, Rafael J. Wysocki wrote:
> On Sunday, 17 June 2007 23:49, Bartlomiej Zolnierkiewicz wrote:
> > On Sunday 17 June 2007, Andrew Morton wrote:
> > > On Sun, 17 Jun 2007 20:53:41 +0200 Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> wrote:
> [--snip--]
> > > 
> > > yup, Reviewed-by: is good and I do think we should start adopting it,
> > > although I haven't thought through exactly how.
> > 
> > Adding Reviewed-by for reviews which highlighted real issues is obvious
> > (with more detailed credits for noticed problems in the patch description).
> 
> Suppose you have modified the patch as a result of a review and you post the
> modified version.  Is that still right to put "Reviewed-by" into it?
> Personally, I don't think so, because that suggests that this particular
> version of the patch has been reviewed and not the previous one.

Well, if you got the "fix issues" part right it in the modified version
it shouldn't really matter. ;)

But yes, we may wait with adding "Reviewed-by" after the modified patch
has been posted and reviewed.

> > Also when somebody reviewed your patch but the discussions it turned out
> > that the patch is valid - the review itself was still valuable so it would
> > be appropriate to credit the reviewer by adding Reviewed-by:.
> 
> Yes, IMO in such a case it would be appropriate to do that.
> 
> Also, the review need not lead to any negative comments from the reviewer,
> but in that case it's also appropriate to add a "Reviewed-by" to the patch.
> 
> Generally, if someone comments my patches, I add his/her address to the next
> version's CC list, which sort of documents that the reviewer was involved.
> Then, if the reviewer ACKs the patch, that will be recorded.

Same approach here.

> I think that for "Reviewed-by" to work correctly, we ought to have a two-stage
> process of accepting patches, where in the first stage the patch is reviewed
> and if there are no objections, the "Reviewed-by" (or "Acked-by") records are
> added to it in the next stage (the patch itself remains unmodified).
> 
> > > On my darker days I consider treating a Reviewed-by: as a prerequisite for
> > > merging.  I suspect that would really get the feathers flying.
> > 
> > Easy to workaround by a friendly mine "Reviewed-by:" for yours "Reviewed-by:"
> > deals (without any _proper_ review being done in reality)... ;)
> > 
> > > > I also encourage other maintainers/developers to pay more attention to
> > > > adding "Acked-by"/"Reviewed-by" tags and crediting reviewers.  I hope
> > > > that maintainers will promote changes that have been reviewed by others
> > > > by giving them priority over other ones (if the changes are on more-or-less
> > > > the same importance level of course, you get the idea).
> > > > 
> > > > Now what to do with people who ignore reviews and/or have rather high
> > > > regressions/patches ratio?
> > > 
> > > Ignoring a review would be a wildly wrong thing to do.  It's so unusual
> > > that I'd be suspecting a lost email or an i-sent-the-wrong-patch.
> > 
> > It is not unusual et all.  I mean patches which affect code in such way
> > that it is difficult to prove it's (in)correctness without doing time
> > consuming audit.
> > 
> > ie. lets imagine doing a small patch affecting many drivers - you've tested
> > it quickly on your driver/hardware, then you skip the part of verifying
> > correctness of new code in other drivers and just push the patch
> > 
> > As a patch author you can either assume "works for me" and push the patch
> > or do the audit (requires good understanding of the changed code and could
> > be time consuming).  It is usually quite easy to find out which approach
> > the author has choosen - the very sparse patch description combined with
> > the changes in code behavior not mentioned in the patch description should
> > raise the red flag. :)
> 
> First of all, the author should have a good understanding of what he's doing
> and why.  If there are any doubts with respect to that, the patch is likely to
> introduce bugs.
> 
> This also depends on who will be handling the bug reports related to the patch.
> If that will be the patch author, then so be it. ;-)

The problem is that usually Andrew/Adrian/Michal would also be involved.

> > As a reviewer having enough knowledge in the area of code affected by patch
> > you can see the potential problems but you can't prove them without doing
> > the time consuming part.  You may try to NACK the patch if you have enough
> > power but you will end up being bypassed by not proving incorrectness of
> > the patch (not to mention that developer will feel bad about you NACKing
> > his patch).
> 
> Well, IMHO, the author of the patch should convince _you_ that the patch is
> correct, not the other way around.  If you have doubts and make him think
> twice of the code and he still can't prove his point, this means that he
> doesn't understand what he's doing well enough.

This is a nice theory, practise differs greatly.

Sometimes you are not in position to prevent suspicious patches from being
merged and sometimes you just don't want to do it for various reasons (not
discouring the developer and preventing his personal vendetta against you :).

> > Now the funny thing is that despite the fact that audit takes 
> > more time/knowledge then making the patch you will end up with zero credit
> > if patch turns out to be (luckily) correct.  Even if you find out issues
> > and report them you are still on mercy of author for being credited so
> > from personal POV you are much better to wait and fix issues after they
> > hit mainline kernel.  You have to choose between being a good citizen and
> > preventing kernel regressions or being bastard and getting the credit. ;)
> 
> Unless you are the poor soul having to handle bug reports related to the
> problem.
> 
> > If you happen to be maintainer of the affected code the choice is similar
> > with more pros for letting the patch in especially if you can't afford the
> > time to do audit (and by being maintainer you are guaranteed to be heavily
> > time constrained).
> > 
> > I hope this makes people see the importance of proper review and proper
> > recognition of reviewers in preventing kernel regressions.
> > 
> > > As for high regressions/patches ratio: that'll be hard to calculate and
> > > tends to be dependent upon the code which is being altered rather than who
> > > is doing the altering: some stuff is just fragile, for various reasons.
> > > 
> > > One ratio which we might want to have a think about is the patches-sent
> > > versus reviews-done ratio ;)
> > 
> > Sounds like a good idea.
> > 
> > > > I think that we should have info about regressions integrated into SCM,
> > > > i.e. in git we should have optional "fixes-commit" tag and we should be
> > > > able to do some reverse data colletion.   This feature combined with
> > > > "Author:" info after some time should give us some very interesting
> > > > statistics (Top Ten "Regressors").  It wouldn't be ideal (ie. we need some
> > > > patches threshold to filter out people with 1 patch and >= 1 regression(s),
> > > > we need to remember that some code areas are more difficult than the others
> > > > and that patches are not equal per se etc.) however I believe than making it
> > > > into Top Ten "Regressors" should give the winners some motivation to improve
> > > > their work ethic.  Well, in the worst case we would just get some extra
> > > > trivial/documentation patches. ;-)
> > > 
> > > We of course do want to minimise the amount of overhead for each developer. 
> > > I'm a strong believer in specialisation: rather than requiring that *every*
> > > developer/maintainer integrate new steps in their processes it would be
> > > better to allow them to proceed in a close-to-usual fashion and to provide
> > > for a specialist person (or team) to do the sorts of things which you're
> > > thinking about.
> > 
> > Makes sense... however we need to educate each and every developer about
> > importance of the code review and proper recognition of reviewers.
> 
> I don't think that the education alone will be enough.  IMO we need to have a
> system that promotes the reviewing of code.

Sure, we need to start somewhere...

Bart

Re: How to improve the quality of the kernel?

[Michal Piotrowski]

On 17/06/07, Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> wrote:
>
> Hi,
>
> On Sunday 17 June 2007, Adrian Bunk wrote:
> > On Sun, Jun 17, 2007 at 03:17:58PM +0200, Michal Piotrowski wrote:
> > > On 17/06/07, Adrian Bunk <bunk@stusta.de> wrote:
> > >...
> > >> Fine with me, but:
> > >>
> > >> There are not so simple cases like big infrastructure patches with
> > >> 20 other patches in the tree depending on it causing a regression, or
> > >> even worse, a big infrastructure patch exposing a latent old bug in some
> > >> completely different area of the kernel.
> > >
> > > It is different case.
> > >
> > > "If the patch introduces a new regression"
> > >
> > > introduces != exposes an old bug
> >
> > My remark was meant as a note "this sentence can't handle all
> > regressions" (and for a user it doesn't matter whether a new
> > regression is introduced or an old regression exposed).
> >
> > It could be we simply agree on this one.  ;-)
> >
> > > Removal of 20 patches will be painful, but sometimes you need to
> > > "choose minor evil to prevent a greater one" [1].
> > >
> > >> And we should be aware that reverting is only a workaround for the real
> > >> problem which lies in our bug handling.
> > >...
> >
> > And this is something I want to emphasize again.
> >
> > How can we make any progress with the real problem and not only the
> > symptoms?
> >
> > There's now much money in the Linux market, and the kernel quality
> > problems might result in real costs in the support of companies like
> > IBM, SGI, Redhat or Novell (plus it harms the Linux image which might
> > result in lower revenues).
> >
> > If [1] this is true, it might even pay pay off for them to each assign
> > X man hours per month of experienced kernel developers to upstream
> > kernel bug handling?
> >
> > This is just a wild thought and it might be nonsense - better
> > suggestions for solving our quality problems would be highly welcome...
>
> IMO we should concentrate more on preventing regressions than on fixing them.
> In the long-term preventing bugs is cheaper than fixing them afterwards.
>
> First let me tell you all a little story...
>
> Over two years ago I've reviewed some _cleanup_ patch and noticed three bugs
> in it (in other words I potentially prevented three regressions).  I also
> asked for more thorough verification of the patch as I suspected that it may
> have more problems.  The author fixed the issues and replied that he hasn't
> done the full verification yet but he doesn't suspect any problems...
>
> Fast forward...
>
> Year later I discover that the final version of the patch hit the mainline.
> I don't remember ever seeing the final version in my mailbox (there are no
> cc: lines in the patch description) and I saw that I'm not credited in the
> patch description.  However the worse part is that it seems that the full
> verification has never been done.  The result?   Regression in the release
> kernel (exactly the issue that I was worried about) which required three
> patches and over a month to be fixed completely.  It seems that a year
> was not enough to get this ~70k _cleanup_ patch fully verified and tested
> (it hit -mm soon before being merged)...
>
> From reviewer's POV: I have invested my time into review, discovered real
> issues and as a reward I got no credit et all and extra frustration from the
> fact that part of my review was forgotten/ignored (the part which resulted in
> real regression in the release kernel)...  Oh and in the past the said
> developer has already been asked (politely in private message) to pay more
> attention to his changes (after I silently fixed some other regression caused
> by his other patch).
>
> But wait there is more, I happend to be the maintainer of the subsystem which
> got directly hit by the issue and I was getting bugreports from the users about
> the problem...  :-)
>
> It wasn't my first/last bad experience as a reviewer... finally I just gave up
> on reviewing other people patches unless they are stricly for IDE subsystem.
>
> The moral of the story is that currently it just doesn't pay off to do
> code reviews.  From personal POV it pays much more to wait until buggy patch
> hits the mainline and then fix the issues yourself (at least you will get
> some credit).  To change this we should put more ephasize on the importance
> of code reviews by "rewarding" people investing their time into reviews
> and "rewarding" developers/maintainers taking reviews seriously.
>
> We should credit reviewers more, sometimes it takes more time/knowledge to
> review the patch than to make it so getting near to zero credit for review
> doesn't sound too attractive.  Hmm, wait it can be worse - your review
> may be ignored... ;-)
>
> From my side I think I'll start adding less formal "Reviewed-by" to IDE
> patches even if the review resulted in no issues being found (in additon to
> explicit "Acked-by" tags and crediting people for finding real issues - which
> I currently always do as a way for showing my appreciation for their work).
>
> I also encourage other maintainers/developers to pay more attention to
> adding "Acked-by"/"Reviewed-by" tags and crediting reviewers.  I hope
> that maintainers will promote changes that have been reviewed by others
> by giving them priority over other ones (if the changes are on more-or-less
> the same importance level of course, you get the idea).

I think that this is a very good idea - especially for large, intrusive patches.
Long {Acked,Reviewed,Signed-off,Tested}-by list will be welcome.

>
> Now what to do with people who ignore reviews and/or have rather high
> regressions/patches ratio?
>
> I think that we should have info about regressions integrated into SCM,
> i.e. in git we should have optional "fixes-commit" tag and we should be
> able to do some reverse data colletion.   This feature combined with
> "Author:" info after some time should give us some very interesting
> statistics (Top Ten "Regressors").  It wouldn't be ideal (ie. we need some
> patches threshold to filter out people with 1 patch and >= 1 regression(s),
> we need to remember that some code areas are more difficult than the others
> and that patches are not equal per se etc.) however I believe than making it
> into Top Ten "Regressors" should give the winners some motivation to improve
> their work ethic.  Well, in the worst case we would just get some extra
> trivial/documentation patches. ;-)
>
> Sorry for a bit chaotic mail but I hope that message is clear.
>
> Thanks,
> Bart
>

Regards,
Michal

-- 
LOG
http://www.stardust.webpages.pl/log/

Re: regression tracking (Re: Linux 2.6.21)

[Martin Bligh]

Linus Torvalds wrote:
> 
> On Thu, 14 Jun 2007, Oleg Verych wrote:
>> I'm seeing this long (198) thread and just have no idea how it has
>> ended (wiki? hand-mailing?).
> 
> I'm hoping it's not "ended".
> 
> IOW, I really don't think we _resolved_ anything, although the work that 
> Adrian started is continuing through the wiki and other people trying to 
> track regressions, and that was obviously something good.
> 
> But I don't think we really know where we want to take this thing in the 
> long run. I think everybody wants a better bug-tracking system, but 
> whether something that makes people satisfied can even be built is open. 
> It sure doesn't seem to exist right now ;)

I know you hate bugzilla ... but at least I can try to make that bit
of the process work better.

The new version just rolled out does have a simple "regression" checkbox
(and you can search on it), which will hopefully help people keep track
of the ones already in bugzilla more easily.

Thanks to Jon T, Dave J et al. for helping to figure out methods and
implement them.

M.

Re: regression tracking (Re: Linux 2.6.21)

[Rafael J. Wysocki]

On Tuesday, 19 June 2007 02:28, Martin Bligh wrote:
> Linus Torvalds wrote:
> > 
> > On Thu, 14 Jun 2007, Oleg Verych wrote:
> >> I'm seeing this long (198) thread and just have no idea how it has
> >> ended (wiki? hand-mailing?).
> > 
> > I'm hoping it's not "ended".
> > 
> > IOW, I really don't think we _resolved_ anything, although the work that 
> > Adrian started is continuing through the wiki and other people trying to 
> > track regressions, and that was obviously something good.
> > 
> > But I don't think we really know where we want to take this thing in the 
> > long run. I think everybody wants a better bug-tracking system, but 
> > whether something that makes people satisfied can even be built is open. 
> > It sure doesn't seem to exist right now ;)
> 
> I know you hate bugzilla ... but at least I can try to make that bit
> of the process work better.
> 
> The new version just rolled out does have a simple "regression" checkbox
> (and you can search on it), which will hopefully help people keep track
> of the ones already in bugzilla more easily.
> 
> Thanks to Jon T, Dave J et al. for helping to figure out methods and
> implement them.

Yes, good work, thanks a lot for it!  The new interface is much better and more
useful.

Greetings,
Rafael


PS
BTW, would that be possible to create the "Hibernation/Suspend" subcategory
of "Power Management" that I asked for some time ago, please? :-)

-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: regression tracking (Re: Linux 2.6.21)

[Martin J. Bligh]

> Yes, good work, thanks a lot for it!  The new interface is much better and more
> useful.
>
> Greetings,
> Rafael
>
>
> PS
> BTW, would that be possible to create the "Hibernation/Suspend" subcategory
> of "Power Management" that I asked for some time ago, please? :-)
>   

Oops. Sorry. Done.

M.

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 19:37, Andi Kleen wrote:
> > My personal experience with bugzilla is that it's very unfriendly to
> > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > debug patches, system information etc., but not for _reporting_ new ones.
> 
> What did you find unfriendly? 

- You are required to select a category and 'component' for your report, which
often is difficult (especially if you're not a kernel expert)
- You need to have a bugzilla account (or to create one, if you don't)
- If you want to add an address to the CC list, it must be known to bugzilla
and there's no (obvious) way to check which addresses are known (bugzilla
rejects the report if there's a 'wrong' email address in the list)  [IMO this is
really really broken.]
- You are asked to provide many details that need not be relevant and casual
reporters don't know that they can skip this part
- Attaching files is tedious and referring to attachments unintuitive

> While I also cannot say I love it I don't find it any less unfriendly
> than an email.

That depends (please see below).

> Besides the primary point of bug tracking is not to be friendly
> to someone, but to (a) fix the bugs and (b) know how many bugs
> there for a given release. Any replacement would need to solve
> this problem too.

For _tracking_ bugs, the bugzilla is more-or-less suitable.
For _reporting_ bugs, IMHO, it's not.

And I think they are two _totally_ conceptually different things.  You report
a bug to let somebody know that there's a problem and this doesn't necessarily
mean that the problem has to be tracked.  It may be very simple and immediately
resolvable, in which case registering it in bugzilla is a loss of time and
resources.  If the problem _turns_ _out_ to be difficult, _then_ it'll need to
be tracked, but usually it's not known whether or not this is the case until
someone 'in the know' looks at the initial report.

For this reason there should be a simple means of filing initial bug reports
with someone to look at them and forward them to appropriate people who will
decide if the problem needs to be tracked.  If they do, it's time to use
bugzilla.  Not earlier.

> Email does not solve it as far as I can see.

You are right, email is not suitable for tracking bugs.  Still, it works quite
well as a means of sending initial reports.

Greetings,
Rafael

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Rafael J. Wysocki wrote:
>
> - You are required to select a category and 'component' for your report, which
>   often is difficult (especially if you're not a kernel expert)
> - You need to have a bugzilla account (or to create one, if you don't)

Amen.

Both of those are show-stoppers. It may be ok for kernel developers, but 
it's not good for random people. It's one of the reasons I don't generally 
use bugzilla for other projects - if they require me to sign up etc, they 
can take care of their own bugs.

(For the same reason I consider closed mailing lists to be useless for 
bugreports. If you have to be a member to send email, it's not a 
bug-report thing, it's just a secret society).

I realize that spam is a problem, but there are better spam solutions than 
alienating the people who just want to send a report.

Also, I don't know if anybody has ever tried to avoid sending duplicate 
bugs, but every time I use bugzilla to send a bug-report (which I do for 
things like FC7 live CD's breaking etc where I care enough, and I have a 
bugzilla account on that bugzilla _anyway_ for other reasons), I am ready 
to _kill_ somebody whenever I see that "humorous" 

	zarro bugs found

message, after it made it almost impossible for me to even figure out how 
to do a good search in the first place!

So I gnash my teeth, and fill in the bug report anyway, and if it's a 
duplicate becasue bugzilla didn't have any sane way to get any kind of 
overview at all, hey, it's a duplicate. Nothing I can do about it.

And the reason I gnash my teeth is that I realize that because I can't 
even get any overview of the bugzilla entries as a random submitter (and 
dammit, I probably have a better idea of what the problem might be than 
most people), I sure as hell can't expect a random user to be better. So I 
bet my bad reports are a lot better than the average, and a lot of people 
probably just give up and don't report anything at all.

		Linus

Re: Linux 2.6.21

[Andi Kleen]

On Sun, Apr 29, 2007 at 08:50:55PM +0200, Rafael J. Wysocki wrote:
> On Sunday, 29 April 2007 19:37, Andi Kleen wrote:
> > > My personal experience with bugzilla is that it's very unfriendly to
> > > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > > debug patches, system information etc., but not for _reporting_ new ones.
> > 
> > What did you find unfriendly? 
> 
> - You are required to select a category and 'component' for your report, which
> often is difficult (especially if you're not a kernel expert)

Usually there is other and then someone else figures it out.

> - You need to have a bugzilla account (or to create one, if you don't)
> - If you want to add an address to the CC list, it must be known to bugzilla
> and there's no (obvious) way to check which addresses are known (bugzilla
> rejects the report if there's a 'wrong' email address in the list)  [IMO this is
> really really broken.]

The Novell bugzilla actually has that fixed. You have a search email button
to look up addresses.  Perhaps that feature will be ported someday into
the kernel.org one (I would like to have it too) 

> - You are asked to provide many details that need not be relevant and casual
> reporters don't know that they can skip this part
> - Attaching files is tedious and referring to attachments unintuitive

Anyways that are mostly all detail (except the registration requirement) that
could be probably all easily fixed.

> And I think they are two _totally_ conceptually different things.  You report
> a bug to let somebody know that there's a problem and this doesn't necessarily

The problem is we need a way to route those reports to the right people.
Routing it to a single person or broadcasting it just doesn't scale.
And the best way I know of is to use some database that keeps track of the state.

That is what bugzilla is essentially.

> For this reason there should be a simple means of filing initial bug reports
> with someone to look at them and forward them to appropriate people who will
> decide if the problem needs to be tracked.  If they do, it's time to use
> bugzilla.  Not earlier.

The only sane way to do that would be to save them somewhere and keep
a list and then let a group of people process them.

Hmm, wait... sounds like bugzilla, doesn't it?

> You are right, email is not suitable for tracking bugs.  Still, it works quite
> well as a means of sending initial reports.

I disagree. It works small scale but does not really scale well.

-Andi

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 21:14, Andi Kleen wrote:
> On Sun, Apr 29, 2007 at 08:50:55PM +0200, Rafael J. Wysocki wrote:
> > On Sunday, 29 April 2007 19:37, Andi Kleen wrote:
> > > > My personal experience with bugzilla is that it's very unfriendly to
> > > > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > > > debug patches, system information etc., but not for _reporting_ new ones.
> > > 
> > > What did you find unfriendly? 
> > 
> > - You are required to select a category and 'component' for your report, which
> > often is difficult (especially if you're not a kernel expert)
> 
> Usually there is other and then someone else figures it out.
> 
> > - You need to have a bugzilla account (or to create one, if you don't)
> > - If you want to add an address to the CC list, it must be known to bugzilla
> > and there's no (obvious) way to check which addresses are known (bugzilla
> > rejects the report if there's a 'wrong' email address in the list)  [IMO this is
> > really really broken.]
> 
> The Novell bugzilla actually has that fixed. You have a search email button
> to look up addresses.  Perhaps that feature will be ported someday into
> the kernel.org one (I would like to have it too) 
> 
> > - You are asked to provide many details that need not be relevant and casual
> > reporters don't know that they can skip this part
> > - Attaching files is tedious and referring to attachments unintuitive
> 
> Anyways that are mostly all detail (except the registration requirement) that
> could be probably all easily fixed.
> 
> > And I think they are two _totally_ conceptually different things.  You report
> > a bug to let somebody know that there's a problem and this doesn't necessarily
> 
> The problem is we need a way to route those reports to the right people.
> Routing it to a single person or broadcasting it just doesn't scale.
> And the best way I know of is to use some database that keeps track of the state.
> 
> That is what bugzilla is essentially.
> 
> > For this reason there should be a simple means of filing initial bug reports
> > with someone to look at them and forward them to appropriate people who will
> > decide if the problem needs to be tracked.  If they do, it's time to use
> > bugzilla.  Not earlier.
> 
> The only sane way to do that would be to save them somewhere and keep
> a list and then let a group of people process them.

But emailed reports _are_ saved anyway and we _know_ how to get a copy.
>From lkml.org, for example.  Why don't we use that?  The only missing piece
is the 'keep a list' thing, but that's not a rocket science, IMHO.

[For example, you can create a bugzilla entry with a link to the lkml.org copy
of the relevant message, so why to require the reporter to file the report with
the bugzilla himself?]

_Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
keep track of each thread separately, so you can browse any of them at any
time.  In particular, you can see the _history_ of each bug report sent to LKML
if you have a link to any message in its thread.

Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
you'll even be able to use the lkml.org archives plus wget and a couple of
shell scripts to cherry pick the links to all bug reports sent to the list
within a given time interval.

All of this functionality is out there already.

> Hmm, wait... sounds like bugzilla, doesn't it?
> 
> > You are right, email is not suitable for tracking bugs.

Well, now, I think that even this statement is not exactly right. :-)

> > Still, it works quite well as a means of sending initial reports.
>
> I disagree. It works small scale but does not really scale well.

IMO that depends on how you handle it.

Greetings,
Rafael

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
>...
> But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> From lkml.org, for example.  Why don't we use that?  The only missing piece
> is the 'keep a list' thing, but that's not a rocket science, IMHO.
> 
> [For example, you can create a bugzilla entry with a link to the lkml.org copy
> of the relevant message, so why to require the reporter to file the report with
> the bugzilla himself?]
> 
> _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> keep track of each thread separately, so you can browse any of them at any
> time.  In particular, you can see the _history_ of each bug report sent to LKML
> if you have a link to any message in its thread.
> 
> Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> you'll even be able to use the lkml.org archives plus wget and a couple of
> shell scripts to cherry pick the links to all bug reports sent to the list
> within a given time interval.
> 
> All of this functionality is out there already.
>...

How can I get the functionality "show me all unfixed SATA bugs"?

That's one of the important functionalities of every bug tracking 
system.

> Greetings,
> Rafael

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 22:43, Adrian Bunk wrote:
> On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> >...
> > But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> > From lkml.org, for example.  Why don't we use that?  The only missing piece
> > is the 'keep a list' thing, but that's not a rocket science, IMHO.
> > 
> > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > of the relevant message, so why to require the reporter to file the report with
> > the bugzilla himself?]
> > 
> > _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> > keep track of each thread separately, so you can browse any of them at any
> > time.  In particular, you can see the _history_ of each bug report sent to LKML
> > if you have a link to any message in its thread.
> > 
> > Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> > you'll even be able to use the lkml.org archives plus wget and a couple of
> > shell scripts to cherry pick the links to all bug reports sent to the list
> > within a given time interval.
> > 
> > All of this functionality is out there already.
> >...
> 
> How can I get the functionality "show me all unfixed SATA bugs"?
> 
> That's one of the important functionalities of every bug tracking 
> system.

That's the missing piece, obviously.

BTW, I didn't want to say that one could entirely replace a bug-tracking system
with tracking the LKML archives.  What I wanted to say was that the email
messages sent to the LKML were easily trackable and could be hooked up into a
bug-tracking system, for example with the help of URLs.

In such a setup people could send initial reports to the LKML and the links to
these messages might be put into a bug-tracking system as soon as it turned
out that the bugs were worthy of tracking.

Greetings,
Rafael

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 12:00:28AM +0200, Rafael J. Wysocki wrote:
> On Sunday, 29 April 2007 22:43, Adrian Bunk wrote:
> > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > >...
> > > But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> > > From lkml.org, for example.  Why don't we use that?  The only missing piece
> > > is the 'keep a list' thing, but that's not a rocket science, IMHO.
> > > 
> > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > of the relevant message, so why to require the reporter to file the report with
> > > the bugzilla himself?]
> > > 
> > > _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> > > keep track of each thread separately, so you can browse any of them at any
> > > time.  In particular, you can see the _history_ of each bug report sent to LKML
> > > if you have a link to any message in its thread.
> > > 
> > > Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> > > you'll even be able to use the lkml.org archives plus wget and a couple of
> > > shell scripts to cherry pick the links to all bug reports sent to the list
> > > within a given time interval.
> > > 
> > > All of this functionality is out there already.
> > >...
> > 
> > How can I get the functionality "show me all unfixed SATA bugs"?
> > 
> > That's one of the important functionalities of every bug tracking 
> > system.
> 
> That's the missing piece, obviously.
> 
> BTW, I didn't want to say that one could entirely replace a bug-tracking system
> with tracking the LKML archives.  What I wanted to say was that the email
> messages sent to the LKML were easily trackable and could be hooked up into a
> bug-tracking system, for example with the help of URLs.
> 
> In such a setup people could send initial reports to the LKML and the links to
> these messages might be put into a bug-tracking system as soon as it turned
> out that the bugs were worthy of tracking.

Who is doing this "might be put", and why don't you start with asking 
the submitter to submit bugs in a bug tracking system and forward the 
bug report from the bug tracking system (manually or automatically) to 
the developers and linux-kernel?

> Greetings,
> Rafael

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Monday, 30 April 2007 00:00, Adrian Bunk wrote:
> On Mon, Apr 30, 2007 at 12:00:28AM +0200, Rafael J. Wysocki wrote:
> > On Sunday, 29 April 2007 22:43, Adrian Bunk wrote:
> > > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > > >...
> > > > But emailed reports _are_ saved anyway and we _know_ how to get a copy.
> > > > From lkml.org, for example.  Why don't we use that?  The only missing piece
> > > > is the 'keep a list' thing, but that's not a rocket science, IMHO.
> > > > 
> > > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > > of the relevant message, so why to require the reporter to file the report with
> > > > the bugzilla himself?]
> > > > 
> > > > _Moreover_, some LKML archives, for example at http://marc.info/?l=linux-kernel,
> > > > keep track of each thread separately, so you can browse any of them at any
> > > > time.  In particular, you can see the _history_ of each bug report sent to LKML
> > > > if you have a link to any message in its thread.
> > > > 
> > > > Really, if we ask reporters to put '[BUG]' in the subjects of their messages,
> > > > you'll even be able to use the lkml.org archives plus wget and a couple of
> > > > shell scripts to cherry pick the links to all bug reports sent to the list
> > > > within a given time interval.
> > > > 
> > > > All of this functionality is out there already.
> > > >...
> > > 
> > > How can I get the functionality "show me all unfixed SATA bugs"?
> > > 
> > > That's one of the important functionalities of every bug tracking 
> > > system.
> > 
> > That's the missing piece, obviously.
> > 
> > BTW, I didn't want to say that one could entirely replace a bug-tracking system
> > with tracking the LKML archives.  What I wanted to say was that the email
> > messages sent to the LKML were easily trackable and could be hooked up into a
> > bug-tracking system, for example with the help of URLs.
> > 
> > In such a setup people could send initial reports to the LKML and the links to
> > these messages might be put into a bug-tracking system as soon as it turned
> > out that the bugs were worthy of tracking.
> 
> Who is doing this "might be put", and why don't you start with asking 
> the submitter to submit bugs in a bug tracking system and forward the 
> bug report from the bug tracking system (manually or automatically) to 
> the developers and linux-kernel?

Because quite often I know what the problem is after having asked the reporter
a couple of simple questions or I can forward his report to a list on which
there are the right people and the problem gets fixed quickly, so it just
doesn't need to be formally registered.

The problem with the bugzilla is that it requires a considerable setup for each
bug which is a loss of time if the bug is trivial.  Using the bugzilla for
handling trivial bugs just doesn't make sense, IMHO.  It's too heavywieght
for that.  [It _is_ useful nevertheless.  For example, the suspend metabug that
you've created for me is really a nice thing, so thanks a lot again. :-)
Perhaps something like this might be used for tracking the regressions ...]

Unfortunately, you can't say whether or not the bug is trivial until you see
the report.  If you've got it from bugzilla, you're stuck with it, so it's just
more efficient to use something else in the initial phase of resolving
problems.

Greetings,
Rafael

Re: Linux 2.6.21

[Alexey Dobriyan]

On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> [For example, you can create a bugzilla entry with a link to the lkml.org copy
> of the relevant message, so why to require the reporter to file the report with
> the bugzilla himself?]

Last time I did this, bugzilla at osdl.org won't let me add original
reporter to goddamn CC list. It would be el neat, because not everyone
followed instructions and forwarding emails between reporter and
bugzilla sucks.

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 22:52, Alexey Dobriyan wrote:
> On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > of the relevant message, so why to require the reporter to file the report with
> > the bugzilla himself?]
> 
> Last time I did this, bugzilla at osdl.org won't let me add original
> reporter to goddamn CC list. It would be el neat, because not everyone
> followed instructions and forwarding emails between reporter and
> bugzilla sucks.

That's related to what I said before.  The requirement that the addresses on
the CC list must be 'known' to bugzilla is deadly wrong in every case I can
imagine.

Re: Linux 2.6.21

[Andrew Morton]

On Mon, 30 Apr 2007 00:09:06 +0200 "Rafael J. Wysocki" <rjw@sisk.pl> wrote:

> On Sunday, 29 April 2007 22:52, Alexey Dobriyan wrote:
> > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > of the relevant message, so why to require the reporter to file the report with
> > > the bugzilla himself?]
> > 
> > Last time I did this, bugzilla at osdl.org won't let me add original
> > reporter to goddamn CC list. It would be el neat, because not everyone
> > followed instructions and forwarding emails between reporter and
> > bugzilla sucks.
> 
> That's related to what I said before.  The requirement that the addresses on
> the CC list must be 'known' to bugzilla is deadly wrong in every case I can
> imagine.

But unknown-to-bugzilla email addresses are accepted when they're sent to
bugme-daemon@kernel-bugs.osdl.org.  This is why I'll very often switch a
bug report to email, copying individuals and mailing lists and
bugme-daemon.  Then bugzilla just sits silently in the background recording
everything.

But once a bug has switched to email, it needs to stay there - it would be
bad if someone were to update the bug via the web UI because none of the
emailed participants would know of the update.  So i'll often explicitly
ask "please follow up via emailed reply-to-all".

It's not great, but there's certainly enough material here for people to get
in and work on the bug, should they be so inclined.

My overall approach with this stuff is: short-term bugs are handled via email
and long-term ones are tracked in buzilla.

Hence someone (Hi, Mom) needs to track all the emailed-only bug reports and get
them filed in bugzilla once they go stale.

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Monday, 30 April 2007 08:30, Andrew Morton wrote:
> On Mon, 30 Apr 2007 00:09:06 +0200 "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > On Sunday, 29 April 2007 22:52, Alexey Dobriyan wrote:
> > > On Sun, Apr 29, 2007 at 10:18:10PM +0200, Rafael J. Wysocki wrote:
> > > > [For example, you can create a bugzilla entry with a link to the lkml.org copy
> > > > of the relevant message, so why to require the reporter to file the report with
> > > > the bugzilla himself?]
> > > 
> > > Last time I did this, bugzilla at osdl.org won't let me add original
> > > reporter to goddamn CC list. It would be el neat, because not everyone
> > > followed instructions and forwarding emails between reporter and
> > > bugzilla sucks.
> > 
> > That's related to what I said before.  The requirement that the addresses on
> > the CC list must be 'known' to bugzilla is deadly wrong in every case I can
> > imagine.
> 
> But unknown-to-bugzilla email addresses are accepted when they're sent to
> bugme-daemon@kernel-bugs.osdl.org.  This is why I'll very often switch a
> bug report to email, copying individuals and mailing lists and
> bugme-daemon.  Then bugzilla just sits silently in the background recording
> everything.

If I wanted to do this, what would I have to do?  I mean, assume I have a bug
report that I want to send to someone whom the bugzilla doesn't like.
What's the right procedure in such a case?

> But once a bug has switched to email, it needs to stay there - it would be
> bad if someone were to update the bug via the web UI because none of the
> emailed participants would know of the update.  So i'll often explicitly
> ask "please follow up via emailed reply-to-all".
> 
> It's not great, but there's certainly enough material here for people to get
> in and work on the bug, should they be so inclined.
> 
> My overall approach with this stuff is: short-term bugs are handled via email
> and long-term ones are tracked in buzilla.

I think that's reasonable, I've always done it this way, AFAIR.

> Hence someone (Hi, Mom) needs to track all the emailed-only bug reports and get
> them filed in bugzilla once they go stale.

I can do that with suspend/hibernation-related bug reports, but sometimes
I'm not sure who is the right person to notify in the first place.

Re: Bugzilla (was Linux 2.6.21)

[Martin J. Bligh]

Sorry, have been out sick, and someone removed me from the cc list,
which didn't help. In response to various bits:

Firstly a general comment - we're about to upgrade versions, which
will ease a few of these issues. I should really finish the creation
of virtual category owners for *all* categories. Will see if we can
batch that, as it's a total pain to do.

Andi Kleen wrote:
 > - Ask more people to just categorize and reassign bugs (anybody
 > interested?)

The category owners should be able to do that, and help spread the
load. The virtual category owner stuff enables many people to "watch"
new bugs for that category and help out.

 > - Give more people in bugzilla the power to reassign arbitary bugs
 > (bugzilla maintainers would need to do that)

Fairly easy to do, just a permissions issue. Either I can add a bunch
of "known" people, or let everyone do it and then slap people if
they're silly about it.

>> - You are required to select a category and 'component' for your report, which
>> often is difficult (especially if you're not a kernel expert)
> 
> Usually there is other and then someone else figures it out.

I can make that clearer in the form if it helps.

> The Novell bugzilla actually has that fixed. You have a search email button
> to look up addresses.  Perhaps that feature will be ported someday into
> the kernel.org one (I would like to have it too) 

I *think* that's in the new version. Will check.

> The only sane way to do that would be to save them somewhere and keep
> a list and then let a group of people process them.
> 
> Hmm, wait... sounds like bugzilla, doesn't it?

Yes, though we could do with some improved email hooks still, I guess.
I much prefer having people watch categories than spamming lists, but
if people want lists spammed, we can have that.

Re: Linux 2.6.21

[Willy Tarreau]

On Sun, Apr 29, 2007 at 07:37:25PM +0200, Andi Kleen wrote:
> > My personal experience with bugzilla is that it's very unfriendly to
> > reporters.  IMHO it's suitable for tracking unresolved problems along with
> > debug patches, system information etc., but not for _reporting_ new ones.
> 
> What did you find unfriendly? 

It is too much complicated for new reporters.

I remember I sent a patch by mail for NTP to people at ISC, and they asked
me to pass through bugzilla because it was important for them to track it.
What initially was a 5 minutes email turned to a 30 minutes nightmare with
doubts at every click, and it was even difficult for me to attach the patch.

Later they replied to me by mail, which I consulted from another address, I
replied and was rejected because it was not the same address... I had to be
very very motivated to use such a crap.

Definitely the tool we need if we want to reduce the number of bug reports!

Just my experience as a bug reporter...

Willy

Re: Linux 2.6.21

[Andi Kleen]

On Sun, Apr 29, 2007 at 09:07:43AM -0700, Linus Torvalds wrote:
> 
> 
> On Sun, 29 Apr 2007, Andi Kleen wrote:
> > Linus Torvalds <torvalds@linux-foundation.org> writes:
> > > 
> > > The thing is, bugzilla is totally broken because it's designed to help 
> > > track bugs, but it's *not* designed to actually handle the much harder 
> > > problem, which is to actually get the *right* developers to be aware 
> > > of the *right* bugs!
> > 
> > This means we need people who figure out who to assign bugs too.
> > Aka bugmasters.
> 
> Yes. But not using bugzilla.

And what would you use instead? 

> 
> I don't know if you've noticed already, but I'm not the only one that 
> doesn't have a very high opinon of bugzilla ;)

Sure, but do you have alternatives? 

> What works is somebody who is a bugmaster, and it doesn't really matter 
> *what* bug tracker he points to (bugzilla being one of the possibilities, 
> although not necessarily the best, and absolutely NOT the only choice), 
> and turn them into emails. Once they are emails, bugzilla can track them.

So you want to do the work that should be done by a computer in
a database done by a human.  Does not sound particularly efficient.

> > In theory it could be nearly automated. Figure out what files related
> > to the bug and assign to the last 5 people who submitted patches
> > for them and/or signed off. 
> 
> I do think you're pretty optimistic. I think it's true for trivial driver 
> bugs, but even for trivial driver bugs the initial report is often not 
> enough to pinpoint the driver. 
> 
> Let's take the sis900 bug as an example (not because I want to rag on that 
> being a problem, but it happened to be a _trivial_ bug in 2.6.21, so it's 
> a good case of something really really easy - and if that easy case isn't 
> handled trivially and obviously, then the bug-reporting doesn't work).
> 
> In that case, the initial report was (condensed version, but fairly 
> accurate): "system hangs on boot at random points. 2.6.21-rc7 worked 
> well".
> 
> Now, realistically, if that entry had been in bugzilla, what would you do?

You need a few people in bugzilla that ask the questions to narrow it
down (= bugmasters). e.g. the opensuse bugzilla works this way :- 
everything new gets assigned to a few screening people who 
ask some questions and then reassign to the right people.

> Would bugzilla have helped? HELL NO. It would have been a disaster. It 
> would have wasted reporter time, it would have wasted developer time, and

An unmaintained bugzilla yes. A well maintained one would have someone
asking them the (often quite repetive questions) to narrow it down
to a subsystem or a driver. And then it could have been assigned.

> I think both of these are good ideas, but I don't think it's enough. There 
> must be some better form of bug tracking than bugzilla. Some relly 
> distributed way of letting people work together, *without* having to 
> congregate on some central web-site kind of thing. A "git for bugs", where 
> you can track bugs locally and without a web interface.
> 
> And I think it would be something much closer to "distributed email mbox 
> with status tracking facilities", _not_ a web clicky interface.

There have been a couple of email thread trackers; like jitterbug -- 
in fact bugzilla can do that with an email interfaces. But in my experience 
they don't work well because a bug tracking system has slightly different
requirements than normal emails (e.g. it wants you to roughly
stay on topic for the current bug). With email people always
forget that and in the end you end up with lots of stuff in there
not related to the bug at all.

Also with distributed solutions it would be hard to get
a global "how many regressions do we really have right now" statistic,
which is fairly important.

The web interface is slow and ugly but at least it puts
the people in the right mind set for this unlike email.
And it gives you a central point to get an overview of the bugs.

Anyways I'm sure bugzilla could be improved, I just don't know
of anything better currently.

-Andi

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> On Sun, Apr 29, 2007 at 09:07:43AM -0700, Linus Torvalds wrote:
> >
> > Yes. But not using bugzilla.
> 
> And what would you use instead? 

Didn't you even *read* my email?

I already told you: we have real bugs getting reported and fixed that 
don't hit bugzilla or any bugtracker AT ALL.

This is not a "all or nothing" situation. There is absolutely _zero_ real 
reason to think that everything has to go through bugzilla.

> > I don't know if you've noticed already, but I'm not the only one that 
> > doesn't have a very high opinon of bugzilla ;)
> 
> Sure, but do you have alternatives? 

Yes. The ones that *work*. Plain email is preferably over bugzilla 90% of 
the time. 

But quite frankly, if you think you can make bugzilla work (and realize 
that a lot of people will _not_ be looking at it or reporting bugs into 
it), go ahead. I don't care. The only think I care about is *REALITY*, and 
that means:

 - a lot of reporters will not use bugzilla, because it's damn 
   inconvenient even for reporting. If you propose something that uses 
   _only_ bugzilla, you'd better also have the people who enter other 
   peoples bugreports into there.

 - a lot of developers will not use bugzilla, because it's even more 
   inconvenient for developers, with no sane ways to interact with the 
   right people. So if you propose using bugzilla, you'd really better 
   have the manpower to turn bugzilla into emails (and no, the bugzilla 
   cc list etc is _not_ the primary one - the email cc's are the primary 
   ones, because that's where it is much easier to bring in new people)

In other words, anything that thinks that bugzilla is "primary" is just 
broken. It can be a _part_ of the thing, but drop the belief of it being a 
primary tracker. It's just too inconvenient.

		Linus

Re: Linux 2.6.21

[Andi Kleen]

>  - a lot of reporters will not use bugzilla, because it's damn 
>    inconvenient even for reporting. If you propose something that uses 

Don't think that's true. There are plenty of projects who only
accept bugs through bugzilla (mozilla, various distributions, etc.) 
and I don't see any evidence of your claim being true.

Sure there will be always people who cannot be bothered
to use any kind of interface for bugs, but then
these are unlikely to stay on board during a longer 
remote debugging q'n'a session either. So those people
can be just ignored; they essentially don't exist in
the bug report universe.

Anyways it only works if people are willing to use it too and there
are enough people who maintain bugs (aka ask questions to find out
who to reassign, prune old bugs etc.)  If that's not there then
it won't work well obviously, like it is currently the case.

I don't think the "keep it in Andrew's/Adrian's head" method
is going to scale longer term at least (and one of them has 
already thrown in the towel) 

The "send it to a gigantic mailing list and hope someone catches
it" method also doesn't seem to be that great. At least there
are lots of lost reports in my experience this way.

I suspect the real reason is more "Linus doesn't like web interfaces
for no particular good reason". Not much can be done about that.
Well perhaps someone can write a gopher based bugzilla interface 
or something to solve that instead @)

-Andi

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Andi Kleen wrote:
> 
> I suspect the real reason is more "Linus doesn't like web interfaces
> for no particular good reason". Not much can be done about that.

Right. Dig your head in the sand, and ignore all the other people who 
piped up and said they hate bugzilla too, and find email much more 
convenient. It's "just Linus" in your dreamworld.

		Linus

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 20:09, Andi Kleen wrote:
> >  - a lot of reporters will not use bugzilla, because it's damn 
> >    inconvenient even for reporting. If you propose something that uses 
> 
> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.
> 
> Sure there will be always people who cannot be bothered
> to use any kind of interface for bugs, but then
> these are unlikely to stay on board during a longer 
> remote debugging q'n'a session either. So those people
> can be just ignored; they essentially don't exist in
> the bug report universe.
> 
> Anyways it only works if people are willing to use it too and there
> are enough people who maintain bugs (aka ask questions to find out
> who to reassign, prune old bugs etc.)  If that's not there then
> it won't work well obviously, like it is currently the case.
> 
> I don't think the "keep it in Andrew's/Adrian's head" method
> is going to scale longer term at least (and one of them has 
> already thrown in the towel) 
> 
> The "send it to a gigantic mailing list and hope someone catches
> it" method also doesn't seem to be that great. At least there
> are lots of lost reports in my experience this way.

This, actually, might work if the report is 'flagged' in a specific way.

For example, if there's a message sent to LKML with a combination of '[BUG]'
and 'suspend' in the subject, I have no problems whatsoever with spotting it. ;-)

Greetings,
Rafael

Re: Linux 2.6.21

[Russell King]

On Sun, Apr 29, 2007 at 08:09:09PM +0200, Andi Kleen wrote:
> >  - a lot of reporters will not use bugzilla, because it's damn 
> >    inconvenient even for reporting. If you propose something that uses 
> 
> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.

There's an ARM category in the kernel.org bugzilla.  Folk are completely
free to submit ARM bugs to either the (closed) mailing list or bugzilla.

99.99999999% of bug reports in the ARM community come via the mailing
list.  I think to date there's been about 10 bugzilla entries since
Feb 2004.

This is inspite of me linking to the kernel.org bugzilla from my
website.

So it seems that virtually all the folk involved with the ARM kernel,
given a completely free choice, _prefer_ to send email-based bug
reports over touching bugzilla.

That's quite a different metric to projects forcing bugzilla on people,
and I'd say is a more valid metric to gauge whether bugzilla is really
suitable.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Linux 2.6.21

[Diego Calleja]

So far, it seems that most of people's opinion WRT to bug reporting and trackingcan
be divided into 2 groups:

- People who argues (and they're right) that bugzilla and web interfaces in general
  suck and that email + an "Adrian-like" solution works better

- People who argues that a bug tracker better than a mailing list is absolutely
  needed (and they're right). They also argue that while bugzilla sucks, it's
  better than nothing.

There's a common point between both groups: bugzilla sucks. The ideal
solution would be to replace bugzilla with some alternative and better
opensource bug tracking software, but I doubt it exists (there must be a
reason why everybody uses bugzilla). A good bug tracker should feel like
it makes your work easier, instead of making you feel like you're wasting
time (which is what bugzilla does)

I don't see why a web interface bug tracker should be bad for bug tracking,
as long as it's good and integrates 100% in the mailing lists. In my humble
opinion the "perfect" bug tracker for Linux should be something like this:
- Has a email interface (like the Debian bug tracking database).
- Has a web interface that completely follows the email threads
   (reading/posting), but make the comments real emails, not just
    database fields.
If done well (unlike the current bugzilla-to-email hack), it should possible
to do many nice things, like add a lkml bug report to the bug tracking
database (which shouldn't be a "real" database, but just an lkml mail
archive with a list of message IDs that are considered a bug and its state)
by just replying the thread, CCing the bug tracker and telling him to include
the thread in the database.

So unless someone is willing to write such tool (which I doubt, since it
doesn't looks easy), all this discussion seems pointless, and we should
stick with this http://kernelnewbies.org/known_regressions page
which is showing to be quite useful :)

Re: Linux 2.6.21

[Michal Piotrowski]

Hi Diego,

On 29/04/07, Diego Calleja <diegocg@gmail.com> wrote:
[..]
> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

Thanks for your help with this!

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Gene Heskett]

On Sunday 29 April 2007, Michal Piotrowski wrote:
>Hi Diego,
>
>On 29/04/07, Diego Calleja <diegocg@gmail.com> wrote:
>[..]
>
>> So unless someone is willing to write such tool (which I doubt, since it
>> doesn't looks easy), all this discussion seems pointless, and we should
>> stick with this http://kernelnewbies.org/known_regressions page
>> which is showing to be quite useful :)

Usefull?  For what?  Having seen the link at the time I was fighting with yet 
another rearranged USB setup, and I swear the boot process uses Schrodingers 
Cat to determine which device found in the usb scan gets the first ttyUSB, 
and which gets the second, so I went to this site to see if it could be more 
productive than bugzilla (I'd like to toss about 10 pounds of C4 into the 
only code repository on the planet that thing is built from), but no, sorry, 
no bisquit.

Why?  First, its yet another password I have to scribble on the wall along 
with enough surrounding data so I might be able to find it the next time.

You can't have it even do a search to see if it already has something similar 
without creating an account and logging in.  Since I'm out of wall space, and 
the missus is bugging me to paint over all that, I left.

I repeat:  Usefull?  For what?

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Simon: (to Jayne) "Enemies?  You?  No, how can it be?"
				--Episode #7, "Jaynestown"

Re: Linux 2.6.21

[Bernd Eckenfels]

In article <200704292150.22956.gene.heskett@gmail.com> you wrote:
> You can't have it even do a search to see if it already has something similar 
> without creating an account and logging in.  Since I'm out of wall space, and 
> the missus is bugging me to paint over all that, I left.

Well, thats not a bugzilla problem. upstream bugzilla allows anonymous
search.

Infact bugme.osdl.org allows search right on the frontpage. And if you want
to dig deeper, use the query function.

This is the quicksearch on "USB":

<http://bugme.osdl.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug_status=OPEN&bug_status=ASSIGNED&bug_status=REOPENED&field0-0-0=product&type0-0-0=substring&value0-0-0=usb&field0-0-1=component&type0-0-1=substring&value0-0-1=usb&field0-0-2=short_desc&type0-0-2=substring&value0-0-2=usb&field0-0-3=status_whiteboard&type0-0-3=substring&value0-0-3=usb>

> I repeat:  Usefull?  For what?

Try again.

Gruss
Bernd

Re: Linux 2.6.21

[Gene Heskett]

On Monday 30 April 2007, Bernd Eckenfels wrote:
>In article <200704292150.22956.gene.heskett@gmail.com> you wrote:
>> You can't have it even do a search to see if it already has something
>> similar without creating an account and logging in.  Since I'm out of wall
>> space, and the missus is bugging me to paint over all that, I left.
>
>Well, thats not a bugzilla problem. upstream bugzilla allows anonymous
>search.
>
>Infact bugme.osdl.org allows search right on the frontpage. And if you want
>to dig deeper, use the query function.
>
>This is the quicksearch on "USB":
>
><http://bugme.osdl.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=NEW&bug
>_status=OPEN&bug_status=ASSIGNED&bug_status=REOPENED&field0-0-0=product&type
>0-0-0=substring&value0-0-0=usb&field0-0-1=component&type0-0-1=substring&valu
>e0-0-1=usb&field0-0-2=short_desc&type0-0-2=substring&value0-0-2=usb&field0-0
>-3=status_whiteboard&type0-0-3=substring&value0-0-3=usb>
>
And that output I'll have to admit, is much more usefull.  However, in your 
wildest dreams I couldn't recompose that link line if I was stuck in a pool 
drain 10 feet down and that turned on the air hose I had in my hands.

>> I repeat:  Usefull?  For what?
>
>Try again.
>
>Gruss
>Bernd



-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
He who is content with his lot probably has a lot.

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 09:40:07PM +0200, Diego Calleja wrote:
>...
> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

This list currently contains 29 known regressions.
Someone has to manually add them.
Someone has to manually track the status of all of them.
Someone has to manually group related regressions (e.g. in the same 
subsystem) together.

The kernel Bugzilla currently contains 1600 open bugs.

Maintaining a regression list by hand was really a pain when during 
2.6.21-rc we had 36 known regressions.

Any approach that does not involve some kind of tracker with some kind 
of database simply doesn't scale.

Bugzilla might not be perfect, but it works and it's better than doing 
it by hand.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> 
> The kernel Bugzilla currently contains 1600 open bugs.

Adrian, why do you keep harping on this, and ignoring reality?

Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.

How many of those are interesting and valid? How many of them are 
relevant? How many of them are duplicates?

You don't know. Nobody does. So why do you bother reporting that number?

That number is exactly as relevant as the number of dog-hairs on our couch 
("in the millions"). An impressively large number, definitely uncountable, 
and definitely also not relevant to anythign at all. Not at all unlike 
that "1600 open bugs" number that you bring up.

Do you think the number of dog-hairs on our couch is an argument for or 
against people trying to track regressions? If not, why do you keep 
bringing up bugzilla?

		Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 01:33:16PM -0700, Linus Torvalds wrote:
> 
> 
> On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > 
> > The kernel Bugzilla currently contains 1600 open bugs.
> 
> Adrian, why do you keep harping on this, and ignoring reality?
> 
> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.

OK, how do you suggest to track bugs in a way that doesn't suck?

Bug reports to linux-kernel have the big problem that they are lost if 
no developer immediately picks them up.

> How many of those are interesting and valid? How many of them are 
> relevant? How many of them are duplicates?
> 
> You don't know. Nobody does. So why do you bother reporting that number?

What I do know is that the majority of them has never been proper 
debugged by a kernel developer knowing the subsystem in question.

> That number is exactly as relevant as the number of dog-hairs on our couch 
> ("in the millions"). An impressively large number, definitely uncountable, 
> and definitely also not relevant to anythign at all. Not at all unlike 
> that "1600 open bugs" number that you bring up.
> 
> Do you think the number of dog-hairs on our couch is an argument for or 
> against people trying to track regressions? If not, why do you keep 
> bringing up bugzilla?

I tracked 2.6.21-rc regressions, and I do not scale for higher numbers 
of bugs. When I had to track 36 known regressions it was a real 
nightmare.

Bugzilla tracks regressions and scales for higher numbers of bugs.

Let me ask you two questions:
- Do you think regression tracking makes any sense at all?
- If yes, which scalable way of regression tracking would in your
  opinion not suck?

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > 
> > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> 
> OK, how do you suggest to track bugs in a way that doesn't suck?

I've tried to explain.

Bugzilla can be one _part_ of it, but anybody who thinks it's the "main 
part" is really not being realistic. It's too cumbersome, and it's too 
stupid. 

Quite frankly, "lkml + google" is probably in many ways a *better* way to 
search for problems. But yes, some manual smarts (and the _occasional_ 
pointer to bugzilla) is probably currently the only option.

Exactly because I don't think anybody has shown any better automation than 
bugzilla. But that doesn't make bugzilla "the One Choice". That's not how 
it works. If there is no automation, manual tracking is still better than 
*crap* automation.

> Bug reports to linux-kernel have the big problem that they are lost if 
> no developer immediately picks them up.

..and this is different from bugzilla exactly _how_?

Those things are lost too. As you yourself have pointed out. The fact that 
you can search for them is _exactly_ as relevant as the fact that you can 
search for lkml on google.

> What I do know is that the majority of them has never been proper 
> debugged by a kernel developer knowing the subsystem in question.

And you blame the developers, but not bugzilla? Why are you so unable to 
see bugzilla as part of the *cause* of the problem? You're perfectly happy 
to blame other things, but bugzilla is somehow above blame?

		Linus

Re: Linux 2.6.21

[Anton Altaparmakov]

On 29 Apr 2007, at 22:24, Linus Torvalds wrote:
> Exactly because I don't think anybody has shown any better  
> automation than
> bugzilla. But that doesn't make bugzilla "the One Choice". That's  
> not how
> it works. If there is no automation, manual tracking is still  
> better than
> *crap* automation.

Have you seen/used RT? -> http://bestpractical.com/rt

We use it here at work and it works great.  People can report bugs  
both by email or via web interface.  We get everything that comes in  
emailed to us and we can respond by email and RT recognizes the  
responses being in the same thread and lumps them into the same bug   
(and when the origin was by email that is even without evil bug  
numbers appearing in the subject with the help of some perl scrip  
magic (aka RT action script)).  The only time I ever go into the web  
interface is about once a week to have a look at my list of open bugs  
and to do some tidying like merging bug reports and things like that.

It also has some cool features like "extract this into the FAQ" and  
there is a "FAQ" in RT that contains an autogenerated FAQ from what  
people have pulled out in that way.

Only problem is for the kernel we would need a beefy system (needs  
fast database or it gets very slow when you get into 100k+ bugs  
region) and someone who knows RT well and has a lot of spare time to  
set it up to your liking and then to maintain it...  (RT takes a  
while to set up because you can tweak just about everything and you  
can add/modify/remove functionality at will as it is very modular and  
written in Perl so pretty much anyone can adapt it to do exactly what  
they want without even needing to wait for lengthy recompiles to  
happen...)

You could for example automate sorting of bug reports into queues  
(e.g. SCSI, Net, FS, etc) by grepping the emailed bug report (or  
website generated one although on the website people can choose the  
queue by hand if they want) and sorting appropriately.  Admittedly  
for this to be at all useful someone would have to spend some time  
working out intelligent things to grep for or all bugs would match to  
all queues when they contain dmesg output for example... (-;

This might not be perfect but in comparison to bugzilla it is  
actually usable and at least we here at work like it so much we  
adopted it into our workflow voluntarily which says something...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 02:24:03PM -0700, Linus Torvalds wrote:
> 
> 
> On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > > 
> > > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> > 
> > OK, how do you suggest to track bugs in a way that doesn't suck?
> 
> I've tried to explain.
> 
> Bugzilla can be one _part_ of it, but anybody who thinks it's the "main 
> part" is really not being realistic. It's too cumbersome, and it's too 
> stupid. 

I do completely agree with you on this.

The main parts are people doing some sorting and forwarding of an 
incoming bug (currently mostly Andrew) and someone with deeper subsystem 
knowledge looking deeper into a bug (currently often missing).

> Quite frankly, "lkml + google" is probably in many ways a *better* way to 
> search for problems. But yes, some manual smarts (and the _occasional_ 
> pointer to bugzilla) is probably currently the only option.
> 
> Exactly because I don't think anybody has shown any better automation than 
> bugzilla. But that doesn't make bugzilla "the One Choice". That's not how 
> it works. If there is no automation, manual tracking is still better than 
> *crap* automation.

I had the regressions stored in a plain textfile.

For getting regressions reasonably grouped for my regression emails, I 
used paper, pen and scissors - and this is not a joke.

That really didn't scale when we had 36 regressions.

So some tool is needed if the bug numbers are bigger - no matter whether 
it's Bugzilla or speaking plain SQL to a database, or anything else.

> > Bug reports to linux-kernel have the big problem that they are lost if 
> > no developer immediately picks them up.
> 
> ..and this is different from bugzilla exactly _how_?
> 
> Those things are lost too. As you yourself have pointed out. The fact that 
> you can search for them is _exactly_ as relevant as the fact that you can 
> search for lkml on google.

It depends on how you look at bugs.

My ideal was always that reported bugs should be fixed.

If you accept that this is anyway impossible because more bugs get added 
than could get fixed you might not need any tracking at all.

> > What I do know is that the majority of them has never been proper 
> > debugged by a kernel developer knowing the subsystem in question.
> 
> And you blame the developers, but not bugzilla? Why are you so unable to 
> see bugzilla as part of the *cause* of the problem? You're perfectly happy 
> to blame other things, but bugzilla is somehow above blame?

If Andrew forwarded a bug reported in Bugzilla to a developer, and 
the developer doesn't answer, is this Bugzilla's fault? Or in any other 
way worse than a bug report direct to the developer?

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Mon, 30 Apr 2007, Adrian Bunk wrote:
> 
> My ideal was always that reported bugs should be fixed.

..and this is where we differ.

OF COURSE bugs should be fixed. But you seem to think that there is 
something magical and special about every single bug-report.

You have a new home assignment: watch the "every sperm is sacred" thing 
from Monty Python's "Meaning of Life". Google for it.

And if you cannot appreciate the absurdity and humor of that thing, maybe 
you should think about it a bit more.

And once you _can_ appreciate the humor of that song/skit, look yourself 
in the mirror, and ask yourself: "is every bug report sacred?"

Really?

> If you accept that this is anyway impossible because more bugs get added 
> than could get fixed you might not need any tracking at all.

That's a TOTALLY IDIOTIC argument.

That goes from "every sperm is sacred" to "sperm doesn't count at all".

Can you not see how stupid that statement of yours really is? Can you not 
see that anybody who thinks in those kinds of black-and-white terms is 
simply not FUNCTIONAL!

Bugs are neither sacred, _nor_ should they be ignored. 

Ponder that, grasshopper. And until you can see that things are not 
"either-or", "black-and-white", "all or nothing", I don't think I really 
can have anything worthwhile to add in this discussion to you. People who 
think in absolutes are simply not worth talking to.

			Linus

Re: Linux 2.6.21

[Linus Torvalds]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1023 bytes --]



On Mon, 30 Apr 2007, Linus Torvalds wrote:

> Ponder that, grasshopper. And until you can see that things are not 
> "either-or", "black-and-white", "all or nothing", I don't think I really 
> can have anything worthwhile to add in this discussion to you. People who 
> think in absolutes are simply not worth talking to.

This is another case of "perfect is the enemy of good".

Tryng to reach perfect is not only guaranteed to fail, but trying to reach 
it AND NOT REALIZING that it's stupid and wrong is actually much WORSE 
than just trying to do a reasonable job.

And if you put some _totally_idiotic_ expectation that all bugreports can 
be fixed, and should always be totally blocking, that's guaranteed to just 
cause a totally unusuable bug reporting system.

And your bugzilla arguments seem to be exactly that. A naïve and totally 
unrealistic expectation of "every bugreport is sacred" is BAD.

In other words: perfection not only isn't even possible, BUT IT IS NOT 
EVEN WORTH TRYING TO REACH FOR!

			Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 11:20:46AM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 30 Apr 2007, Adrian Bunk wrote:
> > 
> > My ideal was always that reported bugs should be fixed.
> 
> ..and this is where we differ.
> 
> OF COURSE bugs should be fixed. But you seem to think that there is 
> something magical and special about every single bug-report.
> 
> You have a new home assignment: watch the "every sperm is sacred" thing 
> from Monty Python's "Meaning of Life". Google for it.

I like the Flying Circus and the other Monty Python films (including 
"The Crimson Permanent Assurance"), but "Meaning of Life" didn't impress 
me. But I have the song somewhere if this counts.  ;-)

> And if you cannot appreciate the absurdity and humor of that thing, maybe 
> you should think about it a bit more.
> 
> And once you _can_ appreciate the humor of that song/skit, look yourself 
> in the mirror, and ask yourself: "is every bug report sacred?"
> 
> Really?
> 
> > If you accept that this is anyway impossible because more bugs get added 
> > than could get fixed you might not need any tracking at all.
> 
> That's a TOTALLY IDIOTIC argument.
> 
> That goes from "every sperm is sacred" to "sperm doesn't count at all".
> 
> Can you not see how stupid that statement of yours really is? Can you not 
> see that anybody who thinks in those kinds of black-and-white terms is 
> simply not FUNCTIONAL!
> 
> Bugs are neither sacred, _nor_ should they be ignored. 
> 
> Ponder that, grasshopper. And until you can see that things are not 
> "either-or", "black-and-white", "all or nothing", I don't think I really 
> can have anything worthwhile to add in this discussion to you. People who 
> think in absolutes are simply not worth talking to.

I never expected the reality to be come as white as my ideal or the 
washed things in washing powder ads.

My ideal was white, and the shade of grey of the current reality is 
darker than I think it should be.

At least theoretically reachable are things like:
- every incoming bug report is quickly handled by one or more
  kernel developers who know the drivers and subsystems involved
- there's a last -rc kernel published for a few days of testing,
  and except for the Makefile change the -final is identically to it
  (or a new -rc published)

There are sometimes nonsensical bug reports and "handling" could be 
rejecting a bug e.g. due to a tainted kernel.

And sometimes mysterious bugs are more or less undebuggable.

But these two points would have resulted in 2.6.21 being released more 
or less at the same time, but with a dozen regressions less.

> 			Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Vegard Nossum]

On Mon, April 30, 2007 8:57 pm, Adrian Bunk wrote:
> I never expected the reality to be come as white as my ideal or the
> washed things in washing powder ads.

This reminds me very much of what the brilliant computing scientist Edsger
W. Dijkstra more than once wrote:

`Confusing "love of perfection" with "claim of perfection", people will
accuse you of the latter and then blame you for the first.' (EWD709)

Also relevant to the discussion, I think, but in another way, is this:

`One of them is the dogma that striving for perfection is
counterproductive in the sense that it would make software development
much too expensive. But what are the main causes of the soaring costs of
software development? A major cost, in terms of both manpower and
unforeseen delays, is debugging, and one can save a lot by investing more
in preventing the bugs from entering the design in the first place. Since
the errors are so expensive, in general the high-quality design is also by
far the cheaper. Another major cause is that many systems are built on
shifting foundations in the sense that the underlying software of
operating systems and compilers is too shaky to be stable, with the result
that each new release of that underlying software requires possibly
extensive adaptation of what has been built on top of it. Finally, many of
the tools the programmer is supposed to work with are so poorly documented
that they force him to find out by experiment what they might be able to
do for him. Since these experiments can be pretty expensive and
time-consuming and —inductive reasoning being what it is— an educated
guess is the best the poor programmer can hope for, the poor programmer is
really in a miserable position. So here you see three major sources of
cost explosion traced down to someone's assumption that striving for
perfection is counterproductive!' (EWD952)

For the complete documents, see
http://www.cs.utexas.edu/users/EWD/transcriptions/EWD07xx/EWD709.html and
http://www.cs.utexas.edu/users/EWD/transcriptions/EWD09xx/EWD952.html,
respectively.


Regards from Vegard's

Re: Linux 2.6.21

[Johannes Stezenbach]

On Sun, Apr 29, 2007, Adrian Bunk wrote:
> On Sun, Apr 29, 2007 at 01:33:16PM -0700, Linus Torvalds wrote:
> > On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > > 
> > > The kernel Bugzilla currently contains 1600 open bugs.
> > 
> > Adrian, why do you keep harping on this, and ignoring reality?
> > 
> > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> 
> OK, how do you suggest to track bugs in a way that doesn't suck?
> 
> Bug reports to linux-kernel have the big problem that they are lost if 
> no developer immediately picks them up.

I suspect some bug reports get ignored deliberately.

Why? Because it's hard to write good bug reports, and thus
a lot of them suck.

In some cases you are lucky and get a test case which makes
the bug easily reproducible, or you get a nice, comprehensible
Oops message, and can pinpoint the bug right away. But usually you
have to interact with the user to get more information, have her
try various patches or do a bisect. And sometimes this interaction
drives you crazy because the user is slow to answer, gives you
incomplete or even false information, and doesn't do what you
ask her to do to narrow the bug, or you just have no clue
what the bug could be etc.

Probably every developer went through this experience a couple of
times, wasting a lot of time and causing a lot of grief. Thus it's
just natural that you learn from the experience to ignore every
bug report which even remotely smells fishy ;-/
Y'know, a lot of developers don't just work for the money, they
work for the fun of it. Thus they try to avoid pain and grief. :-)

Bugzilla just makes this visible because it doesn't forget.
Which is stupid and discouraging for both (potential) bug
reporters and developers.


The lesson to learn is that there are some very valid
reasons why bug reports get ignored (some not mentioned here),
and there's nothing you can do about it. And it has nothing to
do with the method or tool used for reporting or tracking.

And I also think that ignoring bad bug reports _increases_
the software quality, because you can use the saved time
working on something productive. And it makes developers
happier :-)

[Just to avoid misunderstandings: By no means do I advocate
ignoring *every* bug report. But ignoring the bad ones
is just the sane thing to do.]


Johannes

Re: Linux 2.6.21

[Indan Zupancic]

Hello,

On Mon, April 30, 2007 00:36, Johannes Stezenbach wrote:
> The lesson to learn is that there are some very valid
> reasons why bug reports get ignored (some not mentioned here),
> and there's nothing you can do about it. And it has nothing to
> do with the method or tool used for reporting or tracking.
>
> And I also think that ignoring bad bug reports _increases_
> the software quality, because you can use the saved time
> working on something productive. And it makes developers
> happier :-)
>
> [Just to avoid misunderstandings: By no means do I advocate
> ignoring *every* bug report. But ignoring the bad ones
> is just the sane thing to do.]

I don't know, but what about telling the hapless person who went
through the process of posting a bug what's wrong with the bug report?
Or is software quality the only thing you care about, and you don't
want to waste time on learning people to write better bug reports?

If you want to scare away bug reporters, just ignore their first (and
thus likely bad) bug report they write. There isn't much less motivating
than a deafening silence. Just compare it with writing a patch.

Though, in a sense, if the software quality is measured by the number of
bug reports, your tactic might "improve" it indeed.

That said, if someone is an obvious idiot, ignoring saves time. But I
think that's quite rare, and in general you should give the reporter
feedback, and then ignore the bug report. (Until it improves.)

Greetings,

Indan

Re: Linux 2.6.21

[Johannes Stezenbach]

On Mon, Apr 30, 2007, Indan Zupancic wrote:
> On Mon, April 30, 2007 00:36, Johannes Stezenbach wrote:
> > The lesson to learn is that there are some very valid
> > reasons why bug reports get ignored (some not mentioned here),
> > and there's nothing you can do about it. And it has nothing to
> > do with the method or tool used for reporting or tracking.
> >
> > And I also think that ignoring bad bug reports _increases_
> > the software quality, because you can use the saved time
> > working on something productive. And it makes developers
> > happier :-)
> >
> > [Just to avoid misunderstandings: By no means do I advocate
> > ignoring *every* bug report. But ignoring the bad ones
> > is just the sane thing to do.]
> 
> I don't know, but what about telling the hapless person who went
> through the process of posting a bug what's wrong with the bug report?
> Or is software quality the only thing you care about, and you don't
> want to waste time on learning people to write better bug reports?
> 
> If you want to scare away bug reporters, just ignore their first (and
> thus likely bad) bug report they write. There isn't much less motivating
> than a deafening silence. Just compare it with writing a patch.
> 
> Though, in a sense, if the software quality is measured by the number of
> bug reports, your tactic might "improve" it indeed.
> 
> That said, if someone is an obvious idiot, ignoring saves time. But I
> think that's quite rare, and in general you should give the reporter
> feedback, and then ignore the bug report. (Until it improves.)

Well, I'm a moody bastard, and I would hope other people handle
this better than I. However, all the bugzillas of this world
are full with old, ignored bugs, amd I thought this might
serve as an explanation why.

Developers are just humans and if they have no incentive to
act on a bug report they will ignore it. I think this is a
fact that you have to deal with.

It's also not necessarily the fault of the reporter if
a bug report gets ignored, but for every report a developer
has to make a decision to handle it or not, and there are
lots of reasons why he may decide to not handle it, or
at least not now (and then forget about it).

But I'm quite sure that an important bug would be reported
again until fixed.


Johannes

Re: Linux 2.6.21

[Indan Zupancic]

On Mon, April 30, 2007 01:41, Johannes Stezenbach wrote:
> Developers are just humans and if they have no incentive to
> act on a bug report they will ignore it. I think this is a
> fact that you have to deal with.

Reporters are just humans too and if they have no incentive to
post bugs they won't. So it's a lose/lose situation, really.
With a group of people working together they should try to
motivate each other, not demoralize everyone.

(I know, each bug report is a pain, voicing someone's failure.
So ignoring it might make people feel better, but it doesn't
fix anything.)


> It's also not necessarily the fault of the reporter if
> a bug report gets ignored, but for every report a developer
> has to make a decision to handle it or not, and there are
> lots of reasons why he may decide to not handle it, or
> at least not now (and then forget about it).

True. There's also a difference between a bad bug report and
one that a specific developer won't handle. In the former case
anyone could recognize it and tell the reporter about it. The
latter is a bit trickier, but if the developer thinks about
looking at it later, he better can tell the reporter just that.
A short "I'll take a look at it, later, when I've more time."
is so much better than plain silence.


> But I'm quite sure that an important bug would be reported
> again until fixed.

I wouldn't be so sure about that. What's worse, why would the
reporter bother telling that the bug is fixed in version N+1?
No one cared about it anyway, so there's no one to tell it to.
That would explain a lot open bugs too.

Greetings,

Indan

Re: Linux 2.6.21

[Matthias Andree]

On Mon, 30 Apr 2007, Indan Zupancic wrote:

> I don't know, but what about telling the hapless person who went
> through the process of posting a bug what's wrong with the bug report?

It's a tedious process you keep doing over and over and over and over again,
and my experience shows it's sheer luck if people can actually fill in
the missing bits given the list.

Usually you have to ask thrice to obtain even the most essential
information such as version. Let alone vendor patches.

Anyways, the solution to this problem is someone _politely_ asking
reporters to provide necessary information and also point out that they
cannot ever hope to have their bug fixed without making a best-effort
attempt at answering all questions the first time they're being asked.

There are notable exceptions, people pinpointing code fragments at fault
and everything, but those are usually tech people and not end users.

> That said, if someone is an obvious idiot, ignoring saves time. But I
> think that's quite rare, and in general you should give the reporter
> feedback, and then ignore the bug report. (Until it improves.)

And that is what happens all too often (not in absolute figures, but in
the developer's perception of it) - insufficient information to debug.

Yes I know, some of the bugs hide themselves so well you actually need
four or five reports by different people to actually pinpoint the bug,
perhaps accompanied by insufficient interface documentation that make it
difficult to verify assumptions/expectations or assess potential
solutions (such as the res_init() issue in fetchmail, or probably the khubd going
south issue in Linux), but that's not the point.

-- 
Matthias Andree

Re: Linux 2.6.21

[Diego Calleja]

El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> escribió:

> Bugzilla might not be perfect, but it works and it's better than doing 
> it by hand.

The good thing about the wiki is that it doesn't exclude bugzilla. It's
just a "regressions list", it doesn't intends to replace bugzilla. If a bug
doesn't gets fixed for a while, I don't think it's very useful to keep it
forever in the list like you do in the bugzilla, because I don't think
it's possible to fix every single bug, and it steals you time to fix bugs
that you are able to fix.

It's not great but it's the best clone of you we've found 8)

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 10:56:57PM +0200, Diego Calleja wrote:
> El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> escribió:
> 
> > Bugzilla might not be perfect, but it works and it's better than doing 
> > it by hand.
> 
> The good thing about the wiki is that it doesn't exclude bugzilla. It's
> just a "regressions list", it doesn't intends to replace bugzilla. If a bug
> doesn't gets fixed for a while, I don't think it's very useful to keep it
> forever in the list like you do in the bugzilla, because I don't think
> it's possible to fix every single bug, and it steals you time to fix bugs
> that you are able to fix.
> 
> It's not great but it's the best clone of you we've found 8)

What exactly is the purpose of the 2.6.21 regressions list in the Wiki?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Michal Piotrowski]

On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Sun, Apr 29, 2007 at 10:56:57PM +0200, Diego Calleja wrote:
> > El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> escribió:
> >
> > > Bugzilla might not be perfect, but it works and it's better than doing
> > > it by hand.
> >
> > The good thing about the wiki is that it doesn't exclude bugzilla. It's
> > just a "regressions list", it doesn't intends to replace bugzilla. If a bug
> > doesn't gets fixed for a while, I don't think it's very useful to keep it
> > forever in the list like you do in the bugzilla, because I don't think
> > it's possible to fix every single bug, and it steals you time to fix bugs
> > that you are able to fix.
> >
> > It's not great but it's the best clone of you we've found 8)
>
> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?

It's for -stable team.

I didn't saw any 2.6.22-stuff regression yet.

>
> cu
> Adrian

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 11:16:51PM +0200, Michal Piotrowski wrote:
> On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
>> On Sun, Apr 29, 2007 at 10:56:57PM +0200, Diego Calleja wrote:
>> > El Sun, 29 Apr 2007 22:17:29 +0200, Adrian Bunk <bunk@stusta.de> 
>> escribió:
>> >
>> > > Bugzilla might not be perfect, but it works and it's better than doing
>> > > it by hand.
>> >
>> > The good thing about the wiki is that it doesn't exclude bugzilla. It's
>> > just a "regressions list", it doesn't intends to replace bugzilla. If a 
>> bug
>> > doesn't gets fixed for a while, I don't think it's very useful to keep 
>> it
>> > forever in the list like you do in the bugzilla, because I don't think
>> > it's possible to fix every single bug, and it steals you time to fix 
>> bugs
>> > that you are able to fix.
>> >
>> > It's not great but it's the best clone of you we've found 8)
>>
>> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
>
> It's for -stable team.
>...

Did Greg or Chris say they have spare time for going through this list?

> Regards,
> Michal

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Michal Piotrowski]

On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Sun, Apr 29, 2007 at 11:16:51PM +0200, Michal Piotrowski wrote:
> > On 29/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> >> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> >
> > It's for -stable team.
> >...
>
> Did Greg or Chris say they have spare time for going through this list?

It's not a list "hello -stable team, here are a new regressions, please fix it".

> cu
> Adrian

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Thomas Gleixner]

On Sun, 2007-04-29 at 23:21 +0200, Adrian Bunk wrote:
> >> > It's not great but it's the best clone of you we've found 8)
> >>
> >> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> >
> > It's for -stable team.
> >...
> 
> Did Greg or Chris say they have spare time for going through this list?

Don't be silly, did any of the developers say, that he has spare time to
read your regression lists ?

Michal posted it to LKML with the relevant developers in CC including
the stable team, so they are in the loop for updates, which is a Good
Thing!

Hey, you did this yourself. It is a great help and as your lists did,
Michals list pointed me to a bug, which I would have missed otherwise. 

So what are you complaining about ? Folks stepped up and built a
regression list and posted it to LKML. What's wrong with that ?

	tglx

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 11:52:01PM +0200, Thomas Gleixner wrote:
> On Sun, 2007-04-29 at 23:21 +0200, Adrian Bunk wrote:
> > >> > It's not great but it's the best clone of you we've found 8)
> > >>
> > >> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> > >
> > > It's for -stable team.
> > >...
> > 
> > Did Greg or Chris say they have spare time for going through this list?
> 
> Don't be silly, did any of the developers say, that he has spare time to
> read your regression lists ?

It worked because several people (including Linus) emphasized that 
fixing regressions from this list was important.

And it failed because many regressions still stayed unfixed and some 
even undebugged.

> Michal posted it to LKML with the relevant developers in CC including
> the stable team, so they are in the loop for updates, which is a Good
> Thing!
> 
> Hey, you did this yourself. It is a great help and as your lists did,
> Michals list pointed me to a bug, which I would have missed otherwise. 
> 
> So what are you complaining about ? Folks stepped up and built a
> regression list and posted it to LKML. What's wrong with that ?

If it works it's perfectly fine.

> 	tglx

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Thomas Gleixner]

On Mon, 2007-04-30 at 00:19 +0200, Adrian Bunk wrote:
> > Don't be silly, did any of the developers say, that he has spare time to
> > read your regression lists ?
> 
> It worked because several people (including Linus) emphasized that 
> fixing regressions from this list was important.

Right. Simply because these lists are assembled by someone

- who knows how to pick that reports from the mailinglists
- who knows how to sort them in a useful way
- who knows how to add the relevant folks on CC
- ....

Can you see the pattern ?

> And it failed because many regressions still stayed unfixed and some 
> even undebugged.

No it failed not. It is not perfect. Way more bugs, which have been
fixed or are in the debugging process, would have been unnoticed and
ignored otherwise.

> > So what are you complaining about ? Folks stepped up and built a
> > regression list and posted it to LKML. What's wrong with that ?
> 
> If it works it's perfectly fine.

It will work not much different from your lists. It'll be not perfect
either.

	tglx

Re: Linux 2.6.21

[Andi Kleen]

> Right. Simply because these lists are assembled by someone
> 
> - who knows how to pick that reports from the mailinglists
> - who knows how to sort them in a useful way
> - who knows how to add the relevant folks on CC

That all needs to be done by someone initially yes. But then tracking what happens
afterwards is something that can be distributed. A difficult bug can take a long
time to resolve and generate a lot of messages; you don't want to require
the initial sorter to handle all that too.

It's much more scalable to let the developers update the state themselves
then once they handle the bug.

-Andi

Re: Linux 2.6.21

[Michal Piotrowski]

On 30/04/07, Andi Kleen <andi@firstfloor.org> wrote:
> > Right. Simply because these lists are assembled by someone
> >
> > - who knows how to pick that reports from the mailinglists
> > - who knows how to sort them in a useful way
> > - who knows how to add the relevant folks on CC
>
> That all needs to be done by someone initially yes. But then tracking what happens
> afterwards is something that can be distributed. A difficult bug can take a long
> time to resolve and generate a lot of messages; you don't want to require
> the initial sorter to handle all that too.
>
> It's much more scalable to let the developers update the state themselves
> then once they handle the bug.

The actual list of known regressions is wiki based. Everyone can
update bug status, add references etc.

>From time to time, I'll send this list to right people.

>
> -Andi
>

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Andi Kleen]

> The actual list of known regressions is wiki based. Everyone can
> update bug status, add references etc.

Well do they know about it? 

Also something a little more structured would seem better for this.
How do you query a wiki? 

-Andi

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 12:33:30AM +0200, Thomas Gleixner wrote:
> On Mon, 2007-04-30 at 00:19 +0200, Adrian Bunk wrote:
>...
> > And it failed because many regressions still stayed unfixed and some 
> > even undebugged.
> 
> No it failed not. It is not perfect. Way more bugs, which have been
> fixed or are in the debugging process, would have been unnoticed and
> ignored otherwise.
>...

It depends on what you consider failure and what you consider success.

For me, it failed. Not because it wasn't perfect, but because we could 
have done much better with fixing the known regressions, and also by not 
introducing several regressions between the last -rc and the final 
kernel (and people who did test -rc7 and would most likely also have 
tested an -rc8 ran into them).

> 	tglx

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Michal Piotrowski]

On 30/04/07, Adrian Bunk <bunk@stusta.de> wrote:
> On Mon, Apr 30, 2007 at 12:33:30AM +0200, Thomas Gleixner wrote:
> > On Mon, 2007-04-30 at 00:19 +0200, Adrian Bunk wrote:
> >...
> > > And it failed because many regressions still stayed unfixed and some
> > > even undebugged.
> >
> > No it failed not. It is not perfect. Way more bugs, which have been
> > fixed or are in the debugging process, would have been unnoticed and
> > ignored otherwise.
> >...
>
> It depends on what you consider failure and what you consider success.

I hope that this discussion about bugs will change something in Linux
regressions front.

Huge thanks to you for that.

>
> For me, it failed. Not because it wasn't perfect, but because we could
> have done much better with fixing the known regressions, and also by not
> introducing several regressions between the last -rc and the final
> kernel (and people who did test -rc7 and would most likely also have
> tested an -rc8 ran into them).
>
> >       tglx
>
> cu
> Adrian

Regards,
Michal

-- 
Michal K. K. Piotrowski
Kernel Monkeys
(http://kernel.wikidot.com/start)

Re: Linux 2.6.21

[Diego Calleja]

El Sun, 29 Apr 2007 23:10:28 +0200, Adrian Bunk <bunk@stusta.de> escribió:

> What exactly is the purpose of the 2.6.21 regressions list in the Wiki?

AFAIK, submitting its contents to the list periodically CCing the developers,
like you did with your lists.

If developers care to fix it or not or how much Linus cares about that list before
releasing a new version is another question. I think it's useful because it makes
those bugs look more important than the 1600 stored in the bugzilla...it won't
help to fix those 1600, but it attracts some attention over the "release critical"
ones and encourages developers to fix them, even if not all of them get fixed.

I don't think you can do many other things to get as much bugs fixed as possible,
unless we reward bug fixers with weekends in the Playboy mansion. I think the
fundamental question here is: is there a way to make hackers follow and fix _all_
the bugs? I'd love it was possible, but AFAIK all the projects that have tried to
be ultra-stable and have adopted a policy to fullfill such goal have fallen behind
of competing projects that cared more about working in improving their software.

Re: Linux 2.6.21

[Rafael J. Wysocki]

On Sunday, 29 April 2007 23:51, Diego Calleja wrote:
> El Sun, 29 Apr 2007 23:10:28 +0200, Adrian Bunk <bunk@stusta.de> escribió:
> 
> > What exactly is the purpose of the 2.6.21 regressions list in the Wiki?
> 
> AFAIK, submitting its contents to the list periodically CCing the developers,
> like you did with your lists.
> 
> If developers care to fix it or not or how much Linus cares about that list before
> releasing a new version is another question. I think it's useful because it makes
> those bugs look more important than the 1600 stored in the bugzilla...it won't
> help to fix those 1600, but it attracts some attention over the "release critical"
> ones and encourages developers to fix them, even if not all of them get fixed.
> 
> I don't think you can do many other things to get as much bugs fixed as possible,
> unless we reward bug fixers with weekends in the Playboy mansion. I think the
> fundamental question here is: is there a way to make hackers follow and fix _all_
> the bugs? I'd love it was possible, but AFAIK all the projects that have tried to
> be ultra-stable and have adopted a policy to fullfill such goal have fallen behind
> of competing projects that cared more about working in improving their software.

Apart from this many bugs are found and get fixed in the process of developing
new code, so the 'ultra stable' approach is not really practical.

Greetings,
Rafael

Re: Linux 2.6.21

[Francois Romieu]

Diego Calleja <diegocg@gmail.com> :
[...]
> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

The rtl8139 entry is outdated.

-- 
Ueimor

Re: Linux 2.6.21

[Lennart Sorensen]

On Sun, Apr 29, 2007 at 09:40:07PM +0200, Diego Calleja wrote:
> So far, it seems that most of people's opinion WRT to bug reporting and trackingcan
> be divided into 2 groups:
> 
> - People who argues (and they're right) that bugzilla and web interfaces in general
>   suck and that email + an "Adrian-like" solution works better
> 
> - People who argues that a bug tracker better than a mailing list is absolutely
>   needed (and they're right). They also argue that while bugzilla sucks, it's
>   better than nothing.
> 
> There's a common point between both groups: bugzilla sucks. The ideal
> solution would be to replace bugzilla with some alternative and better
> opensource bug tracking software, but I doubt it exists (there must be a
> reason why everybody uses bugzilla). A good bug tracker should feel like
> it makes your work easier, instead of making you feel like you're wasting
> time (which is what bugzilla does)

Debian has a bug track system which interacts mainly with the users
through email.  Seems rather nice to use and doesn't make you sign up to
submit things, and has no issues with mailing lists being "subscribed"
to a bug.  Probably a bit complicated to setup though.

> I don't see why a web interface bug tracker should be bad for bug tracking,
> as long as it's good and integrates 100% in the mailing lists. In my humble
> opinion the "perfect" bug tracker for Linux should be something like this:
> - Has a email interface (like the Debian bug tracking database).
> - Has a web interface that completely follows the email threads
>    (reading/posting), but make the comments real emails, not just
>     database fields.
> If done well (unlike the current bugzilla-to-email hack), it should possible
> to do many nice things, like add a lkml bug report to the bug tracking
> database (which shouldn't be a "real" database, but just an lkml mail
> archive with a list of message IDs that are considered a bug and its state)
> by just replying the thread, CCing the bug tracker and telling him to include
> the thread in the database.

So in other words, basically the debian bug track system, except
perhaps with an ability to submit bugs through a web interface too
rather than just email and reportbug (which I believe uses email).

> So unless someone is willing to write such tool (which I doubt, since it
> doesn't looks easy), all this discussion seems pointless, and we should
> stick with this http://kernelnewbies.org/known_regressions page
> which is showing to be quite useful :)

--
Len Sorensen

Re: Linux 2.6.21

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: Sun, 29 Apr 2007 20:09:09 +0200

> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.

That explains why my bugs don't get looked at for months if
not years when I submit them to such projects.

I reported a bug that eats people's hard disks due to a bug
in the X.ORG PCI support code on sparc, NOBODY has fixed
the bug in 2 years even though a full bugzilla entry with
even a full patch fix is in there.

Bugzilla sucks, emails rules because it is in your face and
gets people to work on things.

Re: Linux 2.6.21

[Andi Kleen]

> I reported a bug that eats people's hard disks due to a bug
> in the X.ORG PCI support code on sparc, NOBODY has fixed
> the bug in 2 years even though a full bugzilla entry with
> even a full patch fix is in there.

Well but at least they could find it again if they wanted.
If you sent it by email and it had gotten lost for some reason
(nobody interested, which seems to be the real issue here) then
it would be lost forever.

Of course a database is not a silver bullet by itself, it still
needs people to use it correctly and then actually work on the bugs.

A database just a tool to move some work humans are bad at (keeping track 
of a lot of data and deriving trends out of it) to a a computer which is 
much better at this.

-Andi

Re: Linux 2.6.21

[David Miller]

From: Andi Kleen <andi@firstfloor.org>
Date: Sun, 29 Apr 2007 23:26:43 +0200

> > I reported a bug that eats people's hard disks due to a bug
> > in the X.ORG PCI support code on sparc, NOBODY has fixed
> > the bug in 2 years even though a full bugzilla entry with
> > even a full patch fix is in there.
> 
> Well but at least they could find it again if they wanted.
> If you sent it by email and it had gotten lost for some reason
> (nobody interested, which seems to be the real issue here) then
> it would be lost forever.

WRONG!  If I have sent it to the main developer list the damn patch
would be applied by now.

WHY?

BECAUSE EMAIL ENGAGES PEOPLE AND BUGZILLA DOES NOT!

Nobody looks at the bugzilla because there is too much junk in there
to make the signal any useful to search for, there's simply too much
noise.

Re: Linux 2.6.21

[Andi Kleen]

> BECAUSE EMAIL ENGAGES PEOPLE AND BUGZILLA DOES NOT!
> 
> Nobody looks at the bugzilla because there is too much junk in there
> to make the signal any useful to search for, there's simply too much
> noise.

That means just x.org doesn't have a working bugmaster setup.
Again a technical solution doesn't fix misorganization or missing people;
it just pushes some work humans are at to computers if you have the right structure

The normal bugzilla workflow is that some people categorize the bugs, 
ask the necessary questions and then figure out which developer
to assign it to. Then the developer doesn't end up with "too much noise"
but just a limited set of bugs to look at.

And the responsible developer then gets an email and looks at the bug.

This already happens to some limited fashion in kernel.org bugzilla:
I get bugs assigned occasionally and while it's slow I tend to look
at (near) all of them and try to improve things there.

If a single developer ends up with too many bugs this way or there
is nobody to assign a bug to or nobody processes the incoming bugs
then the project has a problem. Yes bugzilla doesn't work then 
if the project is not well organized.

But that's in no way different from what would happen with your
email sent to a mailing list if it had the same problem.

-Andi (who finds it a bit bizarre he has to explain the concept
of "things can get easier when some work is pushed to computers"
concept to a hacker) 

Re: Linux 2.6.21

[Simon Arlott]

On 29/04/07 19:09, Andi Kleen wrote:
>>  - a lot of reporters will not use bugzilla, because it's damn 
>>    inconvenient even for reporting. If you propose something that uses 
> 
> Don't think that's true. There are plenty of projects who only
> accept bugs through bugzilla (mozilla, various distributions, etc.) 
> and I don't see any evidence of your claim being true.

These projects are probably losing plenty of trivial bug reports 
from people who shouldn't have to register with a bug tracker and 
fill in a long form every time to submit a short bug report.

> Sure there will be always people who cannot be bothered
> to use any kind of interface for bugs, but then

Most of the time bugzilla appears to be a great way to ignore 
bugs. Every large project seems to have one with bugs that 
stay open for years - bugzilla's own inability to quick search 
without javascript[1] being a good example (it's fixed now). 
If no one can get around to fixing reported bugs why should 
anyone bother submitting more?

They even released their software with a link to the bug if 
you tried to do a quick search without javascript:
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=70907
"THIS BUG IS FIXED IN BUGZILLA 2.22. D..."

> these are unlikely to stay on board during a longer 
> remote debugging q'n'a session either. So those people
> can be just ignored; they essentially don't exist in
> the bug report universe.

How will the people you ignore get their bugs fixed?

> I don't think the "keep it in Andrew's/Adrian's head" method
> is going to scale longer term at least (and one of them has 
> already thrown in the towel) 

People are doing something about this, at least for tracking 
the regressions at http://kernelnewbies.org/known_regressions 
(which is read-only unless you register, people have already 
commented on this).

> The "send it to a gigantic mailing list and hope someone catches
> it" method also doesn't seem to be that great. At least there
> are lots of lost reports in my experience this way.

I have had experience of this with the dvb mailing list (this was 
a couple of months before their news post referencing bugzilla 
too), even when I included a patch to fix it.

Someone suggested another way to fix a bug and I replied that it 
worked - that fix never went anywhere else and other people could 
be having the same problem today because it's still not been 
applied anywhere. (I will submit a -1 +1 patch myself when 
I have time this week).

> I suspect the real reason is more "Linus doesn't like web interfaces
> for no particular good reason". Not much can be done about that.
> Well perhaps someone can write a gopher based bugzilla interface 
> or something to solve that instead @) 

Bugzilla's advanced search interface is far too crowded, that 
should be clear to anyone. The simpler search usually isn't much 
use because either it finds too many bugs or "zarro boogs" and 
you're left wondering if the bug is there or not.

-- 
Simon Arlott

Re: Linux 2.6.21

[Matthias Andree]

On Sun, 29 Apr 2007, Linus Torvalds wrote:

> What works is somebody who is a bugmaster, and it doesn't really matter 
> *what* bug tracker he points to (bugzilla being one of the possibilities, 
> although not necessarily the best, and absolutely NOT the only choice), 
> and turn them into emails. Once they are emails, bugzilla can track them.

While I'm not reading this entire thread for lack of time:

This looks exactly like the kind of bug tracking that Timo Sirainen of
Dovecot fame concocted and talked about on the dovecot-users list,
quote:

| Dovecot BTS
| -----------
|
|The preferred way to report bugs is to send them to dovecot-bugs at
|dovecot.org. The only thing it does is prefix the subject line with [BUG
|#nnn] and forward it to dovecot at dovecot.org.
|
|Now everyone can reply to it just as it was a normal mailing list mail.
|As long the subject contains the "[BUG #nnn]" prefix, it's part of the
|bug.
|
|Existing mailing list threads can also be turned into bugs by replying
|to the thread's root message with To: dovecot-bugs at dovecot.org. This
|again causes the new reply to contain [BUG #nnn] prefix.
|
|Then comes the web part. [...]"      -- quoting Timo Sirainen
<http://www.dovecot.org/list/dovecot/2007-January/018786.html>

Perhaps it's a stripped-down sibling of the Debian BTS without itself
knowing :-) anyways, it's E-Mail centric, low ceremony, devised from the
currently implemented workflow.

I haven't followed its details, portability, shape, state or anything,
but the requirements appear very similar to Linus's -- at least to me
with entirely outside view (I've never used kernel bugzilla), so it
might be a starting point, conceptionally or with some luck even
implementation-wise.

(Yes I know it's going to be tough to obtain all the precioussssssss
bugssssssssssss from bugzzzzzzzzzzzzzzzilla but anyways, if nobody likes
to use it, something will be done and if only neglect...)

HTH

-- 
Matthias Andree

Re: Linux 2.6.21

[Theodore Tso]

On Sun, Apr 29, 2007 at 03:15:42PM +0200, Andi Kleen wrote:
> This means we need people who figure out who to assign bugs too.
> Aka bugmasters.
> 
> BTW one big problem in our current bugzilla is that a lot of people
> cannot reassign bugs they don't own. I sometimes see bugs that I don't
> own bug I know who is responsible, but bugzilla doesn't allow me to do it.
> 
> So I think what would help:
> 
> - Ask more people to just categorize and reassign bugs (anybody interested?)
> - Give more people in bugzilla the power to reassign arbitary bugs
> (bugzilla maintainers would need to do that)

Folks might want to take a look at the Debian Bug Tracking System
(BTS).  It has a web interface which you can use to query history, but
*everything* is e-mail driven, and the way you submit, close, update,
tag/classfy bugs --- everything --- is via e-mail.

More importantly, anyone is allowed to recategorize and reassign bugs.
If someone does so maliciously or incorrectly, you can always revert
it, and if someone is being truly malicious, you can always blacklist
that one person.  It this respect, it is far more wiki-like than
bugzilla, which has always been too much like a straightjacket.

It's not perfect, but it's better than bugzilla --- but then again,
just about *anything* would be better than bugzilla.  (Hmm, except
maybe SourceForge's very tragic bug tracking system... :-)

Of course, as Linus has said, it's not a complete solution --- you
still need humans to be smart about things --- but if the goal is to
make it easier to archive and track information about a bug, at
*least* with the Debian BTS, when you reply to an e-mail message, the
reply is automatically appended to the bug log!
	
						- Ted

Re: Linux 2.6.21

[Dave Jones]

On Sun, Apr 29, 2007 at 07:55:35PM -0400, Theodore Tso wrote:

 > but if the goal is to
 > make it easier to archive and track information about a bug, at
 > *least* with the Debian BTS, when you reply to an e-mail message, the
 > reply is automatically appended to the bug log!

bugzilla does that too.

	Dave

-- 
http://www.codemonkey.org.uk

Re: Linux 2.6.21

[Björn Steinbrink]

[Oops, the first try of this mail got out from my local address, sorry]

On 2007.04.29 19:55:35 -0400, Theodore Tso wrote:
> On Sun, Apr 29, 2007 at 03:15:42PM +0200, Andi Kleen wrote:
> > This means we need people who figure out who to assign bugs too.
> > Aka bugmasters.
> > 
> > BTW one big problem in our current bugzilla is that a lot of people
> > cannot reassign bugs they don't own. I sometimes see bugs that I don't
> > own bug I know who is responsible, but bugzilla doesn't allow me to do it.
> > 
> > So I think what would help:
> > 
> > - Ask more people to just categorize and reassign bugs (anybody interested?)
> > - Give more people in bugzilla the power to reassign arbitary bugs
> > (bugzilla maintainers would need to do that)
> 
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.
> 
> More importantly, anyone is allowed to recategorize and reassign bugs.
> If someone does so maliciously or incorrectly, you can always revert
> it, and if someone is being truly malicious, you can always blacklist
> that one person.  It this respect, it is far more wiki-like than
> bugzilla, which has always been too much like a straightjacket.
> 
> It's not perfect, but it's better than bugzilla --- but then again,
> just about *anything* would be better than bugzilla.  (Hmm, except
> maybe SourceForge's very tragic bug tracking system... :-)
> 
> Of course, as Linus has said, it's not a complete solution --- you
> still need humans to be smart about things --- but if the goal is to
> make it easier to archive and track information about a bug, at
> *least* with the Debian BTS, when you reply to an e-mail message, the
> reply is automatically appended to the bug log!

I've started hacking on some bash scripts to do the e-mail part, based
on the requirements I've gathered from reading this thread. So far, I
got one script that can be plugged into procmail and processes mails to
do the following (right now):

 - Create a bug
 - Set the bug type to "regression"
 - Update the timestamp of the last action for that bug
 - Assign the bug to a subsystem
 - Track the submitter and whoever grabs the bug

More (hopefully) to come...

Those commands are just added to any email in the thread discussing a
bug like this:
@bugthing
mine		# Mark myself as owner
regression	# Mark this bug as a regression (used for reports)
subsystem mm	# Assign the bug to mm
needinfo	# Tell the tracker to bug the reporter if nothing happens
Thanks

That block can appear anywhere in the email, so if you're discussing
some problem on lkml and want to track that bug, you can just add such a
block to your email and turn an untracked email conversion into a
tracked bug.

Tracking does _not_ mean that all emails are stored, those can be looked
up on lkml.org or MARC, where the created reports will probably contain
URLs to the latter, because it supports lookups based on message ids.

Tracking does just mean that the state of the bug is stored somewhere.
That means (currently, suggestions welcome):

 - What's its name? (E-Mail subject)
 - Who reported it?
 - Who (if any) stepped up to own that bug?
 - What type of bug is it?
 - Which subsystem does it belong to?
 - What's its current state? (new, owned, fixed, ...)
 - When did the last action on this bug happen?

Based on that information, I've started writing some scripts that create
"reports" (all of them currently being pretty incomplete):

 - Bugs that belong to a specific subsystem (on request, currently
   through a procmail triggered script; this is meant to satisfy
   Adrian's request of asking for example for all SATA bugs.)

 - Bugs that are not owned and didn't see any action for one week
   (cronjob; to lkml, the submitter, the recipients of the bug mail and
   maybe folks who subscribed to such mails)

 - All bugs marked as regressions (complete list to lkml, personalised
   lists to each owner of any regressions, probably sent on request)

 - Annoy the bug submitter if someone tagged the bug with "moreinfo" and
   there wasn't any reply from the submitter within one week

 - Annoy the bug owner if he didn't do anything about his bug within
   one week

Those "reports" should push the relevant parties into looking at those
bugs again while being terse, like Adrian's regression list and not
annoying anyone but those who are most likely to be interested.

Bugs in which noone shows interest (i.e. nothing happens at all), should
probably be removed after 2-3 weeks, with a notice going to the
submitter and recipients of the bug's emails as well as those subscribed
to receiving reports of non-owned inactive bugs. Well, unless those bugs
are regressions, we should probably bug about them until they get
closed.


That's my plan so far. It's probably going to be ugly and all, but I
wanted to get something done quick (and learn a bit about bash scripting
along the way) to see if it works at all, and what you guys think about
that approach. Maybe I can write a less ugly version later, or someone
picks up the approach and writes a better implementation himself. But
maybe the preliminary one (once it is somewhat useable) could be tested
by a few brave folks to see if the approach works out at all...

I don't know about the Debian BTS internals, but IMHO, the mentioned
capability of being email-based isn't enough for Linux.  Things might
start out on lkml as some small issue, but suddenly there's some reason
to track a bug that is discussed in an already existing thread. If you
file it into an email based system, you loose the entire history of that
discussion if you look at it from the BTS.  With the tracker-only
approach, you simply stay email based and if required, you just lookup
the right thread using the information from the tracker and go back to
email, if you don't have those emails, the public archives most probably
do.

And far more important are IMHO the reports, which should distribute the
work of tracking bugs a bit. If a regression is found, the reporter, the
owner or anyone reading the thread, can mark it as such and noone is
forced to track _all_ regressions. The wiki helps with this, too, but
IMHO it's neither easy nor fast enough to look up and change that patch
manually for anyone to enjoy that. Also, the automatic reminders should
help (a bit) to not loose too many bugs, just because the person
responsible for it got distracted (and forgot about it) or because the
bug submitter forgets that he needed to provide more info, which he
couldn't do at the time the request hit his inbox (I do that all the
time).

Does this sound like something usable in the long run? Do I just waste
my time? Do I miss something very important?

Björn

Re: Linux 2.6.21

[Andi Kleen]

Theodore Tso <tytso@mit.edu> writes:
> 
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.

Bugzilla can do that too. But I'm not convinced this is a good idea.
We had this some years ago with the jitterbug experiment and usually
it tended to faithfully keep track of very long off topic threads
that had drifted long from the original bug. The resulting collections
of emails usually not very useful.

While other interfaces might be a culture shock for some people
at least they force them to concentrate on the particular issue --
contribute to a specific bug.

-Andi

Re: Linux 2.6.21

[Kyle Moffett]

It might not be bad to write up an email-based BTS-alike bug-tracking  
system just for the Linux kernel.  It should probably even be  
implemented 100% via email at first, with a web-based status viewer  
as a later add-on.  Here's a possible email format:

[kbugger: action1 arg1 arg2 ..., action2 arg1 arg2 ...]

Make it almost totally message-id and thread based, and make it an  
implicit part of LKML (IE: subscribe the kbugger program to LKML).   
People who are flagged "admin" may ban/unban users and make certain  
large-scale changes.  Supported actions:

create, create-parent, create-thread:
   Create a new bug associated with this message.
   The arguments specify the title.
   This would automatically happen for new threads with titles like  
"[BUG] foo: It's broken"

merge:
   Merges the current bug and/or email thread into an existing bug.
   The arguments are a list of bug numbers and/or message-IDs to  
merge together with this one.

prune, prune-parent, prune-thread:
   Prunes a given thread from the current bug.
   Optional argument specifies a referential message-ID

settitle:
   Change the title of the current bug

fixed, broken:
   Mark the bug as fixed or broken in a particular version/configuration
   Arguments are used as opaque strings representing configurations  
where it is known to be fixed or broken.  For example [kbugger: fixed  
2.6.16 2.6.20-x86, broken 2.6.20-ppc] would just store the list of  
strings and statuses.  If the bug was auto-created with a title like  
"[BUG ppc] foo: It's broken" or "[BUG 2.6.20] bar: I dunno", then the  
argument to the [BUG] title portion will be auto-passed to [kbugger:  
broken].

status:
   Get a brief status report on the current bug.

info:
   Get a detailed status report on the current bug.

history:
   Get detailed information about the history of the current bug.   
This only sends the reply to the author.

stop:
   Stop parsing the rest of this email.  Useful when teaching  
somebody about kbugger commands via an email sent to kbugger itself.

The results of the kbugger statements in an email will be sent as a  
reply to the original message and "To:" the original sender, "CC:"- 
ing the targets of the message, so if the original [kbugger: create]  
post went to LKML then the reply will go there as well for people to  
read and for it to be archived by kbugger as part of the status  
history of that bug.  All emails it receives will be autoparsed for  
commands, however it should be coded to ignore all text in emailed  
patches, and it should support the [kbugger: stop] command to halt  
parsing for cases where you need to send plain kbugger commands via  
email.

If somebody was feeling unusually brave they might add some  
keyword<=>author bayesian tracking so that it could automatically  
discover the keywords in emails that particular authors reply to and  
helpfully forward a kbugger info email to that person with a link to  
the archives for the original thread.  If somebody didn't want to  
receive such info messages they could click a link or send a  
[kbugger: no-auto-forward] command to blacklist themselves from  
receiving automatic forwards ([kbugger: auto-forward] to re-enable).   
Maybe even [kbugger: not-for-me ...] and [kbugger: for-me] commands  
which take bug numbers and message-IDs to tune the keyword lists.

I may try working on something like that this week if I get the time.

Cheers,
Kyle Moffett

Re: Linux 2.6.21

[Johannes Stezenbach]

On Sun, Apr 29, 2007, Theodore Tso wrote:
> 
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.

Plus it has the very user-friendly reportbug and querybts
commandline interfaces. No going to web pages to query
bugs, and you can just download the email thread for
a bug report as an mbox file and then reply via email.

(Querybts currently only works if you know the package
name, it can't search across all packages so it wouldn't
be that useful for the kernel. But for Debian packages
this tool is gold.)


Johannes

Re: Linux 2.6.21

[David Lang]

On Sun, 29 Apr 2007, Theodore Tso wrote:

> On Sun, Apr 29, 2007 at 03:15:42PM +0200, Andi Kleen wrote:
>> This means we need people who figure out who to assign bugs too.
>> Aka bugmasters.
>>
>> BTW one big problem in our current bugzilla is that a lot of people
>> cannot reassign bugs they don't own. I sometimes see bugs that I don't
>> own bug I know who is responsible, but bugzilla doesn't allow me to do it.
>>
>> So I think what would help:
>>
>> - Ask more people to just categorize and reassign bugs (anybody interested?)
>> - Give more people in bugzilla the power to reassign arbitary bugs
>> (bugzilla maintainers would need to do that)
>
> Folks might want to take a look at the Debian Bug Tracking System
> (BTS).  It has a web interface which you can use to query history, but
> *everything* is e-mail driven, and the way you submit, close, update,
> tag/classfy bugs --- everything --- is via e-mail.
>
> More importantly, anyone is allowed to recategorize and reassign bugs.
> If someone does so maliciously or incorrectly, you can always revert
> it, and if someone is being truly malicious, you can always blacklist
> that one person.  It this respect, it is far more wiki-like than
> bugzilla, which has always been too much like a straightjacket.
>
> It's not perfect, but it's better than bugzilla --- but then again,
> just about *anything* would be better than bugzilla.  (Hmm, except
> maybe SourceForge's very tragic bug tracking system... :-)
>
> Of course, as Linus has said, it's not a complete solution --- you
> still need humans to be smart about things --- but if the goal is to
> make it easier to archive and track information about a bug, at
> *least* with the Debian BTS, when you reply to an e-mail message, the
> reply is automatically appended to the bug log!

this, and the fact that anyone can add to the bug log by just sending an e-mail 
are a nice feature

however, I had a reason to take a look at the debian BTS late last week to see 
if the bugs and patch that I sent to the sysklog maintainer (both debian and 
upstream) got included in debian 4.0.

talk about depressing.

there are about a dozen bugs _with_ patches sitting in the queue for several 
years

David Lang

Re: Linux 2.6.21

[Russell King]

On Sun, Apr 29, 2007 at 12:49:04AM +0200, Adrian Bunk wrote:
> On Sat, Apr 28, 2007 at 09:27:01PM +0100, Russell King wrote:
> > On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> > > We are already quite good at ignoring bug reports that come through 
> > > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> > > than 1600 open bugs because this tells how bad we are at handling bugs.
> > > How many thousand bug reports have been ignored during the same time on 
> > > linux-kernel?
> > 
> > However, look at this bug:
> > 
> >   http://bugme.osdl.org/show_bug.cgi?id=7760
> > 
> > It's outside my knowledge to be able to fix for various reasons:
> >...
> > I'm personally very tempted to close it as "won't fix" (I wish there was
> > a "can't fix" category.)
> >...
> 
> So this is a completely debugged bug in a well-maintained subsystem
> (no matter what the status in Bugzilla is).

You're being very optimistic.

I'm not sure where you get the idea that it's "completely debugged".
It isn't - I've no real idea what the problem is, let alone what the
solution might be.  I've only one guess based upon what is sane in
the kernel, and that isn't even based on the data provided in the
bug report.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Linux 2.6.21

[Linus Torvalds]

On Sat, 28 Apr 2007, Adrian Bunk wrote:
> 
> We are already quite good at ignoring bug reports that come through 
> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> than 1600 open bugs because this tells how bad we are at handling bugs.

No, it just shows that bugzilla doesn't matter for most of the kernel.

Don't say that "bugzilla tells how bad we are at handling bugs". It tells 
how bad *bugzilla* is for handling bugs, nothing more.

Trying to play politics by pointing to bugzilla is pointless. Bugzilla is 
used for a few subsystems (ACPI seems to use it actively, for example), 
but I doubt most developers use it.

Would be be good to have a better bug-tracking setup? Yes. But I think it 
takes man-power, and it would take something *fundamentally* better than 
bugzilla.

Maybe the new "http://kernelnewbies.org/known_regressions" thing will 
evolve to something worth tracking. Right now, bugzilla isn't it (although 
it can be a useful tracking place for individual bugs, *once* you've found 
and gotten the right developer involved - but that's a huge step that 
bugzilla generally does *not* do for us).

			Linus

Re: Linux 2.6.21

[Markus Rechberger]

On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Sat, 28 Apr 2007, Adrian Bunk wrote:
> >
> > We are already quite good at ignoring bug reports that come through
> > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
> > than 1600 open bugs because this tells how bad we are at handling bugs.
>
> No, it just shows that bugzilla doesn't matter for most of the kernel.
>
> Don't say that "bugzilla tells how bad we are at handling bugs". It tells
> how bad *bugzilla* is for handling bugs, nothing more.
>

I totally disagree here, bugzilla is a very good tool. If someone is
too lazy to look at it it's his problem.
Kernel Janitors can pick out some bugs which aren't addressed by
anyone or got left behind. I also found some bugs there which could
have been solved by anyone here, the matter is just that many people
aren't interested in mainly bug fixing and many also work on different
other topics here.

How else should bugs get handled, sending them to the lkml?
I'm 100% sure some bugreports will also get lost then, but on the lkml
they'll very likely remain lost whereas in the bugzilla they'll remain
as open.

> Trying to play politics by pointing to bugzilla is pointless. Bugzilla is
> used for a few subsystems (ACPI seems to use it actively, for example),
> but I doubt most developers use it.
>

as for the em28xx I actively use it, but I also set up a mailinglist
etc. and there are many supporters already...

> Would be be good to have a better bug-tracking setup? Yes. But I think it
> takes man-power, and it would take something *fundamentally* better than
> bugzilla.
>

what are your suggestions to improve a bugreporting tool, I'm very
sure that many people, especially people who want to get into existing
projects here, would love to contribute.

> Maybe the new "http://kernelnewbies.org/known_regressions" thing will
> evolve to something worth tracking. Right now, bugzilla isn't it (although
> it can be a useful tracking place for individual bugs, *once* you've found
> and gotten the right developer involved - but that's a huge step that
> bugzilla generally does *not* do for us).
>

I'd say this is a personal opinion, some people will get along with it
and some of them will not...

Markus

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Markus Rechberger wrote:
> 
> I totally disagree here, bugzilla is a very good tool. If someone is
> too lazy to look at it it's his problem.

You must be doing things very differently from a lot of other people if 
you think that's the case.

> Kernel Janitors can pick out some bugs which aren't addressed by
> anyone or got left behind.

IF that happened, it would actually be great. That's what I'm arguing for. 
And it was basically what Adrian was doing!

> How else should bugs get handled, sending them to the lkml?

Actually, looking at Adrian's regression lists, yes. lkml worked better 
than bugzilla did. By at _least_ a factor of two.

> I'm 100% sure some bugreports will also get lost then, but on the lkml
> they'll very likely remain lost whereas in the bugzilla they'll remain
> as open.

What's the difference between bugzilla and lkml.org? Both have search 
buttons. Both archive the old stuff. Both can be pointed to.

> what are your suggestions to improve a bugreporting tool, I'm very
> sure that many people, especially people who want to get into existing
> projects here, would love to contribute.

I don't know what the perfect setup is, but I do know that bugzilla is 
very close to be totally useless for the top-level maintainers.

Try to think like a person who doesn't maintain *one* specific file in the 
kernel, but who can actually make a good judgement about a lot of things, 
or at least funnel a problem report to the right person?

And now, imagine that that person is also fairly busy (exactly *because* 
he's not looking at a single file, he may be maintaining a huge subsystem 
that has multiple submaintainers etc).

And ask yourself whether bugzilla really helps.

> I'd say this is a personal opinion, some people will get along with it
> and some of them will not...

I think bugzilla really only works for very "directed" issues. If you 
already know exactly which driver is affected (which is often wrong 
anyway: some of the bugs that were due timer breakage got blamed as disk 
hangs!) it's almost totally useless.

And yes, maybe that's why you have a much higher opinion of bugzilla than 
I do. To _me_ bugzilla is a total mess. There's absolutely _zero_ useful 
information there. And I'm pretty certain that is true of a *lot* of other 
people too. But if you have a small project, or you maintain a very 
specific (and clearly delineated) part of a big project, bugzilla probably 
looks a lot more palatable.

			Linus

Re: Linux 2.6.21

[Adrian Bunk]

On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
>...
> What's the difference between bugzilla and lkml.org? Both have search 
> buttons. Both archive the old stuff. Both can be pointed to.

Mailing lists don't track bugs.

The _only_ reason why I originally started regression lists was because 
several kernel developers were spreading the fairy tale "noone tests -rc 
kernels".

This was only possible because so many bug reports to linux-kernel never 
get any reply, and are therefore lost.

After I started the regression lists, it suddenly turned out how many 
people test -rc kernels, and that even with regression lists at least 
weekly, it still often took weeks ontil some developer even started 
debugging a regression.

So what's the difference between bugzilla and lkml.org?
In Bugzilla we are able to see how bad our bug handling is...

> 			Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Dave Jones]

On Sun, Apr 29, 2007 at 02:05:42AM +0200, Adrian Bunk wrote:
 > On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
 > >...
 > > What's the difference between bugzilla and lkml.org? Both have search 
 > > buttons. Both archive the old stuff. Both can be pointed to.
 > 
 > Mailing lists don't track bugs.

There's no reason they can't.  Store them in folders 'fixed' 'pending'
'notabug' etc. Move mails between them when states change.
reply-to them when necessary. Bounce them. Add people to Ccs. etc. etc.

The only remaining piece of the puzzle is "how does everyone see the
states of the various pools", which could be solved in a number of
ways, daily uploads to a place on kernel org for eg.

Hell, you could store them in a git tree if it came to it.
That would also solve the problem for those with an aversion
to web browsers to see bugs :-)

	Dave

-- 
http://www.codemonkey.org.uk

Re: Linux 2.6.21

[David Lang]

On Sun, 29 Apr 2007, Dave Jones wrote:

> On Sun, Apr 29, 2007 at 02:05:42AM +0200, Adrian Bunk wrote:
> > On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
> > >...
> > > What's the difference between bugzilla and lkml.org? Both have search
> > > buttons. Both archive the old stuff. Both can be pointed to.
> >
> > Mailing lists don't track bugs.
>
> There's no reason they can't.  Store them in folders 'fixed' 'pending'
> 'notabug' etc. Move mails between them when states change.
> reply-to them when necessary. Bounce them. Add people to Ccs. etc. etc.

and archive off the old reports into folders as well. While I don't think that 
closeing a report becouse it hasn't seen an update in a week is reasonable, 
closeing a report that hasn't seen an update or retest in a couple 2.6.x 
releases definantly is (this is a 4-6 month period at the rate of change in the 
kernel the odds are pretty good that the code is no longer the same at that 
point)

> The only remaining piece of the puzzle is "how does everyone see the
> states of the various pools", which could be solved in a number of
> ways, daily uploads to a place on kernel org for eg.

if people are interested in doing this it's not a technicly hard problem.

get bugs to a 'bug maintainer' e-mail address. this can either be a copy of the 
full lkml firehose, or volunteers who pull selected messages from the list, with 
a server that discards duplicates it's safe to have multiple people bounce the 
same message to the bug list, if they forward the message (to add a comment on 
who they think may need to work on it) then it will take manual weeding out of 
duplicates

have an IMAP server available for this address. make it read-only to the world 
and read-write to a list of volunteers who will sort the messages.

sort the messages into the different catagories, with a subfolder for each bug 
(note: the structure at this point is arbtrary, the volunteers can further 
orginise the folders)

with a good IMAP server you can add the address of the particular bug folder to 
the cc list if you want (bugs+drivers.ethernet.3com.123@kernel.org for a complex 
example) to have the follow-ups go directly into that folder.

now the problem with this is that developers would still have to look at it for 
an overview (the volunteers would still need to copy the developers when they 
create a new bug)

the cyrus mail server will scale well for this sort of thing, and I beleive that 
it can also export the mailboxes as news feeds as well (I haven't ever 
configured this so I can't say the exact details)

for searching, just make it available to google (I'm sure google would cooperate 
with this sort of thing to find the best way to do it)

the key is to find people to volunteer to do the sorting. the hosting of this is 
pretty simple (for that matter, I'll offer to host it if nobody else steps up)

David Lang

> Hell, you could store them in a git tree if it came to it.
> That would also solve the problem for those with an aversion
> to web browsers to see bugs :-)

git is not particulary good for searching the contents of things.

on that note, I wonder if google has a git:// search as part of theit git code 
project? if not they should.

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 05:27:56PM -0400, Dave Jones wrote:
> On Sun, Apr 29, 2007 at 02:05:42AM +0200, Adrian Bunk wrote:
>  > On Sat, Apr 28, 2007 at 04:40:31PM -0700, Linus Torvalds wrote:
>  > >...
>  > > What's the difference between bugzilla and lkml.org? Both have search 
>  > > buttons. Both archive the old stuff. Both can be pointed to.
>  > 
>  > Mailing lists don't track bugs.
> 
> There's no reason they can't.  Store them in folders 'fixed' 'pending'
> 'notabug' etc. Move mails between them when states change.
> reply-to them when necessary. Bounce them. Add people to Ccs. etc. etc.
> 
> The only remaining piece of the puzzle is "how does everyone see the
> states of the various pools", which could be solved in a number of
> ways, daily uploads to a place on kernel org for eg.
> 
> Hell, you could store them in a git tree if it came to it.
> That would also solve the problem for those with an aversion
> to web browsers to see bugs :-)

That would be an implementation of a git based bug tracker.  :-)

> 	Dave

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Bob Tracy]

Linus Torvalds wrote:
> On Sun, 29 Apr 2007, Markus Rechberger wrote:
> > How else should bugs get handled, sending them to the lkml?
> 
> Actually, looking at Adrian's regression lists, yes. lkml worked better 
> than bugzilla did. By at _least_ a factor of two.

Since 1992, lkml (with "Cc:" to the appropriate subsystem mailing list
if applicable) and the presumed responsible parties are the only channels
I've used to report the bugs I encounter.

Other methods come and go, but old habits die hard, particularly when
they have a knack for producing the desired result.  Historically,
requiring a developer to fire up a GUI to read a bug report decreases
the chance that bug report will be noticed.  The development community
can do whatever flips its collective switch as far as tracking bugs,
but the bugs have to be reported and noticed before tracking becomes a
meaningful activity.

One more thought and I'll get off your screens...  We've steadfastly
resisted making lkml and friends subscriber-only mailing lists precisely
because we don't want to miss a potential bug report because a would-be
submitter isn't subscribed.  If people aren't looking for bug reports
here, what's the point?

--Bob Tracy
rct@frus.com

Re: Linux 2.6.21

[Markus Rechberger]

On 4/29/07, Bob Tracy <rct@gherkin.frus.com> wrote:
> Linus Torvalds wrote:
> > On Sun, 29 Apr 2007, Markus Rechberger wrote:
> > > How else should bugs get handled, sending them to the lkml?
> >
> > Actually, looking at Adrian's regression lists, yes. lkml worked better
> > than bugzilla did. By at _least_ a factor of two.
>
> Since 1992, lkml (with "Cc:" to the appropriate subsystem mailing list
> if applicable) and the presumed responsible parties are the only channels
> I've used to report the bugs I encounter.
>

since there are subsystems out there which are managed separatly this
doesn't work out.
I wasn't happy when I noticed that patches got applied to the
sourcecode I contributed without notifying me while I still worked on
that code separatly
It was moreover the fault of the subsystem maintainer to not notify me
back then but a centralized bugreporting (as bugzilla) tool would at
least have notified me, or I would have been able to see the suggested
changes there.

But I agree with that if you're only 1 level far away from the linux kernel.

> Other methods come and go, but old habits die hard, particularly when
> they have a knack for producing the desired result.  Historically,
> requiring a developer to fire up a GUI to read a bug report decreases
> the chance that bug report will be noticed.  The development community
> can do whatever flips its collective switch as far as tracking bugs,
> but the bugs have to be reported and noticed before tracking becomes a
> meaningful activity.
>
> One more thought and I'll get off your screens...  We've steadfastly
> resisted making lkml and friends subscriber-only mailing lists precisely
> because we don't want to miss a potential bug report because a would-be
> submitter isn't subscribed.  If people aren't looking for bug reports
> here, what's the point?
>

it's just easy to miss something here, if an ext3 bug comes in and all
people who're involved in the ext3 filesytem are on vacation I'm sure
they won't read all the mails which came in during a week, now take a
part of the kernel which is smaller than the ext3 filesystem (eg. usb
gadgets, smaller drivers)

Markus

Re: Linux 2.6.21

[Markus Rechberger]

On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Sun, 29 Apr 2007, Markus Rechberger wrote:
> >
> > I totally disagree here, bugzilla is a very good tool. If someone is
> > too lazy to look at it it's his problem.
>
> You must be doing things very differently from a lot of other people if
> you think that's the case.
>

Well I'm behind the stuff I'm doing because I'm interested in it. And
if some bugs are introduced by my work or derived by my work I'd like
to get them cleaned up in the end.
If I see that someone reports bugs which doesn't really address my
work at all I just forward them to the subsystem/maintainer who's "in
charge" (if someone can say it that way for an open source project)

> > Kernel Janitors can pick out some bugs which aren't addressed by
> > anyone or got left behind.
>
> IF that happened, it would actually be great. That's what I'm arguing for.
> And it was basically what Adrian was doing!
>

I'm very sure that happens maybe it's just not visible to everyone
because there are so many open issues. (I just take myself as an
example here, I didn't do too much with other bugs but at least some
of my work closed 5 other bugs this year beside the bugreports I'm
getting directly)

> > How else should bugs get handled, sending them to the lkml?
>
> Actually, looking at Adrian's regression lists, yes. lkml worked better
> than bugzilla did. By at _least_ a factor of two.
>

Yes Adrian did a very good job with collecting every bugreport and
sending the mails to all corresponding subsystems.

> > I'm 100% sure some bugreports will also get lost then, but on the lkml
> > they'll very likely remain lost whereas in the bugzilla they'll remain
> > as open.
>
> What's the difference between bugzilla and lkml.org? Both have search
> buttons. Both archive the old stuff. Both can be pointed to.
>

Both have search buttons yes, but the lkml doesn't leave an unread
mail open ontop of the lkml as bugzilla does if you look for open bugs
in a subsystem.

> > what are your suggestions to improve a bugreporting tool, I'm very
> > sure that many people, especially people who want to get into existing
> > projects here, would love to contribute.
>
> I don't know what the perfect setup is, but I do know that bugzilla is
> very close to be totally useless for the top-level maintainers.
>
> Try to think like a person who doesn't maintain *one* specific file in the
> kernel, but who can actually make a good judgement about a lot of things,
> or at least funnel a problem report to the right person?
>
> And now, imagine that that person is also fairly busy (exactly *because*
> he's not looking at a single file, he may be maintaining a huge subsystem
> that has multiple submaintainers etc).
>
> And ask yourself whether bugzilla really helps.
>

bugzilla keeps the bugs open at least, at the lkml I use to skip days sometimes.
Many people who consider themself as maintainer of a subsystem are
assigned to a subsection on bugzilla, if it really doesn't work out we
have to change the corresponding maintainer.
If that maintainer doesn't know where to go with that bugreport he can
easily send it to the lkml and some people will recognize the
sender/email and pay extra attention to it (that's just how I think
about it)

> > I'd say this is a personal opinion, some people will get along with it
> > and some of them will not...
>
> I think bugzilla really only works for very "directed" issues. If you
> already know exactly which driver is affected (which is often wrong
> anyway: some of the bugs that were due timer breakage got blamed as disk
> hangs!) it's almost totally useless.
>
> And yes, maybe that's why you have a much higher opinion of bugzilla than
> I do. To _me_ bugzilla is a total mess. There's absolutely _zero_ useful
> information there. And I'm pretty certain that is true of a *lot* of other
> people too. But if you have a small project, or you maintain a very
> specific (and clearly delineated) part of a big project, bugzilla probably
> looks a lot more palatable.

well are there any bugs that cannot be forwarded/directed to a
corresponding maintainer?
Maybe I don't see something here, can you point me out to a bugreport
which cannot be handled at all?
As a reference I'll take following bugreport:
http://thread.gmane.org/gmane.linux.kernel/521185

the bug doesn't even mention what device is affected, asking for
further detailed information (dmesg) shows up what's left at least..
(in the meanwhile the bug even got solved)

Markus

Re: Linux 2.6.21

[David Miller]

From: "Markus Rechberger" <mrechberger@gmail.com>
Date: Sun, 29 Apr 2007 00:58:09 +0200

> On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >
> >
> > On Sat, 28 Apr 2007, Adrian Bunk wrote:
> > >
> > > We are already quite good at ignoring bug reports that come through
> > > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
> > > than 1600 open bugs because this tells how bad we are at handling bugs.
> >
> > No, it just shows that bugzilla doesn't matter for most of the kernel.
> >
> > Don't say that "bugzilla tells how bad we are at handling bugs". It tells
> > how bad *bugzilla* is for handling bugs, nothing more.
> >
> 
> I totally disagree here, bugzilla is a very good tool.

No, Bugzilla really does suck, and I personally refuse to use it when
I have a choice.  And guess what?  You better be concerned about that
because I maintain all of the networking code :-)

It puts the onus FAR too much on the developer and not enough on the
reporter and other minions.  We have a small resource of developers,
yet lots of users, bug reporters, and minions, so something that
doesn't take advantage of the larger resource we have is going to
not function efficiently at all.  Yet that is what bugzilla does.

It's made way too much work for me every time I'm come in contact with
it, it wastes my time instead of making good use of it.

As a developer I do not want to get pounded with emails containing
state changes and other bullshit that typically comes with being
assigned to or on the CC of a bugzilla entry.  I don't want to be
reminded that a bug hasn't been touched in weeks, if the reporter
doesn't care I don't care and I'll work on things that people do care
about.  It makes me delete all the bugzilla email, even the ones with
important information in them, because it's rediculious to have to
sift through all of that crap.

People only use bugzilla because it is well understood and nothing
better has reached critical mass yet.

Re: Linux 2.6.21

[David Lang]

On Sat, 28 Apr 2007, David Miller wrote:

> 
> From: "Markus Rechberger" <mrechberger@gmail.com>
> Date: Sun, 29 Apr 2007 00:58:09 +0200
>
>> On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>>>
>>>
>>> On Sat, 28 Apr 2007, Adrian Bunk wrote:
>>>>
>>>> We are already quite good at ignoring bug reports that come through
>>>> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
>>>> than 1600 open bugs because this tells how bad we are at handling bugs.
>>>
>>> No, it just shows that bugzilla doesn't matter for most of the kernel.
>>>
>>> Don't say that "bugzilla tells how bad we are at handling bugs". It tells
>>> how bad *bugzilla* is for handling bugs, nothing more.
>>>
>>
>> I totally disagree here, bugzilla is a very good tool.
>
> No, Bugzilla really does suck, and I personally refuse to use it when
> I have a choice.  And guess what?  You better be concerned about that
> because I maintain all of the networking code :-)
>
> It puts the onus FAR too much on the developer and not enough on the
> reporter and other minions.  We have a small resource of developers,
> yet lots of users, bug reporters, and minions, so something that
> doesn't take advantage of the larger resource we have is going to
> not function efficiently at all.  Yet that is what bugzilla does.

I'll say that as a user I hate having to deal with bugzilla.

there's nothing more frustrating then spending a good chunk of time trying to 
find a similar bug, then jumping through all the bugzilla hoops to file a report 
to eventually (days/weeks later) get a message 'closed becouse it's a duplicate 
report), then have to go and track down what it's a duplicate of, read through 
that bug report, only to find that it's not solved there either, and to top it 
off, the people working on that bug won't see my report or that I'm available to 
troubleshoot it.

from a user poit of view, e-mailing the kernel list (retrying a few days later 
of there is no response) tends to work _much_ better.

David Lang

Re: Linux 2.6.21

[Stefan Richter]

David Lang wrote:
> I'll say that as a user I hate having to deal with bugzilla.
> 
> there's nothing more frustrating then spending a good chunk of time
> trying to find a similar bug, then jumping through all the bugzilla
> hoops to file a report to eventually (days/weeks later) get a message
> 'closed becouse it's a duplicate report), then have to go and track down
> what it's a duplicate of, read through that bug report, only to find
> that it's not solved there either, and to top it off, the people working
> on that bug won't see my report or that I'm available to troubleshoot it.

Ideally, joining duplicate reports should be a low-cost, lossless operation.

That said, when bug B is marked as duplicate of bug A, people at bug A
at least get a link to bug B, aren't they?  If they are too lazy to read
the report B, they obviously are not very interested in A either.  Tough
luck.  Vice versa, people at bug B get notified that the matter is now
continued at bug A and can add their Cc there.  Of course that addition
is one of the very few things that could probably be automated.

Joining duplicate reports at a mailinglist involves responding to
multiple threads and send links into web archives of the list, which
happens to be redundant to and disparate from your local e-mail storage.
 I can't see how this aspect of bug-handling works easier on mailinglists.

> from a user poit of view, e-mailing the kernel list (retrying a few days
> later of there is no response) tends to work _much_ better.

What I from a maintainer's POV agree with is that a report to the
appropriate mailinglist is often easier to triage than a report at
bugzilla, because the reporter often needs initial help to properly
define the problem.  Bugzilla becomes useful after a report reached a
minimum level of quality (after minimum initial triage) and if the bug
can be clearly associated with a maintained subsystem of the kernel (as
e.g. Linus already pointed out in this thread).
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Stefan Richter]

I wrote:
> Joining duplicate reports at a mailinglist involves responding to
> multiple threads and send links into web archives of the list, which
> happens to be redundant to and disparate from your local e-mail storage.
> I can't see how this aspect of bug-handling works easier on mailinglists.

PS:  Of course what _does_ work better on mailinglists than on bugzilla
is to recognize duplicates as such in the first place, when the symptoms
seem only loosely related.  (I.e. seeing the big picture and recognize
patterns.)
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Willy Tarreau]

On Sun, Apr 29, 2007 at 12:58:09AM +0200, Markus Rechberger wrote:
> On 4/29/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >
> >
> >On Sat, 28 Apr 2007, Adrian Bunk wrote:
> >>
> >> We are already quite good at ignoring bug reports that come through
> >> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more
> >> than 1600 open bugs because this tells how bad we are at handling bugs.
> >
> >No, it just shows that bugzilla doesn't matter for most of the kernel.
> >
> >Don't say that "bugzilla tells how bad we are at handling bugs". It tells
> >how bad *bugzilla* is for handling bugs, nothing more.
> >
> 
> I totally disagree here, bugzilla is a very good tool. If someone is
> too lazy to look at it it's his problem.

I'm glad we finally found _the_ person using it !

More seriously, it's so much a complicated interface ! It's hard to
bring more people into a discussion, it's hard to comment on code or
suggested patches, etc... Mail is by far more adapted to the job !

See how many times people do public patch reviews here. You get one
comment every 5-10 lines. I have yet to see how this could be done
in bugzilla.

And maintainers have to _think_ about going there. Mail comes in
without deliberate action. This is especially important because
only your eye is involved in noticing bugs affecting areas where
you can help.

What _may_ be useful would be to send digests or batches of recently
open bugs to LKML. But not all of them. Maybe doing this once a week
in the same way we post patches lists for review. We could have a batch
of "[BUG 1/23] quick subject". At least more people will notice them
and will be able to comment on them. And given the number of people
reading LKML, the bugs should only be posted once. Because if a bug
posted to LKML in a noticeable manner does not get assigned in a week,
it will never be.

Personally, I got used to review Greg & Chris' announces of stable
patches to find if some of them could affect 2.4. It's far easier
to ask for precisions in a mail you weren't expecting than it is to
go somewhere search for something you don't know exists.

Willy

Re: Linux 2.6.21

[Stefan Richter]

Willy Tarreau wrote:
> On Sun, Apr 29, 2007 at 12:58:09AM +0200, Markus Rechberger wrote:
>> I totally disagree here, bugzilla is a very good tool. If someone is
>> too lazy to look at it it's his problem.
> 
> I'm glad we finally found _the_ person using it !
> 
> More seriously, it's so much a complicated interface ! It's hard to
> bring more people into a discussion, it's hard to comment on code or
> suggested patches, etc... Mail is by far more adapted to the job !

To continue on the sarcastic tangent:  This flaw of bugzilla is
irrelevant for subsystems where there are less than three or two persons
who steadily hunt bugs anyway.  At the field I work on, I wouldn't have
anybody else to bring in in the first place, except that I sometimes
suggest to reporters to subscribe to a bug ticket.
-- 
Stefan Richter
-=====-=-=== -=-- ===-=
http://arcgraph.de/sr/

Re: Linux 2.6.21

[Russell King]

On Sun, Apr 29, 2007 at 12:58:09AM +0200, Markus Rechberger wrote:
> I totally disagree here, bugzilla is a very good tool. If someone is
> too lazy to look at it it's his problem.

If you think so, try reading my email and responding constructively
on how the issues there can be resolved.

That email contains good examples where bugzilla fails, and bugs end
up sitting around for ages untouched.  And no, it's not because I'm
"lazy".

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: Linux 2.6.21

[Adrian Bunk]

On Sat, Apr 28, 2007 at 03:33:39PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 28 Apr 2007, Adrian Bunk wrote:
> > 
> > We are already quite good at ignoring bug reports that come through 
> > linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> > than 1600 open bugs because this tells how bad we are at handling bugs.
> 
> No, it just shows that bugzilla doesn't matter for most of the kernel.
> 
> Don't say that "bugzilla tells how bad we are at handling bugs". It tells 
> how bad *bugzilla* is for handling bugs, nothing more.
> 
> Trying to play politics by pointing to bugzilla is pointless. Bugzilla is 
> used for a few subsystems (ACPI seems to use it actively, for example), 
> but I doubt most developers use it.
> 
> Would be be good to have a better bug-tracking setup? Yes. But I think it 
> takes man-power, and it would take something *fundamentally* better than 
> bugzilla.

Bugzilla has an email interface.
Andrew forwards bugs from Bugzilla to developers.

There might be small room for improvements, but I don't see how 
man-power or technology could make a big difference in this area.

> Maybe the new "http://kernelnewbies.org/known_regressions" thing will 
> evolve to something worth tracking. Right now, bugzilla isn't it (although 
> it can be a useful tracking place for individual bugs, *once* you've found 
> and gotten the right developer involved - but that's a huge step that 
> bugzilla generally does *not* do for us).

"*once* you've found and gotten the right developer involved" is the 
real problem, not how to track bugs.

And not only a developer active in this area, more important a 
developer who knows the subsystem/driver involved *and is willing
to work on bug reports*.

*This* is *the* problem.
And no change in bug tracking will help with this problem.

> 			Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Linus Torvalds]

On Sun, 29 Apr 2007, Adrian Bunk wrote:
> 
> Bugzilla has an email interface.
> Andrew forwards bugs from Bugzilla to developers.

Yes. And if you search around, you'll even see that I occasionally use it. 

But it's useful only once the bug has been assigned and somebody has 
actually *looked* at it. The fact that some people do this is a big credit 
(Andrew, Dave J etc)

> There might be small room for improvements, but I don't see how 
> man-power or technology could make a big difference in this area.

I'm talking about getting the developers to _look_ at the bugs in the 
first place. Bugzilla is not very good at that, because it has no useful 
interfaces for doing so (unless you can specify your area of interest so 
exactly that you can actually set yourself up as a maintainer of one 
particular area).

Almost none of the subtle (and thus harder) bugs tend to fall into that 
kind of nice category. 

For example, look at suspend/resume bugs. Do you realize that 99% of them 
are device driver issues, but how the heck do you connect a "my laptop 
does't resume" to the _right_ device driver maintainer?

And do you realize (and acknowledge) that it would be total madness to 
send all suspend/resume bugs to _everybody_ who maintains any device 
driver at all?

See? THAT is the problem with bugzilla. It only works for the "easy" 
cases. It works for the case where a reporter can say with certainty that 
the reason his machine doesn't boot is a particular network device driver 
(like the sis900 regression we had in 2.6.21). But once you know the 
subarea that precisely, bugzilla doesn't even help you that much, and it's 
likely easier and more productive to just send email directly to the right 
person and cc Jeff and netdev or something!

> "*once* you've found and gotten the right developer involved" is the 
> real problem, not how to track bugs.

And I agree 100%.

And bugzilla does *nothing* here.

> *This* is *the* problem.
> And no change in bug tracking will help with this problem.

So why are you pushing bugzilla? 

There are actually better bug trackers around. One of them is "google". 
For oopses, one of the thngs I do is to put in the most relevant 
information (backtrace etc) into google, and ask google to try to find the 
pattern. That sometimes actually does pretty well - you can get a real 
feel for "oh, there's a pattern here - they're all AMD machines with the 
NVidia chipset" kind of thing.

Bugzilla doesn't offer anything even remotely as useful. 

It's the "big picture" that tends to be hard to find.

And that's what *you* gave with the regression report. A summary. You even 
noted some of the patterns, so people actually had them already somewhat 
pre-sorted.

THAT is what we want.

But another way to get the "high-level picture" is to actually just say 
"what's active now". And that's why I think the whole "1600 open 
bug-reports" kind of thing is useless. Bugzilla does *not* have mail 
interfaces for that kind of "these 50 bugs were actively reported on and 
unresolved" in the last week summaries! 

(And if it did, it would still almost certainly need some human care and 
understanding)

		Linus

Re: Linux 2.6.21

[David Miller]

From: Adrian Bunk <bunk@stusta.de>
Date: Sun, 29 Apr 2007 01:04:16 +0200

> Bugzilla has an email interface.
> Andrew forwards bugs from Bugzilla to developers.

Therefore, bugzilla only works at all when Andrew forwards things
around by-hand.

Re: Linux 2.6.21

[Thomas Gleixner]

On Sat, 2007-04-28 at 20:41 -0700, David Miller wrote:
> From: Adrian Bunk <bunk@stusta.de>
> Date: Sun, 29 Apr 2007 01:04:16 +0200
> 
> > Bugzilla has an email interface.
> > Andrew forwards bugs from Bugzilla to developers.
> 
> Therefore, bugzilla only works at all when Andrew forwards things
> around by-hand.

That's not entirely true. There are people watching the bugs which might
be relevant for them on their own.

It does not make bugzilla better though. The user interface sucks and
getting things correlated is simply not possible.

	tglx

Re: Linux 2.6.21

[Borislav Petkov]

On Sat, Apr 28, 2007 at 09:53:20PM +0200, Adrian Bunk wrote:
> On Thu, Apr 26, 2007 at 02:13:08PM -0700, Linus Torvalds wrote:
> >...
> > (I've said this before, but I'll say it again: one thing that would 
> > already make bugzilla better is to just always drop any bug reports that 
> > are more than a week old and haven't been touched. It wouldn't need *much* 
> > touching, but if a reporter cannot be bothered to say "still true with 
> > current snapshot" once a week, then it shouldn't be seen as being somehow 
> > up to those scare resources we call "developers" to have to go through 
> > it).
> >...
> 
> And if it's a bug in an unmaintained subsystem, a user could do this for 
> 100 weeks without any effect.
> There's no value in keeping reporters busy with useless tasks. [1]
> 
> Don't forget:
> A good bug report is an important contribution.
> 
> We are already quite good at ignoring bug reports that come through 
> linux-kernel, and it's an _advantage_ of the kernel Bugzilla to see more 
> than 1600 open bugs because this tells how bad we are at handling bugs.
> How many thousand bug reports have been ignored during the same time on 
> linux-kernel?

... and it gets better: I've been trying to fix/improve stuff I can, with
my limited abilities as coder for some time now in the linux kernel, but there
are simply cases where, although you're trying to even come up with a proper fix
and adhere to all the kernel coding standards and _even_ produce a patch which
can serve as rough cut for a probable fix, your mail simply disappears into the
void unanswered. Then you send again, and again, yet it remains unreplied and then you
give up and turn to something else. Some of those patches, for example, were the
rework of the debugging scheme of libata i did more than a year ago which got
reviewed and then .. forgotten, some kernel janitor cleanups, etc...

-- 
Regards/Gruß,
    Boris.

Re: Linux 2.6.21

[Bill Davidsen]

Linus Torvalds wrote:

> In other words, there's a _reason_ we have staggered development. We have 
> the "crazy development trees" (aka -mm and various other trees), we have 
> the "development tree" (aka Linus' tree), and we have the -stable tree. If 
> the stable tree has a dozen known issues that they'll have to sort out 
> over the next two months, that's *fine*. That's kind of the point of the 
> stable tree.
> 
If the result is fixing things which then don't get fixed in mainline, 
as Adrian notes, then there is something wrong with the process, and why 
will people bother to work on stable if they have doubts that there will 
be long term benefit.

With all the effort the regressions list takes and the stable group puts 
into fixes, someone in charge should insist that regressions fixed in 
stable be fixed in mainline. Since there's only one "someone in charge" 
of policy, I think that's a reasonable commitment to the people doing 
the work.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Bill Davidsen wrote:
>
> If the result is fixing things which then don't get fixed in mainline, as
> Adrian notes

That whole premise is flawed. The *rule* for the stable tree is that 
things don't get merged into the stable tree unless they are fixed in 
mainline already. 

We had that problem in the 2.4.x / 2.5.x split. I think we learnt our 
lesson.

			Linus

Re: Linux 2.6.21

[Bill Davidsen]

Linus Torvalds wrote:
> 
> On Thu, 26 Apr 2007, Bill Davidsen wrote:
>> If the result is fixing things which then don't get fixed in mainline, as
>> Adrian notes
> 
> That whole premise is flawed. The *rule* for the stable tree is that 
> things don't get merged into the stable tree unless they are fixed in 
> mainline already. 
> 
If Adrian cares to note which two regressions he had in mind in his 
previous post <20070426125802.GL3468@stusta.de> or what the exact timing 
was I'll let him continue this.

> We had that problem in the 2.4.x / 2.5.x split. I think we learnt our 
> lesson.
>
I'd love to think that's the case.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Thomas Gleixner]

Adrian,

On Thu, 2007-04-26 at 14:58 +0200, Adrian Bunk wrote:
> I am aware that my work had some effect, and I am aware that my work 
> gets appreciated - there's no need for everyone to repeat this.

Nevertheless, thanks for your efforts and time spent. You did a great
job and I hope you can convince yourself to carry on.

> The point is: I'm not satisfied with the result.

Nobody is satisfied with pending regressions. I can completely
understand your frustration, but you need to adjust your expectations on
that as well.

Your regression lists are extremly useful, as they point folks like me
to the burning points. I try to follow LKML as far as I can, but I have
to admit that I occasionally go the easy way of marking 10000 mails as
read in one go after a week of travelling. I don't do this to my
personal inbox, so your mails get my attention.

> They have both been fixed through -stable, but I also remember a quite 
> experienced kernel maintainer running into one of them after 2.6.20 was 
> released and spending half a day tracking it down - and my answer was
> "known unfixed regression, first reported more than a month ago".

That happens all the time. I have a dozen of boxen around and I can't do
tests on all of them continously. So trapping into some known regression
is nothing which surprises me.

> There is a conflict between Linus trying to release kernels every
> 2 months and releasing with few regressions.

Yes, it's a conflict, but one that is unresolvable except we want to go
back to the 2.4 model which sucked way more than the current one.

> And a serious delay of the next regression-merge window due to unfixed 
> regressions might even have the positive side effect of more developers 
> becoming interested in fixing the current regressions for getting their 
> shiny new regressions^Wfeatures faster into Linus' tree.

Maybe we need to coordinate changes better. 2.6.21 got three big updates
which affected suspend/resume - one of them is my fault. But fiddling
out which one of those - we had nested problems as well - makes it quite
hard to grok them in time, especially if they happen only on one
reporters system.

Your reports are not invalid, when Linus releases a final. They are
still there and worked on.

I believe we are getting better at that, and one reason for this is your
relentless effort to poke the experts^culprits to actually solve the
problems.

> I'm not satisfied with the result, and the world won't stop turning when 
> I'm not tracking 2.6.22-rc regressions.

Please take a couple of days to reconsider. I personally would welcome
if you carry on.

Thanks,

	tglx

Re: Linux 2.6.21

[Linus Torvalds]

On Fri, 27 Apr 2007, Thomas Gleixner wrote:
> 
> Maybe we need to coordinate changes better. 2.6.21 got three big updates
> which affected suspend/resume - one of them is my fault. But fiddling
> out which one of those - we had nested problems as well - makes it quite
> hard to grok them in time, especially if they happen only on one
> reporters system.

Yes. _If_ we had known how painful the timer changes would end up being, 
we'd probably have done them separately from everything else.

That is the kind of thing that looks obvious in hindsight: merge stuff 
that is questionable and scary alone, and don't do anything else that 
release cycle.

But while the timer code is obviously pretty core, I think everybody 
expected it to be a lot easier to merge (and it had existed as patches in 
various forms for some time).

So we simply didn't know beforehand that it was going to cause the kinds 
of regressions it did cause (and in fact, some of the regressions were 
initially blamed on other things entirely - some of them looked like IO 
regressions).

Water under the bridge. It's also easy to say in hindsight that something 
should have been merged separately and been given a release cycle all its 
own.

			Linus

Re: Linux 2.6.21

[Daniel Barkalow]

On Thu, 26 Apr 2007, Adrian Bunk wrote:

> Linus said 2.6.20 was a stable kernel. My impression was that at least 
> two of the regressions from my 2.6.20 regressions list should have been 
> fixed before 2.6.20.
> 
> They have both been fixed through -stable, but I also remember a quite 
> experienced kernel maintainer running into one of them after 2.6.20 was 
> released and spending half a day tracking it down - and my answer was
> "known unfixed regression, first reported more than a month ago".

I think there is an issue with two different things being conflated, and 
this causes real stability problems. 2.6.x is both the first kernel in a 
series that is judged to be "stable" and the kernel that is the split 
between 2.6.x.y and 2.6.x+1. This is a fundamental problem, because it 
means that 2.6.x must have all of the problems that are being debugged by 
the people who understand the areas they are in, because 2.6.x+1 has to 
start so that people who are clueless about all of the areas with 
remaining bugs don't spend their time putting more regressions into their 
submissions for 2.6.x+1.

It is also a problem because it is easily possible for a problem to exist 
in 2.6.x-rcN which can only be correctly fixed by doing intrusive things, 
but can be papered over in an obviously-safe way. (E.g., the issue with 
legacy interrupt delivery when MSI is enabled). The intrusive patch could 
easily break a bunch of unrelated stuff, so that's no good for 2.6.x-rcN, 
but papering over bugs is no good for mainline. These bugs have to be 
fixed after the split, which means that the version at the fork must 
contain the bug.

Furthermore, everybody (people reporting bugs, people fixing them, and 
people merging fixes) seem to doze off late in -rc kernels. Having an 
announcement of something with a qualitatively different version wakes 
them up.

I say have a target of no known regressions in 2.6.21.1, with 2.6.21 being 
pretty good, and don't count too much on the stability of 3-number kernel 
versions.

> And a serious delay of the next regression-merge window due to unfixed 
> regressions might even have the positive side effect of more developers 
> becoming interested in fixing the current regressions for getting their 
> shiny new regressions^Wfeatures faster into Linus' tree.

I think the "stick" can't be delaying the window, because that's too 
broad. I think it has to be making people who are needed for fixing stuff 
miss the window. People aren't going to go learn a new area of the kernel 
to resolve regressions in it, but they're more likely to keep their own 
area clean so that they get to merge every 2 months instead of every 4.

> These are just my personal opinions, and other people consider the 
> resulting 2.6.20 and 2.6.21 kernels OK.

I don't think 2.6.x can be OK, by policy. I think 2.6.20.y got to an OK 
state eventually, which is to say that there's no need now to use a 
2.6.19.y kernel. I think that 2.6.21 isn't OK yet, but I think looking for 
an OK 2.6.21-derived kernel is premature still. Ignoring the version 
scheme entirely, I think the success condition should be that the "latest 
stable version of the Linux kernel" link on www.kernel.org is 
always strictly better than all previous links in that spot, and new 
features get there eventually (ideally, within 4 months of hitting 
mainline). I think this is actually possible, although it would require 
changing the policy for this link. And I don't think it requires a change 
in what goes into Linus's git repository when.

Furthermore, I think we're a lot closer to an OK kernel derived from 
Linus's Apr 25 version than we would be if "2.6.21" had not been released 
at that point. It sounds like more items were resolved in the past few 
days than in the preceding week.

Incidentally, will you continue to track 2.6.21 regressions against 
2.6.20? You said there was at least one that you haven't sent out, and 
there's been movement on several others since your last report.

	-Daniel
*This .sig left intentionally blank*

Re: Linux 2.6.21

[Bill Davidsen]

Adrian Bunk wrote:
> On Wed, Apr 25, 2007 at 08:29:28PM -0700, Linus Torvalds wrote:
>> ...
>> So it's been over two and a half months, and while it's certainly not the 
>> longest release cycle ever, it still dragged out a bit longer than I'd 
>> have hoped for and it should have. As usual, I'd like to thank Adrian (and 
>> the people who jumped on the entries Adrian had) for keeping everybody on 
>> their toes with the regression list - there's a few entries there still, 
>> but it got to the point where we didn't even know if they were real 
>> regressions, and delaying things further just wasn't going to help.
>> ...
> 
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release:
> 14
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release that were first reported in March or earlier:
> 8
> 
> Number of different known regressions compared to 2.6.20 at the time
> of the 2.6.21 release with patches available at the time of the 2.6.21 
> release [1]:
> 3
> 
> What I will NOT do:
> Waste my time with tracking 2.6.22-rc regressions.
> 
> 
> We have an astonishing amount of -rc testers, but obviously not the 
> developer manpower for handling them.
> 
> If we would take "no regressions" seriously, it might take 4 or 5 months 
> between releases due to the lack of developer manpower for handling 
> regressions. But that should be considered OK if avoiding regressions 
> was considered more important than getting as quick as possible to the 
> next two week regression-merge window.
> 
> But releasing with so many known regressions is insulting for the many 
> people who spent their time testing -rc kernels.
> 
Without someone holding Linus feet to the fire the next release may be a 
real POS. I think you have done the perfect job, identifying the show 
stoppers, quantifying the obscure and minor regressions, and serving to 
give testing targets as purported fixes are applied.

I don't think you should judge your work by leaving some targets for 
-stable and 2.6.22, but rather from the number of problems you detected, 
documented, and caused to be addressed.

If it were my week to be God, I would insist that the rcN to final step 
was regressions-only, and that all regressions be classified as (a) 
acceptable results of changes to fix other problems, (b) must be fixed 
before release, or (c) obscure enough to tolerate for a short time, must 
be fixed in stable and mainline before N+1 release.

Measuring releases or your own value against perfection is thankless!

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Jeff Garzik]

IMO, the closer you look, the more warts you find.  Before you starting 
doing your work with kernel regressions, no one was really tracking it. 
  I bet you have helped cut down on the regressions, but I have no good 
way to quantify my gut feeling.

Additional comments on developers and fixing regressions:

* Sometimes seeing a long list, peoples' eyes glaze over.  Its just 
human nature.  A long list also gives us no idea of scale, or severity. 
  I bet a weekly "top 10 bugs and regressions" email would help focus 
developer attention.

* To be effective, lists, either long or top-10, must be pruned if you 
get a sense that only one user is affected.  [With oopses and BUGs as a 
clear exception,] many problems benefit from at least two users 
reporting a bug.

* It gets a bit tiresome to field the large number of driver bug reports 
that eventually turn out to be related to broken interrupt handling 
somehow.  I think we developers need to get better at showing users how 
to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start 
introducing interrupt delivery tests into their probe code.  Overall, 
broken interrupt handling manifests in several ways, most of which 
initially appear symptomatic of a broken driver.

	Jeff

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 02:04:56PM -0400, Jeff Garzik wrote:
> IMO, the closer you look, the more warts you find.  Before you starting 
> doing your work with kernel regressions, no one was really tracking it.  I 
> bet you have helped cut down on the regressions, but I have no good way to 
> quantify my gut feeling.
>
> Additional comments on developers and fixing regressions:
>
> * Sometimes seeing a long list, peoples' eyes glaze over.  Its just human 
> nature.  A long list also gives us no idea of scale, or severity.  I bet a 
> weekly "top 10 bugs and regressions" email would help focus developer 
> attention.

Top 10 ordered by what?

I always tried to put the most mysterious ones like "kwin dies silently" 
or "gammu no longer works" at the top of my lists hoping some developer 
might become interested in getting clues about them.

But when there were 15 outstanding suspend regressions, these were also 
important.

And how to order the rtl8139 netdriver regression reported one month ago 
against the snd_hda_intel regression reported one month ago?

Both are "only" drivers no longer working for some users, but there 
might be many more users for whom they don't work in 2.6.21 because the 
maintainers didn't bother to debug and fix them.

Ideally, maintainers should debug and fix regressions (and other bugs) 
without requiring weekly regression lists.

If bug reports and weekly reminders aren't enough, I don't think 
increasing the level of babysitting would make sense.

> * To be effective, lists, either long or top-10, must be pruned if you get 
> a sense that only one user is affected.  [With oopses and BUGs as a clear 
> exception,] many problems benefit from at least two users reporting a bug.

First of all we had several cases where one report by one user resulted 
in a serious bug being fixed. There might be only one -rc tester with a 
strange workload or some unusual hardware.

A bigger point is that "at least two users reporting a bug" assumes you 
always know whether two bug reports are for the same bug.
Some examples for problems:
- during early 2.6.21-rc, it was not unusual for users to run into
  three unrelated suspend related regressions on one computer
- during 2.6.20-rc, we had 4 similar looking reports, 3 for one
  regression, and the forth for a completely different regression
- during 2.6.21-rc, it turned out the following reports were caused
  by the same regression:
    kwin dies silently
    KDE problem if a sound is played while the display is suspended

> * It gets a bit tiresome to field the large number of driver bug reports 
> that eventually turn out to be related to broken interrupt handling 
> somehow.  I think we developers need to get better at showing users how to 
> isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start 
> introducing interrupt delivery tests into their probe code.  Overall, 
> broken interrupt handling manifests in several ways, most of which 
> initially appear symptomatic of a broken driver.

Regressions have the advantage of being able to compare with a 
known-good kernel.

If a driver works with 2.6.20 but doesn't work with 2.6.21-rc, ask the 
submitter for dmesg's from both and diff them. In my experience that 
usually gives a good indication whether or not it's an interrupt related 
ACPI/PCI issue.

> 	Jeff

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Francois Romieu]

Adrian Bunk <bunk@stusta.de> :
[...]
> And how to order the rtl8139 netdriver regression reported one month ago 
> against the snd_hda_intel regression reported one month ago?

Pointer for the rtl8139 regression please ?

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: Linux 2.6.21

[Jeff Garzik]

Francois Romieu wrote:
> Pointer for the rtl8139 regression please ?

I'm guessing it's this one:

> 	Subject    : boot failure: rtl8139: exception in interrupt routine
> 	References : http://lkml.org/lkml/2007/3/31/160
> 	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
> 	Status     : unknown


The poster says rtl8139, but doesn't provide more info.  His lspci says 
"RTL8169SC", which sounds more like r8169 to me.

	Jeff

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 03:13:15PM -0400, Jeff Garzik wrote:
> Francois Romieu wrote:
>> Pointer for the rtl8139 regression please ?
>
> I'm guessing it's this one:
>
>> 	Subject    : boot failure: rtl8139: exception in interrupt routine
>> 	References : http://lkml.org/lkml/2007/3/31/160
>> 	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
>> 	Status     : unknown
>
>
> The poster says rtl8139, but doesn't provide more info.  His lspci says 
> "RTL8169SC", which sounds more like r8169 to me.

Interesting.

I didn't notice it, Andrew didn't notice it, and it seems that although 
it was in three posted versions of my regression lists, noone looked at 
this bug (or it would have been noticed earlier).

> 	Jeff

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Stephen Clark]

Jeff Garzik wrote:

>Francois Romieu wrote:
>  
>
>>Pointer for the rtl8139 regression please ?
>>    
>>
>
>I'm guessing it's this one:
>
>  
>
>>	Subject    : boot failure: rtl8139: exception in interrupt routine
>>	References : http://lkml.org/lkml/2007/3/31/160
>>	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
>>	Status     : unknown
>>    
>>
>
>
>The poster says rtl8139, but doesn't provide more info.  His lspci says 
>"RTL8169SC", which sounds more like r8169 to me.
>
>	Jeff
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>
Yeah, that was my bad it is a RTL8169SC, and the 
problem was intermittent
sometimes is cause a panic othertimes it didn't.

It is laptop that does not have a serial port and 
I could not couldn't
get  the
kernel to boot using a usb serial port so I 
couldn't get a screen capture of
the intermittant panic.

Steve

-- 

"They that give up essential liberty to obtain 
temporary safety,
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government 
grows, liberty
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Francois Romieu]

Jeff Garzik <jeff@garzik.org> :
> Francois Romieu wrote:
> >Pointer for the rtl8139 regression please ?
> 
> I'm guessing it's this one:
> 
> >	Subject    : boot failure: rtl8139: exception in interrupt routine
> >	References : http://lkml.org/lkml/2007/3/31/160
> >	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
> >	Status     : unknown
> 
> 
> The poster says rtl8139, but doesn't provide more info.  His lspci says 
> "RTL8169SC", which sounds more like r8169 to me.

Yes, thanks.

The r8169 driver has been added a few bugfixes between 2.6.21-rc5 and now.
Some are related to latent, timing changes induced bugs.

Stephen, would you mind testing 2.6.21 and open a PR at bugzilla.kernel.org
if the bug does not go away ?

Bugzilla e-mails end directly in my mailbox. l-k traffic can be temporarily
unnoticed (especially on saturday night).

-- 
Ueimor

Anybody got a battery for my Ultra 10 ?

Re: Linux 2.6.21

[Stephen Clark]

Francois Romieu wrote:

>Jeff Garzik <jeff@garzik.org> :
>  
>
>>Francois Romieu wrote:
>>    
>>
>>>Pointer for the rtl8139 regression please ?
>>>      
>>>
>>I'm guessing it's this one:
>>
>>    
>>
>>>	Subject    : boot failure: rtl8139: exception in interrupt routine
>>>	References : http://lkml.org/lkml/2007/3/31/160
>>>	Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
>>>	Status     : unknown
>>>      
>>>
>>The poster says rtl8139, but doesn't provide more info.  His lspci says 
>>"RTL8169SC", which sounds more like r8169 to me.
>>    
>>
>
>Yes, thanks.
>
>The r8169 driver has been added a few bugfixes between 2.6.21-rc5 and now.
>Some are related to latent, timing changes induced bugs.
>
>Stephen, would you mind testing 2.6.21 and open a PR at bugzilla.kernel.org
>if the bug does not go away ?
>
>Bugzilla e-mails end directly in my mailbox. l-k traffic can be temporarily
>unnoticed (especially on saturday night).
>
>  
>
Sure I'll give it a try in the next couple of days.

Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Stephen Clark]

Jeff Garzik wrote:

>Stephen Clark wrote:
>  
>
>>It is laptop that does not have a serial port and I could not couldn't 
>>get  the
>>kernel to boot using a usb serial port so I couldn't get a screen 
>>capture of
>>the intermittant panic.
>>    
>>
>
>
>If you can write it down with a pen and paper, or manually copy the 
>panic onto another computer (w/ hand and eyes), that would be helpful.
>
>	Jeff
>
>
>
>  
>
Hi Jeff,

It was with FC7 livecd 6.92 - thought I would just try 6.93 which I see 
has appeared at the mirrors.
If it still happens I will copy the exception and send it in.

Steve


-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Stephen Clark]

Stephen Clark wrote:

>Jeff Garzik wrote:
>
>  
>
>>Stephen Clark wrote:
>> 
>>
>>    
>>
>>>It is laptop that does not have a serial port and I could not couldn't 
>>>get  the
>>>kernel to boot using a usb serial port so I couldn't get a screen 
>>>capture of
>>>the intermittant panic.
>>>   
>>>
>>>      
>>>
>>If you can write it down with a pen and paper, or manually copy the 
>>panic onto another computer (w/ hand and eyes), that would be helpful.
>>
>>	Jeff
>>
>>
>>
>> 
>>
>>    
>>
>Hi Jeff,
>
>It was with FC7 livecd 6.92 - thought I would just try 6.93 which I see 
>has appeared at the mirrors.
>If it still happens I will copy the exception and send it in.
>
>Steve
>
>
>  
>
I downloaded and compile 2.6.21 and so far it seem to be running fine on 
my asus based
S96F whitebook laptop - no panic from the rtl8169

Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Adrian Bunk]

On Thu, Apr 26, 2007 at 08:58:51PM +0200, Francois Romieu wrote:
> Adrian Bunk <bunk@stusta.de> :
> [...]
> > And how to order the rtl8139 netdriver regression reported one month ago 
> > against the snd_hda_intel regression reported one month ago?
> 
> Pointer for the rtl8139 regression please ?

Subject    : boot failure: rtl8139: exception in interrupt routine
References : http://lkml.org/lkml/2007/3/31/160
Submitter  : Stephen Clark <Stephen.Clark@seclark.us>
Status     : unknown


cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Stephen Clark]

Jeff Garzik wrote:

>IMO, the closer you look, the more warts you find.  Before you starting 
>doing your work with kernel regressions, no one was really tracking it. 
>  I bet you have helped cut down on the regressions, but I have no good 
>way to quantify my gut feeling.
>
>Additional comments on developers and fixing regressions:
>
>* Sometimes seeing a long list, peoples' eyes glaze over.  Its just 
>human nature.  A long list also gives us no idea of scale, or severity. 
>  I bet a weekly "top 10 bugs and regressions" email would help focus 
>developer attention.
>
>* To be effective, lists, either long or top-10, must be pruned if you 
>get a sense that only one user is affected.  [With oopses and BUGs as a 
>clear exception,] many problems benefit from at least two users 
>reporting a bug.
>
>* It gets a bit tiresome to field the large number of driver bug reports 
>that eventually turn out to be related to broken interrupt handling 
>somehow.  I think we developers need to get better at showing users how 
>to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start 
>introducing interrupt delivery tests into their probe code.  Overall, 
>broken interrupt handling manifests in several ways, most of which 
>initially appear symptomatic of a broken driver.
>
>	Jeff
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>
Jeff,

If hardware worked in the previous version of the kernel can't users expect
 the same hardware to work in this kernel?

Steve

-- 

"They that give up essential liberty to obtain temporary safety, 
deserve neither liberty nor safety."  (Ben Franklin)

"The course of history shows that as a government grows, liberty 
decreases."  (Thomas Jefferson)

Re: Linux 2.6.21

[Jeff Garzik]

Stephen Clark wrote:
> If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?

In general, yes.

	Jeff

Re: Linux 2.6.21

[Gene Heskett]

On Thursday 26 April 2007, Stephen Clark wrote:
>Jeff Garzik wrote:
>>IMO, the closer you look, the more warts you find.  Before you starting
>>doing your work with kernel regressions, no one was really tracking it.
>>  I bet you have helped cut down on the regressions, but I have no good
>>way to quantify my gut feeling.
>>
>>Additional comments on developers and fixing regressions:
>>
>>* Sometimes seeing a long list, peoples' eyes glaze over.  Its just
>>human nature.  A long list also gives us no idea of scale, or severity.
>>  I bet a weekly "top 10 bugs and regressions" email would help focus
>>developer attention.
>>
>>* To be effective, lists, either long or top-10, must be pruned if you
>>get a sense that only one user is affected.  [With oopses and BUGs as a
>>clear exception,] many problems benefit from at least two users
>>reporting a bug.
>>
>>* It gets a bit tiresome to field the large number of driver bug reports
>>that eventually turn out to be related to broken interrupt handling
>>somehow.  I think we developers need to get better at showing users how
>>to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start
>>introducing interrupt delivery tests into their probe code.  Overall,
>>broken interrupt handling manifests in several ways, most of which
>>initially appear symptomatic of a broken driver.
>>
>>	Jeff
>>
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>
>Jeff,
>
>If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?
>
>Steve

I think that is indeed a reasonable expectation.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
It is exactly because a man cannot do a thing that he is a proper judge of it.
		-- Oscar Wilde

Re: Linux 2.6.21

[Gene Heskett]

On Thursday 26 April 2007, Stephen Clark wrote:
>Jeff Garzik wrote:
>>IMO, the closer you look, the more warts you find.  Before you starting
>>doing your work with kernel regressions, no one was really tracking it.
>>  I bet you have helped cut down on the regressions, but I have no good
>>way to quantify my gut feeling.
>>
>>Additional comments on developers and fixing regressions:
>>
>>* Sometimes seeing a long list, peoples' eyes glaze over.  Its just
>>human nature.  A long list also gives us no idea of scale, or severity.
>>  I bet a weekly "top 10 bugs and regressions" email would help focus
>>developer attention.
>>
>>* To be effective, lists, either long or top-10, must be pruned if you
>>get a sense that only one user is affected.  [With oopses and BUGs as a
>>clear exception,] many problems benefit from at least two users
>>reporting a bug.
>>
>>* It gets a bit tiresome to field the large number of driver bug reports
>>that eventually turn out to be related to broken interrupt handling
>>somehow.  I think we developers need to get better at showing users how
>>to isolate driver vs. PCI/ACPI/core bugs.  Maybe drivers need to start
>>introducing interrupt delivery tests into their probe code.  Overall,
>>broken interrupt handling manifests in several ways, most of which
>>initially appear symptomatic of a broken driver.
>>
>>	Jeff
>>
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to majordomo@vger.kernel.org
>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at  http://www.tux.org/lkml/
>
>Jeff,
>
>If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?
>
>Steve

I think that is indeed a reasonable expectation.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
It is exactly because a man cannot do a thing that he is a proper judge of it.
		-- Oscar Wilde

Re: Linux 2.6.21

[Bill Davidsen]

Stephen Clark wrote:

> If hardware worked in the previous version of the kernel can't users expect
> the same hardware to work in this kernel?
> 
Failure of that assumption is the heart of the whole "regression" 
discussion. It's not limited to hardware, kernel security might be an 
issue, some network protocols might work faster and less reliably, etc.

Kernel behavior changes sometimes totally break user software which 
makes unwarranted assumptions. That's not a regression, although users 
may see it that way. When a change in fork() changed the 
child-runs-first behavior, many programs broke, as was true with 
threading changes. Bad reliability is the reward for bad code, but if a 
kernel change makes that obvious some people think it's a regression.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Linux 2.6.21

[Jan De Luyck]

On Thursday 26 April 2007, Linus Torvalds wrote:
> If the goal for 2.6.20 was to be a stable release (and it was), the goal
> for 2.6.21 is to have just survived the big timer-related changes and some
> of the other surprises (just as an example: we were apparently unlucky
> enough to hit what looks like a previously unknown hardware errata in one
> of the ethernet drivers that got updated etc).
>
> So it's been over two and a half months, and while it's certainly not the
> longest release cycle ever, it still dragged out a bit longer than I'd
> have hoped for and it should have. As usual, I'd like to thank Adrian (and
> the people who jumped on the entries Adrian had) for keeping everybody on
> their toes with the regression list - there's a few entries there still,
> but it got to the point where we didn't even know if they were real
> regressions, and delaying things further just wasn't going to help.

I've just compiled 2.6.21, seems to run fine, in tickless mode, on a Dell 
D610.

Is there any way to check how many timer ticks are firing, except for 
monitoring /proc/interrupts?

Thanks for all the hard work!

Jan

-- 
She has an alarm clock and a phone that don't ring -- they applaud.

Re: Linux 2.6.21

[Marat Buharov]

[Offtopic]
Today, April, 26, 21 year has been passed since Chernobyl Nuclear
Power Plant disaster, and Linus announced .... *drum roll* .... 2.6.21
!!! What a mysterious coincidence...

Re: Linux 2.6.21

[Jan Engelhardt]

On Apr 26 2007 12:23, Marat Buharov wrote:
>
> [Offtopic]
> Today, April, 26, 21 year has been passed since Chernobyl Nuclear
> Power Plant disaster, and Linus announced .... *drum roll* .... 2.6.21
> !!! What a mysterious coincidence...

And 2.6.26 will be released on April 01 2008.


Jan
-- 

Re: Linux 2.6.21

[Jan Engelhardt]

On Apr 25 2007 20:29, Linus Torvalds wrote:
>
>If the goal for 2.6.20 was to be a stable release (and it was), the goal
>for 2.6.21 is to have just survived the big timer-related changes and
>some of the other surprises [...] So it's been over two and a half
>months, and while it's certainly not the longest release cycle ever, it
>still dragged out a bit longer than I'd have hoped for and it should
>have.

I really appreciate the lot of -rcs, especially if there are so many 
intrusive changes/regressions. Like Andrew, I have a feeling that it 
gets buggier, but at least, it seems to be made up every ... two 
releases. 2.6.16 was a good one, .18, and .20. I suppose the 
bug/regression distribution between [2.6.16-2.6.17, 2.6.17-2.6.18] was 
biased like [70, 50]. About 2.6.21 - will see, rc has been to my liking.
Since a picture says more than a thousand words, have
http://jengelh.hopto.org/GFX0/kernel-rating-2.6.21.png
(The last kernel to have only 5 -rcs was 2.6.14 - interesting)


Happy hacking,
Jan
-- 

Re: Linux 2.6.21

[Linus Torvalds]

On Thu, 26 Apr 2007, Jan Engelhardt wrote:
>
> I really appreciate the lot of -rcs, especially if there are so many 
> intrusive changes/regressions. Like Andrew, I have a feeling that it 
> gets buggier, but at least, it seems to be made up every ... two 
> releases.

I wouldn't say that, but yes, there is at least *some* tendency to not 
merge scary stuff after a painful release.

For example, I can certainly say that after 2.6.21, I'm likely to be very 
unhappy merging something that isn't "obviously safe". I knew the timer 
changes were potentially painful, I just hadn't realized just *how* 
painful they would be (we had some SATA/IDE changes too, of course, it's 
not all just about the timers, those just ended up being more noticeable 
to me than some of the other things were).

> About 2.6.21 - will see, rc has been to my liking.

I actually hope that 2.6.21 isn't even all that bad, despite all the 
worries about it. And I may be complaining about the problems the timers 
caused, but it was definitely something that was not only worth it, it was 
overdue - and those NO_HZ issues had been brewing literally for years. So 
considering issues like that, I think we're actually doing fairly well.

One of the bigger issues is that I think -mm (and I'm pretty sure Andrew 
will agree with me on this) has really had a rather spotty history. It's 
been unstable enough at times that I suspect people have largely stopped 
testing it, with just the most die-hard testers running -mm.

So -mm is still very useful just because *Andrew* tests it, and finds all 
kinds of issues with it, but I literally suspect that Andrew himself is 
personally a big part of that, which is kind of wasteful - we should be 
able to spread out the pain more. Andrew is also too damn polite when 
something goes wrong ;)

So we should have somebody like Christoph running -mm, and when things 
break, we'll just sic Christoph on whoever broke it, and teach people 
proper fear and respect! As it is, I think people tend to send things to 
-mm a bit *too* eagerly, because there is no downside - Andrew is a "cheap 
date" testing-wise, and always puts out ;)

			Linus

Re: Linux 2.6.21

[Willy Tarreau]

On Thu, Apr 26, 2007 at 09:40:26AM -0700, Linus Torvalds wrote:

> So we should have somebody like Christoph running -mm, and when things 
> break, we'll just sic Christoph on whoever broke it, and teach people 
> proper fear and respect!

And with Al Viro doing random code review and fill in the commits for
regression fixes, even long established developers will check their code
twice before submitting ;-)

Willy

Re: Linux 2.6.21

[Mike Galbraith]

On Thu, 2007-04-26 at 21:02 +0200, Willy Tarreau wrote:
> On Thu, Apr 26, 2007 at 09:40:26AM -0700, Linus Torvalds wrote:
> 
> > So we should have somebody like Christoph running -mm, and when things 
> > break, we'll just sic Christoph on whoever broke it, and teach people 
> > proper fear and respect!
> 
> And with Al Viro doing random code review and fill in the commits for
> regression fixes, even long established developers will check their code
> twice before submitting ;-)

Yeah!  We can call them The Black And Blues Brothers :)

Re: Linux 2.6.21

[Jan Engelhardt]

On Apr 26 2007 09:40, Linus Torvalds wrote:
>On Thu, 26 Apr 2007, Jan Engelhardt wrote:
>>
>For example, I can certainly say that after 2.6.21, I'm likely to be very 
>unhappy merging something that isn't "obviously safe". I knew the timer 
>changes were potentially painful, I just hadn't realized just *how* 
>painful they would be (we had some SATA/IDE changes too, of course, it's 
>not all just about the timers, those just ended up being more noticeable 
>to me than some of the other things were).

Perhaps do one at a time [ at the cost of queueing other stuff, yeah :( ]
Like: 2.6.21 - only NO_HZ & hrtimers, and the SATA code in .22. Probably does
not work out in reality, so perhaps just live with long rc cycles.
(Let rc8 come.)

>So we should have somebody like Christoph running -mm, and when things 
>break, we'll just sic Christoph on whoever broke it, and teach people 
>proper fear and respect! As it is, I think people tend to send things to 
>-mm a bit *too* eagerly, because there is no downside - Andrew is a "cheap 
>date" testing-wise, and always puts out ;)

Yes, perhaps we need a weakchanges-mm ("weak" is inteded, not to be confused
with week) that can carry stuff like doc updates, Kconfig updates, etc. -
patches that are a little more than -trivial.


Jan
-- 

Re: Linux 2.6.21

[Mel Gorman]

On (26/04/07 09:40), Linus Torvalds didst pronounce:
> 
> 
> On Thu, 26 Apr 2007, Jan Engelhardt wrote:
> >
> > I really appreciate the lot of -rcs, especially if there are so many 
> > intrusive changes/regressions. Like Andrew, I have a feeling that it 
> > gets buggier, but at least, it seems to be made up every ... two 
> > releases.
> 
> I wouldn't say that, but yes, there is at least *some* tendency to not 
> merge scary stuff after a painful release.
> 
> For example, I can certainly say that after 2.6.21, I'm likely to be very 
> unhappy merging something that isn't "obviously safe". I knew the timer 
> changes were potentially painful, I just hadn't realized just *how* 
> painful they would be (we had some SATA/IDE changes too, of course, it's 
> not all just about the timers, those just ended up being more noticeable 
> to me than some of the other things were).
> 
> > About 2.6.21 - will see, rc has been to my liking.
> 
> I actually hope that 2.6.21 isn't even all that bad, despite all the 
> worries about it. And I may be complaining about the problems the timers 
> caused, but it was definitely something that was not only worth it, it was 
> overdue - and those NO_HZ issues had been brewing literally for years. So 
> considering issues like that, I think we're actually doing fairly well.
> 
> One of the bigger issues is that I think -mm (and I'm pretty sure Andrew 
> will agree with me on this) has really had a rather spotty history. It's 
> been unstable enough at times that I suspect people have largely stopped 
> testing it, with just the most die-hard testers running -mm.
> 

test.kernel.org also picks up -mm and additional machines run -mm
kernels. It doesn't catch everything but a few bugs get rattled out. That
said, it's automated with Andy Whitcroft and Steve Fox kicking it along
periodically. After spending the day tracking just two issues in -mm and
simple ones at that, the lack of testing may be simply because it's really
bloody boring even with the access to an automated system. There is no
avoiding this really, fixing regressions will never be entertaining.

> So -mm is still very useful just because *Andrew* tests it, and finds all 
> kinds of issues with it, but I literally suspect that Andrew himself is 
> personally a big part of that, which is kind of wasteful - we should be 
> able to spread out the pain more. Andrew is also too damn polite when 
> something goes wrong ;)
> 

A few more "you're a spanner" mails probably would not hurt even though
I'm pretty sure I'll receive a fair number of them :/

> So we should have somebody like Christoph running -mm, and when things 
> break, we'll just sic Christoph on whoever broke it, and teach people 
> proper fear and respect! As it is, I think people tend to send things to 
> -mm a bit *too* eagerly, because there is no downside - Andrew is a "cheap 
> date" testing-wise, and always puts out ;)
> 
> 			Linus

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

Re: Linux 2.6.21

[linux]

Today, 26 april, is the *21*'th anniversary of nuclear 
explosion at Chernobyl's station (ex USSR).
And linux 2.6.*21* is released. Nice!

Re: Linux 2.6.21

[Tomasz Chmielewski]

Adrian Bunk wrote:

> I do hereby promise you to manually ask the submitters of all 1600 open 
> bugs in the kernel Bugzilla within one month whether their problem is 
> still present with 2.6.21 and forwarding all bugs if the answer was 
> "yes" to whoever is the right recipient if you promise me that all bugs 
> where the submitter said "yes" will be debugged by a kernel developer 
> who knows the corresponding subsystem. [4]

Hmm.
I reported a bug on 2006-08-07 ("ISDN/hisax doesn't work on ARM 
architecture"):

http://bugzilla.kernel.org/show_bug.cgi?id=6970


There was not much activity there... and the bug is still opened in 
kernel's bugzilla.


However, I received two emails from Andrew Morton on 02.10.2006 - one 
with a patch, and the second (automated?) which said:

The patch titled

      isdn: work around excessive udelay()

has been removed from the -mm tree.  Its filename is

      isdn-work-around-excessive-udelay.patch

This patch was dropped because it was merged into mainline or a 
subsystem tree


I'm very happy that the bug was fixed, but why wasn't it automatically 
closed in bugzilla (I closed it only today as I looked up into bugzilla)?

Do we have some wrong communication here? How many bugs are there that 
are falsely opened, when in reality they were resolved long ago?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[Tomasz Chmielewski]

David Miller wrote:

>> Don't think that's true. There are plenty of projects who only
>> accept bugs through bugzilla (mozilla, various distributions, etc.) 
>> and I don't see any evidence of your claim being true.
> 
> That explains why my bugs don't get looked at for months if
> not years when I submit them to such projects.
> 
> I reported a bug that eats people's hard disks due to a bug
> in the X.ORG PCI support code on sparc, NOBODY has fixed
> the bug in 2 years even though a full bugzilla entry with
> even a full patch fix is in there.

And how fast was the bug fixed when you posted it to the X.ORG list?


> Bugzilla sucks, emails rules because it is in your face and
> gets people to work on things.

Bugzilla can be configured to send emails, too (to the list for a newly 
reported bug for example).


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[Arkadiusz Miskiewicz]

On Sunday 29 of April 2007, Tomasz Chmielewski wrote:

> > I reported a bug that eats people's hard disks due to a bug
> > in the X.ORG PCI support code on sparc, NOBODY has fixed
> > the bug in 2 years even though a full bugzilla entry with
> > even a full patch fix is in there.
>
> And how fast was the bug fixed when you posted it to the X.ORG list?

Not so long ago I got such reply on xorg@ ml:

,,Sorry, without a bugzilla entry, I lost your bug report. ''

xorg people want bugzilla reports (well, they don't fix them anyway, too).

-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/

Re: Linux 2.6.21

[Tomasz Chmielewski]

Linus Torvalds wrote:

> On Sun, 29 Apr 2007, Adrian Bunk wrote:
>> 
>> The kernel Bugzilla currently contains 1600 open bugs.
> 
> Adrian, why do you keep harping on this, and ignoring reality?
> 
> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> 
> How many of those are interesting and valid? How many of them are 
> relevant? How many of them are duplicates?

And - how many of these bug reports have kernel's bugzilla ever 
forwarded to lkml so that other people could see them?

Is that number zero (because kernel's bugzilla is configured this way)?


-- 
Tomasz Chmielewski

Re: Linux 2.6.21

[Adrian Bunk]

On Sun, Apr 29, 2007 at 11:04:10PM +0200, Tomasz Chmielewski wrote:
> Linus Torvalds wrote:
>
>> On Sun, 29 Apr 2007, Adrian Bunk wrote:
>>> The kernel Bugzilla currently contains 1600 open bugs.
>> Adrian, why do you keep harping on this, and ignoring reality?
>> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
>> How many of those are interesting and valid? How many of them are 
>> relevant? How many of them are duplicates?
>
> And - how many of these bug reports have kernel's bugzilla ever forwarded 
> to lkml so that other people could see them?
>
> Is that number zero (because kernel's bugzilla is configured this way)?

Andrew forwards incoming Bugzilla bugs to the responsible maintainers 
(if there are any).

If it is considered useful it shouldn't be a problem to automatically 
forward all incoming Bugzilla bugs to linux-kernel.

> Tomasz Chmielewski

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Bernd Eckenfels]

In article <20070430030213.GB23336@stusta.de> you wrote:
> If it is considered useful it shouldn't be a problem to automatically 
> forward all incoming Bugzilla bugs to linux-kernel.

Yes, most of it to linux-kernel, some components (netdev@, architecture) to
a more specific list.

Gruss
Bernd

Re: Linux 2.6.21

[Andrew Morton]

(various cc's reestablished.  Please don't remove cc's when dealing with
kernel people).

On Mon, 30 Apr 2007 06:57:08 +0200 Bernd Eckenfels <ecki@lina.inka.de> wrote:

> In article <20070430030213.GB23336@stusta.de> you wrote:
> > If it is considered useful it shouldn't be a problem to automatically 
> > forward all incoming Bugzilla bugs to linux-kernel.
> 
> Yes, most of it to linux-kernel, some components (netdev@, architecture) to
> a more specific list.
> 

I think it would be a net good thing to send bugme-new notifications to lkml,
actually.  There are only maybe 5-10 per day (mad guess).

Re: Linux 2.6.21

[Tomasz Chmielewski]

Adrian Bunk schrieb:
> On Sun, Apr 29, 2007 at 11:04:10PM +0200, Tomasz Chmielewski wrote:
>> Linus Torvalds wrote:
>>
>>> On Sun, 29 Apr 2007, Adrian Bunk wrote:
>>>> The kernel Bugzilla currently contains 1600 open bugs.
>>> Adrian, why do you keep harping on this, and ignoring reality?
>>> Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
>>> How many of those are interesting and valid? How many of them are 
>>> relevant? How many of them are duplicates?
>> And - how many of these bug reports have kernel's bugzilla ever forwarded 
>> to lkml so that other people could see them?
>>
>> Is that number zero (because kernel's bugzilla is configured this way)?
> 
> Andrew forwards incoming Bugzilla bugs to the responsible maintainers 
> (if there are any).
> 
> If it is considered useful it shouldn't be a problem to automatically 
> forward all incoming Bugzilla bugs to linux-kernel.

Why isn't it done yet? If the bugs were already forwarded to 
linux-kernel (and perhaps, to linux-<subsystem> when possible, too), we 
would save at least two days of this long "Linux 2.6.21" thread...

For I somehow feel that most people here dislike bugzilla because of 
misconceptions - which only arose as bugzilla.kernel.org is *really* 
misconfigured.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[Gene Heskett]

On Monday 30 April 2007, Tomasz Chmielewski wrote:
>For I somehow feel that most people here dislike bugzilla because of
>misconceptions - which only arose as bugzilla.kernel.org is *really*
>misconfigured.

Bugzilla was indeed miss-conceived.  It shoulda been on birth control pills.

I'm not claiming that we don't need something, but bugzilla sure as heck isn't 
it.

-- 
Cheers, Gene
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Where's the Coke machine?  Tell me a joke!!

Re: Linux 2.6.21

[Tomasz Chmielewski]

David Miller wrote:

>> > I reported a bug that eats people's hard disks due to a bug
>> > in the X.ORG PCI support code on sparc, NOBODY has fixed
>> > the bug in 2 years even though a full bugzilla entry with
>> > even a full patch fix is in there.
>> 
>> Well but at least they could find it again if they wanted.
>> If you sent it by email and it had gotten lost for some reason
>> (nobody interested, which seems to be the real issue here) then
>> it would be lost forever.
> 
> WRONG!  If I have sent it to the main developer list the damn patch
> would be applied by now.

Why didn't you do it then? Why didn't you send your patch to the main 
developer?
Wouldn't be your problem fixed if you did it?


> WHY?
> 
> BECAUSE EMAIL ENGAGES PEOPLE AND BUGZILLA DOES NOT!

-- 
Tomasz Chmielewski
http://wpkg.org

Re: Linux 2.6.21

[David Miller]

From: Tomasz Chmielewski <mangoo@wpkg.org>
Date: Mon, 30 Apr 2007 08:09:13 +0200

> Why didn't you do it then? Why didn't you send your patch to the main 
> developer?
> Wouldn't be your problem fixed if you did it?

Because all the directions say to report bugs to the bugzilla via
bugs.freedesktop.org, which I did.

I went out of my way to go through the hassle of creating an
account, figuring out which buttons to push, how to attach
a patch, etc.

And my reward for all that effort is that either my work gets
ignored or I should send it to the mailing list?

That's garbage.

If bugs should be reported to the mailing list, then they
should just get rid of bugzilla because it's aparently
serving as a garbage bin.

Re: Linux 2.6.21

[Adrian Bunk]

On Mon, Apr 30, 2007 at 12:01:38AM -0700, David Miller wrote:
>...
> If bugs should be reported to the mailing list, then they
> should just get rid of bugzilla because it's aparently
> serving as a garbage bin.

The first question is not "Bugzilla" but "Does bug tracking make sense?".

Many bug reports to linux-kernel get zero attention and are lost without 
tracking. Is losing bug reports OK or do we need a tracking of bugs?

Tracking means:
- not to miss bug reports
- record discussions and the status of the bugs
- the real value comes from:
  provide useful reports like "all open ACPI bugs" or "all 2.6.21-rc 
  regressions" to people who are actually working on fixing the bugs

I was tracking 2.6.21-rc regressions.
Some people said this was useful.

Manually tracking of three dozen regressions plus sending regular sorted 
reports was at the limit of what I am able to handle manually without 
any tracking tool.

There are two different questions that seem to often be mixed in 
discussions:

The first question is:
Does it make sense to track kernel bugs?

If no, there's no need to discuss Bugzilla or other tools.

If yes, the second question is:
Which tracking tool would make sense for the Linux kernel?
This could be Bugzilla, the Debian bug tracking system, or even 
something written from scratch.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Linux 2.6.21

[Nicolas Mailhot]

Johannes Stezenbach wrote :
> On Sun, Apr 29, 2007, Adrian Bunk wrote:
> > On Sun, Apr 29, 2007 at 01:33:16PM -0700, Linus Torvalds wrote:
> > > On Sun, 29 Apr 2007, Adrian Bunk wrote:
> > > > 
> > > > The kernel Bugzilla currently contains 1600 open bugs.
> > > 
> > > Adrian, why do you keep harping on this, and ignoring reality?
> > > 
> > > Kernel bugzilla has 1600 open bugs BECAUSE IT SUCKS.
> > 
> > OK, how do you suggest to track bugs in a way that doesn't suck?
> > 
> > Bug reports to linux-kernel have the big problem that they are lost if 
> > no developer immediately picks them up.
> 
> I suspect some bug reports get ignored deliberately.

[...]

You're probably not the only one thinking this but you know? This kind
of message makes me barf

> Y'know, a lot of developers don't just work for the money, they
> work for the fun of it. Thus they try to avoid pain and grief. :-)

Y'know, a lot of bug reporters are not paid at all, they try to help by
reporting problems.

> it's hard to write good bug reports,
[...]
> Thus they try to avoid pain and grief. :-)

It's way easier (and rewarding) to flame Linux on MLs, forums, articles
and make it a reputation darker than the darkest night than to have to
engage some developers on a bug report.

> Thus it's
> just natural that you learn from the experience to ignore every
> bug report which even remotely smells fishy ;-/

Thus it's just natural that you learn from the experience to stop
reporting problems and let some other poor sod do it, and instead spend
your time productively somewhere else

> Bugzilla just makes this visible because it doesn't forget.
> Which is stupid and discouraging for both (potential) bug
> reporters and developers.

Are you kidding? Do you think having a painfully drafted e-mail report
ignored is any less discouraging? Why do you think Adrian Bunk is
blowing a fuse today?

> And I also think that ignoring bad bug reports _increases_
> the software quality, because you can use the saved time
> working on something productive. 

All it increases is the lack of testing since ignored people stop
reporting bugs, so you have :
1. a small minority of hardened bug reporters (seems even Adrian is not
hardened enough to keep on it)
2. a majority of newbie reporters that don't know how to do clean
reports yet, and will vanish from the circuit before learning as soon as
they get the "ignored" treatment.

> And it makes developers happier :-)

That's the actual reason and it's a bad one. Unless developers have the
hardware, time and energy to do their own testing. Which is not the case
(read Andrew Morton's periodical complaints)

Now, here is what this bug reporter thinks:

1. bugzilla sucks for all the reasons written before

2. e-mail reports suck more:
- unless you post at the right time you get ignored
- you spend your time re-sending the same info because people don't read
thread histories
- you can't attach significant debugging material such as logs because
you'll hit random list size limits
- you get your mail address harversted by spammers (and if you create a
one-shut address people complain you don't consult it often enough)
- list filters will randomly decide your message is spam, and your
report will be lost
- you can't easily reassign reports once a group told you the bug is
somewhere else, you have to restart from step 1
- you don't even have the sucky target list you have in bugzilla - you
spend more time finding maintainer and list addresses than writing the
report (and then you have to monitor a gazillon of list to check if
someone read your stuff)
- people will write you "this looks the same as foo" and you'll spend
some more hours locating foo instead of having a clear pointer to
another report

e-mail is all nice and dandy for people who:
- have setup the infrastructure on their desktop to process large mail
volumes
- are behind strong spam filters
- have the clout to be listened to at once
- have all the relevant list & maintainers pointers in their head
- have helpers like Adrian to sort their mail for them
- have some personal http space where they can put bulky info the lists
won't accept

For the average bug reporter it's a very hostile system. Most people
don't have any of this, that's what bugzilla provides (and of course if
you have your own personnal infrastructure bugzilla may seem
unnecessary)

Don't get me wrong I had bugs solved through e-mail (or irc), and I had
bugs solved through bugzilla, and every time having a motivated fixer
and a good bug report was more important than the method used. These
days I only report through bugzilla because it makes it easier to attach
stuff and create good reports.

Now, the critical part on any bug reporting system is people like
Adrian. You can have reporter-oriented systems (bugzilla) or
developer-oriented systems (e-mail) what makes them work are the
go-betweens (who happen to do the less sexy and hardest part of the
process). They're the ones who mediate between:
1. reporters who just want to report a problem without spending a month
learning the habits of the people in charge of the code
2. developers who want gold-plated reports

So IMHO:

1. people like Adrian should be the ones choosing the reporting method
so it makes *their* job not the reporter or developer ones easier

2. when they bend backwards to send reports in the developer-preferred
form (like Adrian did) it's criminal to ignore them like a random e-mail
or bugzilla report. The bug may be more work than the technical problem
is worth but that's not the point. People like Adrian are not
replaceable. When they've invested so much time producing nice reports
you have to act on them if only to mark your respect for people doing a
thankless but necessary job.

-- 
Nicolas Mailhot