From: Carlos O'Donell <carlos@redhat.com> To: Steven Rostedt <rostedt@goodmis.org> Cc: linux-api@vger.kernel.org, Thorsten Leemhuis <linux@leemhuis.info>, ksummit-discuss@lists.linuxfoundation.org, Shuah Khan <shuahkh@osg.samsung.com> Subject: Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking Date: Wed, 5 Jul 2017 10:06:24 -0400 [thread overview] Message-ID: <9b377a08-bf38-b41e-040c-41cb078bcfc3@redhat.com> (raw) In-Reply-To: <20170705092757.63dc2328@gandalf.local.home> On 07/05/2017 09:27 AM, Steven Rostedt wrote: >> As a concrete action item, glibc core developers took a harder stance on >> (a) all user-visible bugs need a bug # (forces people to think about the > > Unfortunately, we don't have a good system for a "bug #". Most kernel > developers hate bugzilla, and I think that includes Linus ;-) Which > means, unless Linus builds us a new bug tracking system, there wont be > any mandate for it. Use the XMLRPC API to build a better interface for kernel developers? Our "fixed bugs" list is automatically culled via XMLRPC to generate our release announcement with "fixed bugs." The bug # mandate has had a few key effects. It allows non-developers to search for old similar regressions in an easier fashion than having to trawl the mailing list for incomprehensible (to them) discussions about semantics. The bugs are described and talked about in terms of user facing aspects, not internal implementation details. Regressed bugs can be reopened and discussed on the mailing list with links to the discussions and summaries of conclusions. All of this means we have a cleaner, clearer, description of the problem from the user side. This again needs priority from a group of people for whom time is precious, so you have to get buy in from them. I don't think (a) is needed, but the glibc community found it helpful. >> problem and file a coherent public bug about it) (b) all bugs needs a >> regression test if possible, (c) and if not possible we need to extend > > I would love all bug fixes to come with a test (when possible). We have lots of hardware-specific tests that are marked UNSUPPORTED if say you're not running on AVX512 enabled hardware. >> the testing framework to make it possible (we've started using kernel >> namespaces to create isolated test configurations). > > Well, we have a selftest directory that should include all of these. > And most people run them on either a test box or a VM. Improving the test infrastructure must also be a priority, otherwise you will grow to the limit of that infrastructure. >> This change in reviewer priorities has had a noticeable impact on developer >> priorities over the last 5 years. Timelines for this problem will be >> measured in years. > > Your "b" above is what I would like to push. But who's going to enforce > this? With 10,000 changes per release, and a lot of them are fixes, the > best we can do is the honor system. Start shaming people that don't > have a regression test along with a Fixes tag (but we don't want people > to fix bugs without adding that tag either). There is a fine line one > must walk between getting people to change their approaches to bugs and > regression tests, and pissing them off where they start doing the > opposite of what would be best for the community. I did say "hard problem" earlier didn't I? * Start with yourself. * For everyone you know well, and have met in person, be brutal and require them to submit regression tests with their bug fixes. These people are already committed to getting their fixes in and they will understand you are making an example of them. * For everyone you don't know well, be gentle, and begin reminding them you need a regression test, and if you feel generous try to write one yourself for them. Often the act of writing such a test will show you how hard it is, and what is missing from your infrastructure to make this easy, because if it was easy everyone would do it. YMMV. -- Cheers, Carlos.
WARNING: multiple messages have this Message-ID (diff)
From: Carlos O'Donell <carlos-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> To: Steven Rostedt <rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org> Cc: Thorsten Leemhuis <linux-rCxcAJFjeRkk+I/owrrOrA@public.gmane.org>, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I@public.gmane.org, Shuah Khan <shuahkh-JPH+aEBZ4P+UEJcrhfAQsw@public.gmane.org>, linux-api-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking Date: Wed, 5 Jul 2017 10:06:24 -0400 [thread overview] Message-ID: <9b377a08-bf38-b41e-040c-41cb078bcfc3@redhat.com> (raw) In-Reply-To: <20170705092757.63dc2328-f9ZlEuEWxVcJvu8Pb33WZ0EMvNT87kid@public.gmane.org> On 07/05/2017 09:27 AM, Steven Rostedt wrote: >> As a concrete action item, glibc core developers took a harder stance on >> (a) all user-visible bugs need a bug # (forces people to think about the > > Unfortunately, we don't have a good system for a "bug #". Most kernel > developers hate bugzilla, and I think that includes Linus ;-) Which > means, unless Linus builds us a new bug tracking system, there wont be > any mandate for it. Use the XMLRPC API to build a better interface for kernel developers? Our "fixed bugs" list is automatically culled via XMLRPC to generate our release announcement with "fixed bugs." The bug # mandate has had a few key effects. It allows non-developers to search for old similar regressions in an easier fashion than having to trawl the mailing list for incomprehensible (to them) discussions about semantics. The bugs are described and talked about in terms of user facing aspects, not internal implementation details. Regressed bugs can be reopened and discussed on the mailing list with links to the discussions and summaries of conclusions. All of this means we have a cleaner, clearer, description of the problem from the user side. This again needs priority from a group of people for whom time is precious, so you have to get buy in from them. I don't think (a) is needed, but the glibc community found it helpful. >> problem and file a coherent public bug about it) (b) all bugs needs a >> regression test if possible, (c) and if not possible we need to extend > > I would love all bug fixes to come with a test (when possible). We have lots of hardware-specific tests that are marked UNSUPPORTED if say you're not running on AVX512 enabled hardware. >> the testing framework to make it possible (we've started using kernel >> namespaces to create isolated test configurations). > > Well, we have a selftest directory that should include all of these. > And most people run them on either a test box or a VM. Improving the test infrastructure must also be a priority, otherwise you will grow to the limit of that infrastructure. >> This change in reviewer priorities has had a noticeable impact on developer >> priorities over the last 5 years. Timelines for this problem will be >> measured in years. > > Your "b" above is what I would like to push. But who's going to enforce > this? With 10,000 changes per release, and a lot of them are fixes, the > best we can do is the honor system. Start shaming people that don't > have a regression test along with a Fixes tag (but we don't want people > to fix bugs without adding that tag either). There is a fine line one > must walk between getting people to change their approaches to bugs and > regression tests, and pissing them off where they start doing the > opposite of what would be best for the community. I did say "hard problem" earlier didn't I? * Start with yourself. * For everyone you know well, and have met in person, be brutal and require them to submit regression tests with their bug fixes. These people are already committed to getting their fixes in and they will understand you are making an example of them. * For everyone you don't know well, be gentle, and begin reminding them you need a regression test, and if you feel generous try to write one yourself for them. Often the act of writing such a test will show you how hard it is, and what is missing from your infrastructure to make this easy, because if it was easy everyone would do it. YMMV. -- Cheers, Carlos.
next prev parent reply other threads:[~2017-07-05 14:06 UTC|newest] Thread overview: 144+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-07-02 17:51 [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking Thorsten Leemhuis 2017-07-03 16:30 ` Steven Rostedt 2017-07-03 16:30 ` Steven Rostedt 2017-07-03 18:50 ` Dan Williams 2017-07-03 18:50 ` Dan Williams 2017-07-04 19:03 ` Thorsten Leemhuis 2017-07-04 19:03 ` Thorsten Leemhuis 2017-07-05 12:45 ` Steven Rostedt 2017-07-05 12:45 ` Steven Rostedt 2017-07-05 13:09 ` Carlos O'Donell 2017-07-05 13:09 ` Carlos O'Donell 2017-07-05 13:27 ` Steven Rostedt 2017-07-05 13:27 ` Steven Rostedt 2017-07-05 14:06 ` Greg KH 2017-07-05 14:06 ` Greg KH 2017-07-05 14:28 ` Carlos O'Donell 2017-07-05 14:28 ` Carlos O'Donell 2017-07-05 14:33 ` Steven Rostedt 2017-07-05 14:33 ` Steven Rostedt 2017-07-05 14:52 ` Mark Brown 2017-07-05 14:52 ` Mark Brown 2017-07-05 15:08 ` Carlos O'Donell 2017-07-05 15:08 ` Carlos O'Donell 2017-07-05 16:10 ` Steven Rostedt 2017-07-05 16:10 ` Steven Rostedt 2017-07-06 11:34 ` Laurent Pinchart 2017-07-06 11:34 ` Laurent Pinchart 2017-07-09 13:46 ` Thorsten Leemhuis 2017-07-09 13:46 ` Thorsten Leemhuis 2017-07-05 14:33 ` Mark Brown 2017-07-05 14:33 ` Mark Brown 2017-07-05 14:36 ` Steven Rostedt 2017-07-05 14:36 ` Steven Rostedt 2017-07-05 14:50 ` James Bottomley 2017-07-05 14:50 ` James Bottomley 2017-07-05 14:56 ` Steven Rostedt 2017-07-05 14:56 ` Steven Rostedt 2017-07-05 15:09 ` James Bottomley 2017-07-05 15:09 ` James Bottomley 2017-07-05 15:20 ` Mark Brown 2017-07-05 15:20 ` Mark Brown 2017-07-05 15:40 ` Geert Uytterhoeven 2017-07-05 15:40 ` Geert Uytterhoeven 2017-07-05 15:20 ` Steven Rostedt 2017-07-05 15:20 ` Steven Rostedt 2017-07-05 15:32 ` James Bottomley 2017-07-05 15:32 ` James Bottomley 2017-07-05 15:43 ` Steven Rostedt 2017-07-05 15:43 ` Steven Rostedt 2017-07-05 18:24 ` Daniel Vetter 2017-07-05 18:24 ` Daniel Vetter 2017-07-05 18:17 ` Daniel Vetter 2017-07-05 18:17 ` Daniel Vetter 2017-07-05 15:16 ` Guenter Roeck 2017-07-05 15:16 ` Guenter Roeck 2017-07-05 15:27 ` Steven Rostedt 2017-07-05 15:27 ` Steven Rostedt 2017-07-05 15:36 ` James Bottomley 2017-07-05 15:36 ` James Bottomley 2017-07-05 16:04 ` Steven Rostedt 2017-07-05 16:04 ` Steven Rostedt 2017-07-05 16:58 ` James Bottomley 2017-07-05 16:58 ` James Bottomley 2017-07-05 17:07 ` Steven Rostedt 2017-07-05 17:07 ` Steven Rostedt 2017-07-05 16:48 ` Guenter Roeck 2017-07-05 16:48 ` Guenter Roeck 2017-07-05 16:58 ` Dan Williams 2017-07-05 16:58 ` Dan Williams 2017-07-05 17:02 ` Steven Rostedt 2017-07-05 17:02 ` Steven Rostedt 2017-07-06 9:28 ` Mark Brown 2017-07-06 9:28 ` Mark Brown 2017-07-06 9:41 ` Daniel Vetter 2017-07-06 9:41 ` Daniel Vetter 2017-07-06 14:53 ` Theodore Ts'o 2017-07-06 14:53 ` Theodore Ts'o 2017-07-06 21:28 ` Daniel Vetter 2017-07-06 21:28 ` Daniel Vetter 2017-07-06 14:48 ` James Bottomley 2017-07-06 14:48 ` James Bottomley 2017-07-07 10:03 ` Mark Brown 2017-07-07 10:03 ` Mark Brown 2017-07-31 16:54 ` Eric W. Biederman 2017-07-31 16:54 ` Eric W. Biederman 2017-07-31 20:11 ` Steven Rostedt 2017-07-31 20:11 ` Steven Rostedt 2017-07-31 20:12 ` Eric W. Biederman 2017-07-31 20:12 ` Eric W. Biederman 2017-08-02 16:53 ` Shuah Khan 2017-08-02 16:53 ` Shuah Khan 2017-08-02 17:33 ` Eric W. Biederman 2017-08-02 17:33 ` Eric W. Biederman 2017-08-02 17:46 ` Shuah Khan 2017-08-02 17:46 ` Shuah Khan 2017-08-02 17:58 ` Shuah Khan 2017-08-02 17:58 ` Shuah Khan 2017-08-02 18:04 ` Eric W. Biederman 2017-08-02 18:04 ` Eric W. Biederman 2017-08-02 18:23 ` Randy Dunlap 2017-08-02 18:23 ` Randy Dunlap 2017-08-02 18:42 ` Shuah Khan 2017-08-02 18:42 ` Shuah Khan 2017-08-03 3:03 ` Theodore Ts'o 2017-08-03 3:03 ` Theodore Ts'o 2017-08-03 17:42 ` Bird, Timothy 2017-08-03 17:42 ` Bird, Timothy 2017-08-03 22:11 ` Shuah Khan 2017-08-03 22:11 ` Shuah Khan 2017-08-03 18:51 ` Shuah Khan 2017-08-03 18:51 ` Shuah Khan 2017-08-04 1:15 ` Theodore Ts'o 2017-08-04 1:15 ` Theodore Ts'o 2017-07-07 3:33 ` Fengguang Wu 2017-07-07 3:33 ` Fengguang Wu 2017-07-07 4:52 ` Frank Rowand 2017-07-07 4:52 ` Frank Rowand 2017-07-05 15:32 ` Greg KH 2017-07-05 15:32 ` Greg KH 2017-07-05 15:36 ` Carlos O'Donell 2017-07-05 15:36 ` Carlos O'Donell 2017-07-05 15:52 ` Steven Rostedt 2017-07-05 15:52 ` Steven Rostedt 2017-07-05 18:42 ` Greg KH 2017-07-05 18:42 ` Greg KH 2017-07-05 18:29 ` Daniel Vetter 2017-07-05 18:29 ` Daniel Vetter 2017-07-06 22:24 ` Shuah Khan 2017-07-06 22:24 ` Shuah Khan 2017-07-06 22:32 ` Steven Rostedt 2017-07-06 22:32 ` Steven Rostedt 2017-07-06 22:40 ` Shuah Khan 2017-07-06 22:40 ` Shuah Khan 2017-07-05 16:54 ` Dan Williams 2017-07-05 16:54 ` Dan Williams 2017-07-05 18:45 ` Greg KH 2017-07-05 18:45 ` Greg KH 2017-07-05 19:47 ` Dan Williams 2017-07-05 19:47 ` Dan Williams 2017-07-05 14:06 ` Carlos O'Donell [this message] 2017-07-05 14:06 ` Carlos O'Donell 2017-07-05 15:47 ` Mark Brown 2017-07-05 15:47 ` Mark Brown 2017-07-07 6:15 ` Andrei Vagin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=9b377a08-bf38-b41e-040c-41cb078bcfc3@redhat.com \ --to=carlos@redhat.com \ --cc=ksummit-discuss@lists.linuxfoundation.org \ --cc=linux-api@vger.kernel.org \ --cc=linux@leemhuis.info \ --cc=rostedt@goodmis.org \ --cc=shuahkh@osg.samsung.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.