Re: what trees/branches to test on syzbot

From: Guenter Roeck <groeck@google.com>
To: Dmitry Vyukov <dvyukov@google.com>
Cc: "Theodore Ts'o" <tytso@mit.edu>,
	penguin-kernel@i-love.sakura.ne.jp,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Eric W. Biederman" <ebiederm@xmission.com>,
	linux-kernel <linux-kernel@vger.kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	syzkaller <syzkaller@googlegroups.com>,
	Stephen Rothwell <sfr@canb.auug.org.au>,
	David Miller <davem@davemloft.net>,
	kbuild test robot <fengguang.wu@intel.com>
Subject: Re: what trees/branches to test on syzbot
Date: Tue, 26 Jun 2018 07:54:50 -0700	[thread overview]
Message-ID: <CABXOdTe1HCy78w+G4LuN8S7Ad5kvUxwHsiP2a=8aW-jEvpjdug@mail.gmail.com> (raw)
In-Reply-To: <CACT4Y+YH3Z1q9aVTUgTbTPfDp2ao-z=YAjzBGU5HsjGJ-utgeg@mail.gmail.com>

On Tue, Jun 26, 2018 at 7:38 AM Dmitry Vyukov <dvyukov@google.com> wrote:
>
> On Tue, Jun 26, 2018 at 4:16 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> > On Tue, Jun 26, 2018 at 07:54:53PM +0900, Tetsuo Handa wrote:
> >> I hope we can accept NOW either "reviving linux-next.git" or "allowing debug printk()
> >> patches for linux.git". For example, "INFO: task hung in __sb_start_write" got 900
> >> crashes in 81 days but still unable to find a reproducer. Dmitry tried to reproduce
> >> locally with debug printk() patches but not yet successful. I think that testing with
> >> http://lkml.kernel.org/r/f91e1c82-9693-cca3-4ab7-ecd9d9881fb4@i-love.sakura.ne.jp
> >> on linux.git or linux-next.git is the only realistic way for debugging this bug.
> >> More we postpone revival of the linux-next, more syzbot reports we will get...
> >
> > Here's a proposal for adding linux-next back:
> >
> > *) Subsystems or maintainers need to have a way to opt out of getting
> >    spammed with Syzkaller reports that have no reproducer.  More often
> >    than not, they are not actionable, and just annoy the maintainers,
> >    with the net result that they tune out all Syzkaller reports as
> >    noise.
>
> False. You can count yourself. 2/3 are actionable and fixed.
>

Problem is that some if not many of the other 1/3 will be considered
noise, and even some of the 2/3 will be considered noise because they
have already been fixed by the time they are reported. Same problem as
with, say, stable tree merges: People don't see the thousands of bug
fixes inherited with such merges, but they do see the two or three
regressions. Plus, of course, one can not prove that the thousands of
bug fixes did any good because the fixed bugs are not observable
anymore. The only remedy is to try to reduce regressions down to zero
(or, of course, stop using/merging stable releases).

The same applies here: People won't see the good, they only see the
noise. This is pretty much the reason why I all but stopped reporting
build/boot failures on -next. You would have to reduce the noise
almost down to zero for people to stop complaining, and you would have
to be _really_ sure that the problem was not already fixed or reported
elsewhere.

Guenter

> This also makes the following point ungrounded.
>
> > *) Email reports for failures on linux-next that correspond to known
> >    failures on mainline should be suppressed.  Another way of doing
> >    this would be to only report a problem found by a specific
> >    reproducer to the mailing list unless the recipient has agreed to
> >    be spammed by Syskaller noise.
> >
> > And please please please, Syzkaller needs to figure out how to do
> > bisection runs once you have a reproducer.
> >
> >                                                 - Ted