* syzbot dashboard @ 2018-03-21 16:11 Dmitry Vyukov 2018-03-21 20:32 ` Linus Torvalds 0 siblings, 1 reply; 11+ messages in thread From: Dmitry Vyukov @ 2018-03-21 16:11 UTC (permalink / raw) To: LKML; +Cc: syzkaller Hi, syzkaller/syzbot dashboard is now live at: https://syzkaller.appspot.com syzbot is continuous fuzzing/reporting system based on syzkaller fuzzer, currently it tests mainline tree and well as net-next and bpf-next trees. The dashboard shows info about active bugs reported by syzbot. There are ~130 active bugs and I think ~2/3 of them are actionable (still happen and have a reproducer or are simple enough to debug). For active bugs you can see a bunch of additional info (when it started happening, when it last happened, how frequently, on what branches/commits and more crash reports, reproducers and duplicates). For example: https://syzkaller.appspot.com/bug?id=26cb120b31cd24d984fc16da67f50fb375c432a7 The report is now probably lost on LKML, but the crash still happens and have a reproducer. There are bugs in various pars of net subsystem, some in KVM, crypto, keys, perf, tty, block, android drivers, loop device and a long tail in other subsystems. If you are interested in health of a particular subsystem, take a look for relevant bugs. Some of the active bugs can be in fact already fixed. If so it makes sense to tell syzbot about fixes to keep dashboard clean and enable discovery of new similar bugs. syzbot uses Reported-by tags in commits and #syz directives in emails to track these statuses. More details are available at https://github.com/google/syzkaller/blob/master/docs/syzbot.md If syzkaller is not finding bugs in your subsystem, that's most likely because it's not testing it. To test a particular subsystem syzkaller needs some help in the form of interface descriptions (which are not too hard to write, e.g. https://github.com/google/syzkaller/blob/master/sys/linux/ipc.txt). There is also source coverage report available at https://storage.googleapis.com/syzkaller/cover/upstream.html (50MB), so you can check if syzkaller currently covers your code or not and how well. And there is also a list of all fixed bugs so far: https://syzkaller.appspot.com/?fixed=upstream Thanks ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-03-21 16:11 syzbot dashboard Dmitry Vyukov @ 2018-03-21 20:32 ` Linus Torvalds 2018-03-22 9:45 ` Dmitry Vyukov 0 siblings, 1 reply; 11+ messages in thread From: Linus Torvalds @ 2018-03-21 20:32 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: LKML, syzkaller On Wed, Mar 21, 2018 at 9:11 AM, Dmitry Vyukov <dvyukov@google.com> wrote: > > syzkaller/syzbot dashboard is now live at: > https://syzkaller.appspot.com Ok, this may well be the thing that makes syzbot reports useful, if they point to an external report instead of sending an absolutely huge illegible email that nobody can read. That said, I do think it needs some better summarizing still. For example, landing on that front page and then going "Hmm, let's look at that first report", I click on that [upstream] BUG: corrupted list in remove_wait_queue thing, and get to https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6 fine. That page itself doesn't actually tell me really anything at all, though. So I go to the first thing I see, click "log" and I get 6500 lines of basically line noise. Ok, so the real thing is under "report", which actually looks pretty legible. Looking at a few other of those things, I get the same feeling. Can you put one copy of the "report" in the main page for a bug? I assume they are all slightly different, but there must be some commonality to them that you group the syzcaller bugs by, and giving one of those legible reports (with all the nice filename and line information and basic register state) would likely be a good thing. At that point, *if* a report has a reproducer, then sending reminders to people with "this still happens, here's a link to the syzkaller page for this report" might be much better received than the old huge and very-hard-to-read emails. The reminder might well want to have that legible and short "report" in it too, so that people can just look at that to tell "is this relevant for me" particularly if they perhaps already fixed it? (I only looked at a handful of reports, but the ones I looked at all seemed reasonable - maybe some are less so?) Linus ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-03-21 20:32 ` Linus Torvalds @ 2018-03-22 9:45 ` Dmitry Vyukov 2018-03-22 22:20 ` Linus Torvalds 2018-03-23 20:16 ` Eric Biggers 0 siblings, 2 replies; 11+ messages in thread From: Dmitry Vyukov @ 2018-03-22 9:45 UTC (permalink / raw) To: Linus Torvalds; +Cc: LKML, syzkaller On Wed, Mar 21, 2018 at 9:32 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote: > On Wed, Mar 21, 2018 at 9:11 AM, Dmitry Vyukov <dvyukov@google.com> wrote: >> >> syzkaller/syzbot dashboard is now live at: >> https://syzkaller.appspot.com > > Ok, this may well be the thing that makes syzbot reports useful, if > they point to an external report instead of sending an absolutely huge > illegible email that nobody can read. The dashboard was initially meant to be used only for internal monitoring (communication with kernel developers over emails) and I am too close to the project to see obvious stuff. So thanks for the feedback! Do you mean to replace all attachments in syzbot emails with links to web dashboard content? I've heard opinions both ways (including "I am not clicking on any external links, include all content in the email"). So we need to decide. Provided that lots of email clients seem to inline attachments, rather than show them as separate files, I am leaning towards providing just links. Then emails can be short and tidy. Also will solve the problem of "I don't see original attachments in your forwarded email". > That said, I do think it needs some better summarizing still. > > For example, landing on that front page and then going "Hmm, let's > look at that first report", I click on that > > [upstream] BUG: corrupted list in remove_wait_queue > > thing, and get to > > https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6 > > fine. That page itself doesn't actually tell me really anything at all, though. > > So I go to the first thing I see, click "log" and I get 6500 lines of > basically line noise. > > Ok, so the real thing is under "report", which actually looks pretty legible. > > Looking at a few other of those things, I get the same feeling. Can > you put one copy of the "report" in the main page for a bug? I assume > they are all slightly different, but there must be some commonality to > them that you group the syzcaller bugs by, and giving one of those > legible reports (with all the nice filename and line information and > basic register state) would likely be a good thing. This is now done and deployed, the page show a sample crash at the top: https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6 > At that point, *if* a report has a reproducer, then sending reminders > to people with "this still happens, here's a link to the syzkaller > page for this report" might be much better received than the old huge > and very-hard-to-read emails. Now with your blessing we will do reminders :) Not sure what should be periodicity, though. Obviously less frequent than once per day. But probably more frequent than once per release. For the bugs *without* reproducers, now we know precisely that lots of them are fixable. For bugs *with* reproducers fix rate is 76%, but for bugs *without* reproducers it's only down to 66% and more than 100 bugs were fixed without reproducers. So it can make sense to do reminders for them later too. > The reminder might well want to have that legible and short "report" > in it too, so that people can just look at that to tell "is this > relevant for me" particularly if they perhaps already fixed it? > > (I only looked at a handful of reports, but the ones I looked at all > seemed reasonable - maybe some are less so?) There is some, small fraction of unreasonable crashes. Part of them is due to induced crashes after silent memory corruptions. But one can reply with "#syz invalid" to a syzbot email, and such reports will go off the dashboard. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-03-22 9:45 ` Dmitry Vyukov @ 2018-03-22 22:20 ` Linus Torvalds 2018-03-23 20:16 ` Eric Biggers 1 sibling, 0 replies; 11+ messages in thread From: Linus Torvalds @ 2018-03-22 22:20 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: LKML, syzkaller On Thu, Mar 22, 2018 at 2:45 AM, Dmitry Vyukov <dvyukov@google.com> wrote: > > Do you mean to replace all attachments in syzbot emails with links to > web dashboard content? Yes. I think the email should contain just the "report" and then a link to the full information. That would seem to be a good enough summary to be legible, and sufficient for a developer to decide "am I intested, and does it look like I'm the right target". Of course, any web pointer in an email does mean that a lot of spam logic will immediately sit up and take notice, and I don't know how you'll avoid that. And then if/when a developer says "that looks interesting", following the one link gets him/her all the actual information. Honestly, if somebody isn't willing to click on a link, that person clearly is too lazy to bother with the bug in the first place. But at the same time, the email does need to have enough information that you can sanely make that "is it worth it to go click on a link" or not. In fact, it should be sufficient information that for an obvious bug, you don't even really have to click on the link at all. I do think that that "report" dump is a fairly good approximation of just that: enough information that a clear bug doesn't even need any more, but also not so much information that the email report itself becomes illegible or annoying. But auto-generated emails are still likely to be an annoyance for people who get a lot of email (and we all do), so I'd want to make sure that (a) there's some reasonable ratelimiting. No more than once a week for the same report, for example, and some way for a developer to say "I'm not the right person for this report". (b) only send those emails if there really is enough information: a good report with a real WARNING or BUG with the whole file and line number information, _and_ it has been reproduced by the automation (which hopefully means that there is also a good C reproducer). but given that, I think it sounds like a good balance between "not too annoying" and "honestly useful". Will there be cases when automation does the wrong thing and is annoying? Yes. Unquestionably. Nothing is perfect. But make the "good report" succinct and useful enough, and make the false positives rare enough, and I think people will appreciate it. And maybe practice then shows some other case that really annoys people, but that's more an issue of "this may need some tuning" than anything else. Linus ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-03-22 9:45 ` Dmitry Vyukov 2018-03-22 22:20 ` Linus Torvalds @ 2018-03-23 20:16 ` Eric Biggers 2018-03-23 20:50 ` Linus Torvalds 1 sibling, 1 reply; 11+ messages in thread From: Eric Biggers @ 2018-03-23 20:16 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: Linus Torvalds, LKML, syzkaller On Thu, Mar 22, 2018 at 10:45:43AM +0100, 'Dmitry Vyukov' via syzkaller wrote: > On Wed, Mar 21, 2018 at 9:32 PM, Linus Torvalds > <torvalds@linux-foundation.org> wrote: > > On Wed, Mar 21, 2018 at 9:11 AM, Dmitry Vyukov <dvyukov@google.com> wrote: > >> > >> syzkaller/syzbot dashboard is now live at: > >> https://syzkaller.appspot.com > > > > Ok, this may well be the thing that makes syzbot reports useful, if > > they point to an external report instead of sending an absolutely huge > > illegible email that nobody can read. > > The dashboard was initially meant to be used only for internal > monitoring (communication with kernel developers over emails) and I am > too close to the project to see obvious stuff. So thanks for the > feedback! > > Do you mean to replace all attachments in syzbot emails with links to > web dashboard content? > I've heard opinions both ways (including "I am not clicking on any > external links, include all content in the email"). So we need to > decide. Provided that lots of email clients seem to inline > attachments, rather than show them as separate files, I am leaning > towards providing just links. Then emails can be short and tidy. Also > will solve the problem of "I don't see original attachments in your > forwarded email". > A compromise could be to remove the raw.log from the emails, and send the config in 'defconfig' format rather than the full config; that trims it down from 5256 lines to 925 currently. That would get rid of most of what is bloating the size of the emails, while still keeping the (very useful) C and syzkaller reproducers. Dashboard would still have everything, of course. Eric ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-03-23 20:16 ` Eric Biggers @ 2018-03-23 20:50 ` Linus Torvalds 2018-03-26 8:46 ` Dmitry Vyukov 0 siblings, 1 reply; 11+ messages in thread From: Linus Torvalds @ 2018-03-23 20:50 UTC (permalink / raw) To: Eric Biggers; +Cc: Dmitry Vyukov, LKML, syzkaller On Fri, Mar 23, 2018 at 1:16 PM, Eric Biggers <ebiggers3@gmail.com> wrote: > > A compromise could be to remove the raw.log from the emails, and send the config > in 'defconfig' format rather than the full config I really don't think the config is very legible, and isn't something that people will look at to judge whether it's intrresting or not. It's still "many hundreds of lines of line noise" And it shouldn't really be part of the consideration whether a bug is really worth looking at or not anyway, although *after* you've found the bug as a developer, you may decide "ok, that configuration isn't valid, so I should just disallow it in Kconfig". But that's still a bugfix. So I think the config can stay purely on the web (for when you go "so how do I *reproduce* this?") along with the actual reproducer and the full logs. I'd rather make the emails be small and legible, and contain the minimum amount of information. Because I really do *not* need a few hundred emails from some automated source that each have a roughly thousand-line config attachements. An email with 50 lines of THE ACTUAL DECODED OOPS? Sure. That's critical information. That doesn't make me go "why does this stupid bot spew garbage at me"? Linus ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-03-23 20:50 ` Linus Torvalds @ 2018-03-26 8:46 ` Dmitry Vyukov 2018-04-14 18:25 ` Linus Torvalds 0 siblings, 1 reply; 11+ messages in thread From: Dmitry Vyukov @ 2018-03-26 8:46 UTC (permalink / raw) To: Linus Torvalds; +Cc: Eric Biggers, LKML, syzkaller On Fri, Mar 23, 2018 at 9:50 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote: > On Fri, Mar 23, 2018 at 1:16 PM, Eric Biggers <ebiggers3@gmail.com> wrote: >> >> A compromise could be to remove the raw.log from the emails, and send the config >> in 'defconfig' format rather than the full config > > I really don't think the config is very legible, and isn't something > that people will look at to judge whether it's intrresting or not. > It's still "many hundreds of lines of line noise" > > And it shouldn't really be part of the consideration whether a bug is > really worth looking at or not anyway, although *after* you've found > the bug as a developer, you may decide "ok, that configuration isn't > valid, so I should just disallow it in Kconfig". But that's still a > bugfix. > > So I think the config can stay purely on the web (for when you go "so > how do I *reproduce* this?") along with the actual reproducer and the > full logs. > > I'd rather make the emails be small and legible, and contain the > minimum amount of information. > > Because I really do *not* need a few hundred emails from some > automated source that each have a roughly thousand-line config > attachements. > > An email with 50 lines of THE ACTUAL DECODED OOPS? Sure. That's > critical information. That doesn't make me go "why does this stupid > bot spew garbage at me"? I've switched emails to links instead of attachments, here are few recent examples: https://lkml.org/lkml/2018/3/25/31 https://lkml.org/lkml/2018/3/25/256 https://lkml.org/lkml/2018/3/25/257 Filed https://github.com/google/syzkaller/issues/550 for periodic pings and noted your requirements. In particular, we will need something for "I'm not the right person for this report", and some plan for gradual rollout. It will take some time to implement. Meanwhile developers are welcome to look at the dashboard https://syzkaller.appspot.com/ Thanks again ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-03-26 8:46 ` Dmitry Vyukov @ 2018-04-14 18:25 ` Linus Torvalds 2018-04-16 11:15 ` Dmitry Vyukov 0 siblings, 1 reply; 11+ messages in thread From: Linus Torvalds @ 2018-04-14 18:25 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: Eric Biggers, LKML, syzkaller Coming back to this now that the merge window is almost over ] On Mon, Mar 26, 2018 at 1:46 AM, Dmitry Vyukov <dvyukov@google.com> wrote: > > I've switched emails to links instead of attachments, here are few > recent examples: > https://lkml.org/lkml/2018/3/25/31 > https://lkml.org/lkml/2018/3/25/256 > https://lkml.org/lkml/2018/3/25/257 Looks good to me. I notice that only the last one got any replies, though. I wonder if some people auto-ignore the new reports because of having been burned by the previous "huge illegible emails" issue. I do see syzbot fixes in rdma, though, just not for that cma_listen_on_all issue. So maybe that bug is nastier. Linus ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-04-14 18:25 ` Linus Torvalds @ 2018-04-16 11:15 ` Dmitry Vyukov 2018-04-16 15:55 ` Randy Dunlap 0 siblings, 1 reply; 11+ messages in thread From: Dmitry Vyukov @ 2018-04-16 11:15 UTC (permalink / raw) To: Linus Torvalds; +Cc: Eric Biggers, LKML, syzkaller On Sat, Apr 14, 2018 at 8:25 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote: > Coming back to this now that the merge window is almost over ] > > On Mon, Mar 26, 2018 at 1:46 AM, Dmitry Vyukov <dvyukov@google.com> wrote: >> >> I've switched emails to links instead of attachments, here are few >> recent examples: >> https://lkml.org/lkml/2018/3/25/31 >> https://lkml.org/lkml/2018/3/25/256 >> https://lkml.org/lkml/2018/3/25/257 > > Looks good to me. I notice that only the last one got any replies, though. > > I wonder if some people auto-ignore the new reports because of having > been burned by the previous "huge illegible emails" issue. > > I do see syzbot fixes in rdma, though, just not for that > cma_listen_on_all issue. So maybe that bug is nastier. "possible deadlock in handle_rx" is now marked as fixed with "vhost_net: add missing lock nesting notation". "KASAN: use-after-free Read in pppol2tp_connect (3)" is marked as fixed with "l2tp: fix races in tunnel creation". "KASAN: use-after-free Read in __list_add_valid (5)" is still open. Though, there are still 182 open bugs on dashboard: https://syzkaller.appspot.com/ 80+ have reproducers and happened last time recently enough. Few dozens without reproducers, but as history shows ~2/3 of them are fixable. Part of the remaining ones is probably fixed, but nobody told syzbot about the fixes. Part is probably still unfixed but happen episodically. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-04-16 11:15 ` Dmitry Vyukov @ 2018-04-16 15:55 ` Randy Dunlap 2018-04-18 18:33 ` Dmitry Vyukov 0 siblings, 1 reply; 11+ messages in thread From: Randy Dunlap @ 2018-04-16 15:55 UTC (permalink / raw) To: Dmitry Vyukov, Linus Torvalds; +Cc: Eric Biggers, LKML, syzkaller On 04/16/2018 04:15 AM, Dmitry Vyukov wrote: > On Sat, Apr 14, 2018 at 8:25 PM, Linus Torvalds > <torvalds@linux-foundation.org> wrote: >> Coming back to this now that the merge window is almost over ] >> >> On Mon, Mar 26, 2018 at 1:46 AM, Dmitry Vyukov <dvyukov@google.com> wrote: >>> >>> I've switched emails to links instead of attachments, here are few >>> recent examples: >>> https://lkml.org/lkml/2018/3/25/31 >>> https://lkml.org/lkml/2018/3/25/256 >>> https://lkml.org/lkml/2018/3/25/257 >> >> Looks good to me. I notice that only the last one got any replies, though. >> >> I wonder if some people auto-ignore the new reports because of having >> been burned by the previous "huge illegible emails" issue. >> >> I do see syzbot fixes in rdma, though, just not for that >> cma_listen_on_all issue. So maybe that bug is nastier. > > "possible deadlock in handle_rx" is now marked as fixed with > "vhost_net: add missing lock nesting notation". > "KASAN: use-after-free Read in pppol2tp_connect (3)" is marked as > fixed with "l2tp: fix races in tunnel creation". > "KASAN: use-after-free Read in __list_add_valid (5)" is still open. > > Though, there are still 182 open bugs on dashboard: > https://syzkaller.appspot.com/ > > 80+ have reproducers and happened last time recently enough. Few > dozens without reproducers, but as history shows ~2/3 of them are > fixable. Part of the remaining ones is probably fixed, but nobody told > syzbot about the fixes. Part is probably still unfixed but happen > episodically. > https://syzkaller.appspot.com/bug?id=0627d4551fdc39bf1ef5d82cd9eef587047f7718 is fixed by 9ad553abe66f ("fs/reiserfs/journal.c: add missing resierfs_warning() arg"), which does contain the string: Reported-by: <syzbot+6bd77b88c1977c03f584@syzkaller.appspotmail.com> Is one of those hex strings incorrect? -- ~Randy ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: syzbot dashboard 2018-04-16 15:55 ` Randy Dunlap @ 2018-04-18 18:33 ` Dmitry Vyukov 0 siblings, 0 replies; 11+ messages in thread From: Dmitry Vyukov @ 2018-04-18 18:33 UTC (permalink / raw) To: Randy Dunlap; +Cc: Linus Torvalds, Eric Biggers, LKML, syzkaller On Mon, Apr 16, 2018 at 5:55 PM, Randy Dunlap <rdunlap@infradead.org> wrote: > On 04/16/2018 04:15 AM, Dmitry Vyukov wrote: >> On Sat, Apr 14, 2018 at 8:25 PM, Linus Torvalds >> <torvalds@linux-foundation.org> wrote: >>> Coming back to this now that the merge window is almost over ] >>> >>> On Mon, Mar 26, 2018 at 1:46 AM, Dmitry Vyukov <dvyukov@google.com> wrote: >>>> >>>> I've switched emails to links instead of attachments, here are few >>>> recent examples: >>>> https://lkml.org/lkml/2018/3/25/31 >>>> https://lkml.org/lkml/2018/3/25/256 >>>> https://lkml.org/lkml/2018/3/25/257 >>> >>> Looks good to me. I notice that only the last one got any replies, though. >>> >>> I wonder if some people auto-ignore the new reports because of having >>> been burned by the previous "huge illegible emails" issue. >>> >>> I do see syzbot fixes in rdma, though, just not for that >>> cma_listen_on_all issue. So maybe that bug is nastier. >> >> "possible deadlock in handle_rx" is now marked as fixed with >> "vhost_net: add missing lock nesting notation". >> "KASAN: use-after-free Read in pppol2tp_connect (3)" is marked as >> fixed with "l2tp: fix races in tunnel creation". >> "KASAN: use-after-free Read in __list_add_valid (5)" is still open. >> >> Though, there are still 182 open bugs on dashboard: >> https://syzkaller.appspot.com/ >> >> 80+ have reproducers and happened last time recently enough. Few >> dozens without reproducers, but as history shows ~2/3 of them are >> fixable. Part of the remaining ones is probably fixed, but nobody told >> syzbot about the fixes. Part is probably still unfixed but happen >> episodically. >> > > https://syzkaller.appspot.com/bug?id=0627d4551fdc39bf1ef5d82cd9eef587047f7718 > is fixed by 9ad553abe66f ("fs/reiserfs/journal.c: add missing resierfs_warning() arg"), > > which does contain the string: > Reported-by: <syzbot+6bd77b88c1977c03f584@syzkaller.appspotmail.com> > > Is one of those hex strings incorrect? Hi Randy, They are correct and no action is needed. It's just that the commit did not reach any of the trees that syzbot polls, so it did not know about them yet. You can see that now syzbot has picked the commit. ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2018-04-18 18:33 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-03-21 16:11 syzbot dashboard Dmitry Vyukov 2018-03-21 20:32 ` Linus Torvalds 2018-03-22 9:45 ` Dmitry Vyukov 2018-03-22 22:20 ` Linus Torvalds 2018-03-23 20:16 ` Eric Biggers 2018-03-23 20:50 ` Linus Torvalds 2018-03-26 8:46 ` Dmitry Vyukov 2018-04-14 18:25 ` Linus Torvalds 2018-04-16 11:15 ` Dmitry Vyukov 2018-04-16 15:55 ` Randy Dunlap 2018-04-18 18:33 ` Dmitry Vyukov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).