* LKML admins (syzbot emails are not delivered) @ 2018-01-04 9:09 Dmitry Vyukov 2018-01-04 9:23 ` Greg Kroah-Hartman 2018-01-04 9:25 ` Pavel Machek 0 siblings, 2 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-04 9:09 UTC (permalink / raw) To: LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds; +Cc: syzkaller Hello, Some of syzbot emails don't appear on LKML mailing lists, while they were mailed as any other emails. Here are few examples: "KASAN: use-after-free Read in rds_tcp_dev_event" https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ "general protection fault in __wake_up_common" https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ Does anybody know how to get in contact with real people behind LKML and/or bugzilla? I am trying to understand why this happens, but failed so far (it does not do any obviously prohibited stuff, and replies to these emails are delivered). I tried to use autoanswer@vger.kernel.org (which is referenced from http://vger.kernel.org/majordomo-info.html), but it now always return: 554 5.0.0 Hi [209.85.192.182], unresolvable address: <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org I failed to find any admin email referenced anywhere. On a related note, I also tried to contact bugzilla admins via rt.linuxfoundation.org. But there is complete silence. Does anybody know how to get it touch with these people? Thanks in advance ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:09 LKML admins (syzbot emails are not delivered) Dmitry Vyukov @ 2018-01-04 9:23 ` Greg Kroah-Hartman 2018-01-04 9:56 ` Ozgur 2018-01-04 11:03 ` Dmitry Vyukov 2018-01-04 9:25 ` Pavel Machek 1 sibling, 2 replies; 53+ messages in thread From: Greg Kroah-Hartman @ 2018-01-04 9:23 UTC (permalink / raw) To: Dmitry Vyukov; +Cc: LKML, Andrew Morton, Linus Torvalds, syzkaller On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote: > Hello, > > Some of syzbot emails don't appear on LKML mailing lists, while they > were mailed as any other emails. Here are few examples: > > "KASAN: use-after-free Read in rds_tcp_dev_event" > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > > "general protection fault in __wake_up_common" > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > > Does anybody know how to get in contact with real people behind LKML > and/or bugzilla? > > I am trying to understand why this happens, but failed so far (it does > not do any obviously prohibited stuff, and replies to these emails are > delivered). You should get a bounce notice from vger if the email is being rejected. If not, you might be on the spam filter list, which is listed on vger, did you check that? > I tried to use autoanswer@vger.kernel.org (which is > referenced from http://vger.kernel.org/majordomo-info.html), but it > now always return: > > 554 5.0.0 Hi [209.85.192.182], unresolvable address: > <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org autoanswer is not for admin requests. > I failed to find any admin email referenced anywhere. Look a bit harder, like at the bottom of this page: http://vger.kernel.org/majordomo-info.html :) > On a related note, I also tried to contact bugzilla admins via > rt.linuxfoundation.org. But there is complete silence. Does anybody > know how to get it touch with these people? Did you get an answer back from the rt system? If not, it did not go through, and the help address might have changed. I can dig it up if so... thanks, greg k-h ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:23 ` Greg Kroah-Hartman @ 2018-01-04 9:56 ` Ozgur 2018-01-04 15:31 ` David Miller 2018-01-04 11:03 ` Dmitry Vyukov 1 sibling, 1 reply; 53+ messages in thread From: Ozgur @ 2018-01-04 9:56 UTC (permalink / raw) To: Greg Kroah-Hartman, Dmitry Vyukov Cc: LKML, Andrew Morton, Linus Torvalds, syzkaller, davem 04.01.2018, 12:23, "Greg Kroah-Hartman" <gregkh@linuxfoundation.org>: > On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote: >> Hello, >> >> Some of syzbot emails don't appear on LKML mailing lists, while they >> were mailed as any other emails. Here are few examples: >> >> "KASAN: use-after-free Read in rds_tcp_dev_event" >> https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ >> >> "general protection fault in __wake_up_common" >> https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ >> >> Does anybody know how to get in contact with real people behind LKML >> and/or bugzilla? >> >> I am trying to understand why this happens, but failed so far (it does >> not do any obviously prohibited stuff, and replies to these emails are >> delivered). > > You should get a bounce notice from vger if the email is being rejected. > If not, you might be on the spam filter list, which is listed on vger, > did you check that? > >> I tried to use autoanswer@vger.kernel.org (which is >> referenced from http://vger.kernel.org/majordomo-info.html), but it >> now always return: >> >> 554 5.0.0 Hi [209.85.192.182], unresolvable address: >> <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org Hello, autoanswer is just automated reply address that the lmkl system works and your email arrives. LKML e-mail list implemented SPF, DKIM and DMARC and I think some domains were forbid. For example yandex.ru or mail.ru I think should add David (Miller), I added. > autoanswer is not for admin requests. > >> I failed to find any admin email referenced anywhere. > > Look a bit harder, like at the bottom of this page: > http://vger.kernel.org/majordomo-info.html > > :) > >> On a related note, I also tried to contact bugzilla admins via >> rt.linuxfoundation.org. But there is complete silence. Does anybody >> know how to get it touch with these people? And please check MX: http://vger.kernel.org/mxverify.html Ozgur > Did you get an answer back from the rt system? If not, it did not go > through, and the help address might have changed. I can dig it up if > so... > > thanks, > > greg k-h ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:56 ` Ozgur @ 2018-01-04 15:31 ` David Miller 2018-01-04 19:11 ` Ozgur 0 siblings, 1 reply; 53+ messages in thread From: David Miller @ 2018-01-04 15:31 UTC (permalink / raw) To: ozgur; +Cc: gregkh, dvyukov, linux-kernel, akpm, torvalds, syzkaller From: Ozgur <ozgur@goosey.org> Date: Thu, 04 Jan 2018 12:56:56 +0300 > autoanswer is just automated reply address that the lmkl system works and your email arrives. > LKML e-mail list implemented SPF, DKIM and DMARC and I think some domains were forbid. > For example yandex.ru or mail.ru If I am given specific examples of postings that don't arrive from this point forward I can look into them. But I have to be alerted very quickly after the face rather than a day or so later. I also only process vger.kernel.org bounces and postmaster email about once, maybe twice per day. So please keep this in mind in your expectations. Thank you. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 15:31 ` David Miller @ 2018-01-04 19:11 ` Ozgur 0 siblings, 0 replies; 53+ messages in thread From: Ozgur @ 2018-01-04 19:11 UTC (permalink / raw) To: David Miller; +Cc: gregkh, dvyukov, linux-kernel, akpm, torvalds, syzkaller 04.01.2018, 18:31, "David Miller" <davem@davemloft.net>: > From: Ozgur <ozgur@goosey.org> > Date: Thu, 04 Jan 2018 12:56:56 +0300 Dvyukov, I think you will set a bot sensitive and syzbot send e-mail that it doesn't disturb list members :) David is sometimes nervous. Ozgur >> autoanswer is just automated reply address that the lmkl system works and your email arrives. >> LKML e-mail list implemented SPF, DKIM and DMARC and I think some domains were forbid. >> For example yandex.ru or mail.ru > > If I am given specific examples of postings that don't arrive from this point forward > I can look into them. > > But I have to be alerted very quickly after the face rather than a day or so later. > > I also only process vger.kernel.org bounces and postmaster email about once, maybe > twice per day. So please keep this in mind in your expectations. > > Thank you. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:23 ` Greg Kroah-Hartman 2018-01-04 9:56 ` Ozgur @ 2018-01-04 11:03 ` Dmitry Vyukov 2018-01-04 11:04 ` Dmitry Vyukov 1 sibling, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-04 11:03 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: LKML, Andrew Morton, Linus Torvalds, syzkaller, David Miller [-- Attachment #1: Type: text/plain, Size: 2871 bytes --] On Thu, Jan 4, 2018 at 10:23 AM, Greg Kroah-Hartman < gregkh@linuxfoundation.org> wrote: > On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote: > > Hello, > > > > Some of syzbot emails don't appear on LKML mailing lists, while they > > were mailed as any other emails. Here are few examples: > > > > "KASAN: use-after-free Read in rds_tcp_dev_event" > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > > > > "general protection fault in __wake_up_common" > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > > > > Does anybody know how to get in contact with real people behind LKML > > and/or bugzilla? > > > > I am trying to understand why this happens, but failed so far (it does > > not do any obviously prohibited stuff, and replies to these emails are > > delivered). > > You should get a bounce notice from vger if the email is being rejected. > The problem is that it's not _me_, it's a computer program which uses some mail delivery system which I don't have full visibility into. I don't know if it even gets bounce emails (as far as I understand it's not LKML that generates them, LKML SMTP server just returns some error code and then it's a responsibility of somebody else to represent it by a reply email). If the only way to probe the behavior is to send actual emails to LKML (which have high chances to be actually delivered to all subscribers), it makes testing somewhat problematic. > If not, you might be on the spam filter list, which is listed on vger, > did you check that? I've looked at it manually and did not find any matches. Again, a reply quoting full original text was delivered. > > I tried to use autoanswer@vger.kernel.org (which is > > referenced from http://vger.kernel.org/majordomo-info.html), but it > > now always return: > > > > 554 5.0.0 Hi [209.85.192.182], unresolvable address: > > <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org > > autoanswer is not for admin requests. > Yes, I understand. But that would be a perfect way to probe behavior and look at bounce messages. The problem is that it does not seem to work at all. > > I failed to find any admin email referenced anywhere. > > Look a bit harder, like at the bottom of this page: > http://vger.kernel.org/majordomo-info.html > > :) I will try that one. Thanks. > > > On a related note, I also tried to contact bugzilla admins via > > rt.linuxfoundation.org. But there is complete silence. Does anybody > > know how to get it touch with these people? > > Did you get an answer back from the rt system? If not, it did not go > through, and the help address might have changed. I can dig it up if > so... > Yes, the ticket was filed as: https://rt.linuxfoundation.org/SelfService/Display.html?id=50245 (so emailing helpdesk@kernel.org part worked fine) [-- Attachment #2: Type: text/html, Size: 5631 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:03 ` Dmitry Vyukov @ 2018-01-04 11:04 ` Dmitry Vyukov 2018-01-04 11:20 ` Greg Kroah-Hartman 2018-01-04 23:50 ` Theodore Ts'o 0 siblings, 2 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-04 11:04 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: LKML, Andrew Morton, Linus Torvalds, syzkaller, David Miller On Thu, Jan 4, 2018 at 10:23 AM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > > On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote: > > Hello, > > > > Some of syzbot emails don't appear on LKML mailing lists, while they > > were mailed as any other emails. Here are few examples: > > > > "KASAN: use-after-free Read in rds_tcp_dev_event" > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > > > > "general protection fault in __wake_up_common" > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > > > > Does anybody know how to get in contact with real people behind LKML > > and/or bugzilla? > > > > I am trying to understand why this happens, but failed so far (it does > > not do any obviously prohibited stuff, and replies to these emails are > > delivered). > > You should get a bounce notice from vger if the email is being rejected. The problem is that it's not _me_, it's a computer program which uses some mail delivery system which I don't have full visibility into. I don't know if it even gets bounce emails (as far as I understand it's not LKML that generates them, LKML SMTP server just returns some error code and then it's a responsibility of somebody else to represent it by a reply email). If the only way to probe the behavior is to send actual emails to LKML (which have high chances to be actually delivered to all subscribers), it makes testing somewhat problematic. > > If not, you might be on the spam filter list, which is listed on vger, > did you check that? I've looked at it manually and did not find any matches. Again, a reply quoting full original text was delivered. > > > I tried to use autoanswer@vger.kernel.org (which is > > referenced from http://vger.kernel.org/majordomo-info.html), but it > > now always return: > > > > 554 5.0.0 Hi [209.85.192.182], unresolvable address: > > <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org > > autoanswer is not for admin requests. Yes, I understand. But that would be a perfect way to probe behavior and look at bounce messages. The problem is that it does not seem to work at all. > > > I failed to find any admin email referenced anywhere. > > Look a bit harder, like at the bottom of this page: > http://vger.kernel.org/majordomo-info.html > > :) I will try that one. Thanks. > > > > On a related note, I also tried to contact bugzilla admins via > > rt.linuxfoundation.org. But there is complete silence. Does anybody > > know how to get it touch with these people? > > Did you get an answer back from the rt system? If not, it did not go > through, and the help address might have changed. I can dig it up if > so... Yes, the ticket was filed as: https://rt.linuxfoundation.org/SelfService/Display.html?id=50245 (so emailing helpdesk@kernel.org part worked fine) ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:04 ` Dmitry Vyukov @ 2018-01-04 11:20 ` Greg Kroah-Hartman 2018-01-04 15:35 ` David Miller 2018-03-01 16:22 ` Dmitry Vyukov 2018-01-04 23:50 ` Theodore Ts'o 1 sibling, 2 replies; 53+ messages in thread From: Greg Kroah-Hartman @ 2018-01-04 11:20 UTC (permalink / raw) To: Dmitry Vyukov Cc: LKML, Andrew Morton, Linus Torvalds, syzkaller, David Miller On Thu, Jan 04, 2018 at 12:04:34PM +0100, Dmitry Vyukov wrote: > On Thu, Jan 4, 2018 at 10:23 AM, Greg Kroah-Hartman > <gregkh@linuxfoundation.org> wrote: > > > > On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote: > > > Hello, > > > > > > Some of syzbot emails don't appear on LKML mailing lists, while they > > > were mailed as any other emails. Here are few examples: > > > > > > "KASAN: use-after-free Read in rds_tcp_dev_event" > > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > > > > > > "general protection fault in __wake_up_common" > > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > > > > > > Does anybody know how to get in contact with real people behind LKML > > > and/or bugzilla? > > > > > > I am trying to understand why this happens, but failed so far (it does > > > not do any obviously prohibited stuff, and replies to these emails are > > > delivered). > > > > You should get a bounce notice from vger if the email is being rejected. > > > The problem is that it's not _me_, it's a computer program which uses > some mail delivery system which I don't have full visibility into. But you, or someone, should have access to that email address to see any responses sent to it. If no one can, then it's just a spam bot and if I were vger's postmaster, I would blacklist it as well :) Good luck! greg k-h ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:20 ` Greg Kroah-Hartman @ 2018-01-04 15:35 ` David Miller 2018-01-15 9:43 ` Dmitry Vyukov 2018-03-01 16:22 ` Dmitry Vyukov 1 sibling, 1 reply; 53+ messages in thread From: David Miller @ 2018-01-04 15:35 UTC (permalink / raw) To: gregkh; +Cc: dvyukov, linux-kernel, akpm, torvalds, syzkaller From: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Date: Thu, 4 Jan 2018 12:20:28 +0100 > If no one can, then it's just a spam bot and if I were vger's > postmaster, I would blacklist it as well :) +1 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 15:35 ` David Miller @ 2018-01-15 9:43 ` Dmitry Vyukov 0 siblings, 0 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-15 9:43 UTC (permalink / raw) To: David Miller Cc: Greg Kroah-Hartman, LKML, Andrew Morton, Linus Torvalds, syzkaller On Thu, Jan 4, 2018 at 4:35 PM, David Miller <davem@davemloft.net> wrote: > From: Greg Kroah-Hartman <gregkh@linuxfoundation.org> > Date: Thu, 4 Jan 2018 12:20:28 +0100 > >> If no one can, then it's just a spam bot and if I were vger's >> postmaster, I would blacklist it as well :) > > +1 Just to make it clear, humans were reading all normal communication related to bugs from day 1. It was just bounce emails. Now we are receiving bounce emails as well. So this is resolved. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:20 ` Greg Kroah-Hartman 2018-01-04 15:35 ` David Miller @ 2018-03-01 16:22 ` Dmitry Vyukov 2018-03-08 20:03 ` David Miller 1 sibling, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-03-01 16:22 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: LKML, Andrew Morton, Linus Torvalds, syzkaller, David Miller On Thu, Jan 4, 2018 at 12:20 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: >> > On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote: >> > > Hello, >> > > >> > > Some of syzbot emails don't appear on LKML mailing lists, while they >> > > were mailed as any other emails. Here are few examples: >> > > >> > > "KASAN: use-after-free Read in rds_tcp_dev_event" >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ >> > > >> > > "general protection fault in __wake_up_common" >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ >> > > >> > > Does anybody know how to get in contact with real people behind LKML >> > > and/or bugzilla? >> > > >> > > I am trying to understand why this happens, but failed so far (it does >> > > not do any obviously prohibited stuff, and replies to these emails are >> > > delivered). >> > >> > You should get a bounce notice from vger if the email is being rejected. >> >> >> The problem is that it's not _me_, it's a computer program which uses >> some mail delivery system which I don't have full visibility into. > > But you, or someone, should have access to that email address to see any > responses sent to it. If no one can, then it's just a spam bot and if I > were vger's postmaster, I would blacklist it as well :) > > Good luck! > > greg k-h The problem strikes back. Again people complain that a email does not appear on LKML: https://www.spinics.net/lists/kernel/msg2735905.html Though, it appeared on syzkaller-bugs group: https://groups.google.com/d/msg/syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ syzbot now receives and logs bounce notifications, but there were none... FWIW I've filed support ticket https://rt.linuxfoundation.org/SelfService/Display.html?id=53105 ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-03-01 16:22 ` Dmitry Vyukov @ 2018-03-08 20:03 ` David Miller 2018-03-12 8:56 ` Dmitry Vyukov 0 siblings, 1 reply; 53+ messages in thread From: David Miller @ 2018-03-08 20:03 UTC (permalink / raw) To: dvyukov; +Cc: gregkh, linux-kernel, akpm, torvalds, syzkaller From: Dmitry Vyukov <dvyukov@google.com> Date: Thu, 1 Mar 2018 17:22:54 +0100 > The problem strikes back. > Again people complain that a email does not appear on LKML: > https://www.spinics.net/lists/kernel/msg2735905.html > Though, it appeared on syzkaller-bugs group: > https://groups.google.com/d/msg/syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ > syzbot now receives and logs bounce notifications, but there were none... The problem is that the name 'kernelci.orgbot' has a special character in it, namely "." And when such special characters occur in email headers, the string in question must be surrounded by double quotes. People have to do this when they use my name "David S. Miller" as well, for example. So I think if you put double quotes around kernelci.orgbot when it is provided in email headers, the problem will go away. Please also make sure that multiple From: headers are not being generated or anything like that. Thanks. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-03-08 20:03 ` David Miller @ 2018-03-12 8:56 ` Dmitry Vyukov 2018-03-12 14:03 ` David Miller 0 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-03-12 8:56 UTC (permalink / raw) To: David Miller Cc: Greg Kroah-Hartman, LKML, Andrew Morton, Linus Torvalds, syzkaller On Thu, Mar 8, 2018 at 9:03 PM, David Miller <davem@davemloft.net> wrote: > From: Dmitry Vyukov <dvyukov@google.com> > Date: Thu, 1 Mar 2018 17:22:54 +0100 > >> The problem strikes back. >> Again people complain that a email does not appear on LKML: >> https://www.spinics.net/lists/kernel/msg2735905.html >> Though, it appeared on syzkaller-bugs group: >> https://groups.google.com/d/msg/syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ >> https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ >> syzbot now receives and logs bounce notifications, but there were none... > > The problem is that the name 'kernelci.orgbot' has a special character > in it, namely "." > > And when such special characters occur in email headers, the string > in question must be surrounded by double quotes. > > People have to do this when they use my name "David S. Miller" as > well, for example. > > So I think if you put double quotes around kernelci.orgbot when it > is provided in email headers, the problem will go away. > > Please also make sure that multiple From: headers are not being > generated or anything like that. Thanks for looking into this, David! But I don't see kernelci.orgbot referenced anywhere in the emails. E.g. in the one that was linked above: https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ Which one do you mean? syzbot takes emails from get_maintainers.pl, I guess kernelci.orgbot is not usually there. And syzbot does not includes names, just email addresses. Could a dot in email address produce the same result? E.g. To: foo.bar@something.com ? I see such addresses in some of the emails (but not in the one referenced above). Is it possible to revive autoanswer@vger.kernel.org? It would hopefully allow us to debug this without bothering you. Possibly related, when I am searching for "syzkaller" I am finding entries like the following titled "This message generated a parse failure. Raw output follows here. Please use 'back' to navigate.". https://lkml.org/lkml/2018/3/10/73 https://lkml.org/lkml/2018/2/27/581 https://lkml.org/lkml/2018/2/22/774 Could this "parse failure" explain the undelivered emails? How can I figure out what exactly caused the failure? Thanks again ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-03-12 8:56 ` Dmitry Vyukov @ 2018-03-12 14:03 ` David Miller 0 siblings, 0 replies; 53+ messages in thread From: David Miller @ 2018-03-12 14:03 UTC (permalink / raw) To: dvyukov; +Cc: gregkh, linux-kernel, akpm, torvalds, syzkaller From: Dmitry Vyukov <dvyukov@google.com> Date: Mon, 12 Mar 2018 09:56:42 +0100 > But I don't see kernelci.orgbot referenced anywhere in the emails. > E.g. in the one that was linked above: > https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ I'm not referencing that one above, I'm simply reporting what I've seen in the vger bounce logs. > Could a dot in email address produce the same result? E.g. > To: foo.bar@something.com No. > Is it possible to revive autoanswer@vger.kernel.org? It would > hopefully allow us to debug this without bothering you. Sorry, no engineering resources will be put into the current setup as we prepare a new system based upon mailman. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:04 ` Dmitry Vyukov 2018-01-04 11:20 ` Greg Kroah-Hartman @ 2018-01-04 23:50 ` Theodore Ts'o 2018-01-08 12:01 ` Dmitry Vyukov 2018-01-08 12:11 ` Dmitry Vyukov 1 sibling, 2 replies; 53+ messages in thread From: Theodore Ts'o @ 2018-01-04 23:50 UTC (permalink / raw) To: Dmitry Vyukov Cc: Greg Kroah-Hartman, LKML, Andrew Morton, Linus Torvalds, syzkaller, David Miller On Thu, Jan 04, 2018 at 12:04:34PM +0100, Dmitry Vyukov wrote: > > The problem is that it's not _me_, it's a computer program which uses > some mail delivery system which I don't have full visibility into. I > don't know if it even gets bounce emails (as far as I understand it's > not LKML that generates them, LKML SMTP server just returns some error > code and then it's a responsibility of somebody else to represent it > by a reply email). If the only way to probe the behavior is to send > actual emails to LKML (which have high chances to be actually > delivered to all subscribers), it makes testing somewhat problematic. It looks like you're using the App Engine Mail API. You can configure it to get bounce e-mails[1]. From what I can tell looking at the mail headers, the mail gets sent via an intermediate SMTP host, such as mail-it0-f69.google.com before it is delievered to vger. So if vger's mailer bounces it via an SMTP error, it would be mail-it0-f69.google.com's responsibility to generate a bounce message, which you then should be able to catch. [1] https://cloud.google.com/appengine/docs/standard/go/mail/bounce You should be able to test this by having your app-engine send a message to "invalid-address@vger.kernel.org". I've verified that this will cause an immediate SMTP error: 554 5.0.0 Hi [74.207.234.97], unresolvable address: <invalid-address@vger.kernel.org>; nosuchuser; invalid-address@vger.kernel.org If it doesn't, you should file an internal bug report since that's clearly a bug in the App Engine's mail infrastructure. If you can't get satisfaction that way, my recommendation would be to set up an Linode server (you can't use GCE because GCE blocks outgoing SMTP connections) to be your mail gateway, and send the notifications from a domain such as syzkaller.org, where you have full control of your mail infrastructure, and you don't have to deal with nonsense such as DMARC. It also seems to me, looking at other complaints on this thread, that there is the opportunity for the syzbot to do much more. For example, you can see if it repro's on the last released mainline kernel (such as 4.14) and if so, have the syzbot automatically do a bisection search, so you can make sure the report goes to the best set of developers to fix it, a pointer to the guilty commit. Cheers, - Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 23:50 ` Theodore Ts'o @ 2018-01-08 12:01 ` Dmitry Vyukov 2018-01-08 12:11 ` Dmitry Vyukov 1 sibling, 0 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-08 12:01 UTC (permalink / raw) To: Theodore Ts'o, Dmitry Vyukov, Greg Kroah-Hartman, LKML, Andrew Morton, Linus Torvalds, syzkaller, David Miller On Fri, Jan 5, 2018 at 12:50 AM, Theodore Ts'o <tytso@mit.edu> wrote: > On Thu, Jan 04, 2018 at 12:04:34PM +0100, Dmitry Vyukov wrote: >> >> The problem is that it's not _me_, it's a computer program which uses >> some mail delivery system which I don't have full visibility into. I >> don't know if it even gets bounce emails (as far as I understand it's >> not LKML that generates them, LKML SMTP server just returns some error >> code and then it's a responsibility of somebody else to represent it >> by a reply email). If the only way to probe the behavior is to send >> actual emails to LKML (which have high chances to be actually >> delivered to all subscribers), it makes testing somewhat problematic. > > It looks like you're using the App Engine Mail API. You can configure > it to get bounce e-mails[1]. From what I can tell looking at the mail > headers, the mail gets sent via an intermediate SMTP host, such as > mail-it0-f69.google.com before it is delievered to vger. So if vger's > mailer bounces it via an SMTP error, it would be > mail-it0-f69.google.com's responsibility to generate a bounce message, > which you then should be able to catch. > > [1] https://cloud.google.com/appengine/docs/standard/go/mail/bounce > > You should be able to test this by having your app-engine send a > message to "invalid-address@vger.kernel.org". I've verified that this > will cause an immediate SMTP error: > > 554 5.0.0 Hi [74.207.234.97], unresolvable address: <invalid-address@vger.kernel.org>; nosuchuser; invalid-address@vger.kernel.org > > If it doesn't, you should file an internal bug report since that's > clearly a bug in the App Engine's mail infrastructure. If you can't > get satisfaction that way, my recommendation would be to set up an > Linode server (you can't use GCE because GCE blocks outgoing SMTP > connections) to be your mail gateway, and send the notifications from > a domain such as syzkaller.org, where you have full control of your > mail infrastructure, and you don't have to deal with nonsense such as > DMARC. Thanks, Ted! This is helpful. I've added handler for bounce emails: https://github.com/google/syzkaller/commit/19c05fffcb1860b2dcf17989b40ca16ed259fdea And I do see them after sending to invalid-address@vger.kernel.org. So now hopefully it will sched some light when we get another bounce. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 23:50 ` Theodore Ts'o 2018-01-08 12:01 ` Dmitry Vyukov @ 2018-01-08 12:11 ` Dmitry Vyukov 1 sibling, 0 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-08 12:11 UTC (permalink / raw) To: Theodore Ts'o, Dmitry Vyukov, Greg Kroah-Hartman, LKML, Andrew Morton, Linus Torvalds, syzkaller, David Miller On Fri, Jan 5, 2018 at 12:50 AM, Theodore Ts'o <tytso@mit.edu> wrote: > It also seems to me, looking at other complaints on this thread, that > there is the opportunity for the syzbot to do much more. For example, > you can see if it repro's on the last released mainline kernel (such > as 4.14) and if so, have the syzbot automatically do a bisection > search, so you can make sure the report goes to the best set of > developers to fix it, a pointer to the guilty commit. Hi Ted, I've filed https://github.com/google/syzkaller/issues/501 for this as this come up several times. There is bunch of problems, though: - unreliable reproducers - triggering of unrelated bugs (happens frequently) - flakes (bugs in GCE, crashes in tty on first ssh connection, etc) - bugs (races) that manifest in multiple different ways - bugs that will be attributed to tools improvements (e.g. KASAN, fault injection improvements) - reproducers that need slight changes on different kernel revisions Not sure what quality of bisection we can achieve. And kernel developers tend to be negative to any kind of bot gaffe, so we can lose both ways. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:09 LKML admins (syzbot emails are not delivered) Dmitry Vyukov 2018-01-04 9:23 ` Greg Kroah-Hartman @ 2018-01-04 9:25 ` Pavel Machek 2018-01-04 9:38 ` Mike Galbraith 1 sibling, 1 reply; 53+ messages in thread From: Pavel Machek @ 2018-01-04 9:25 UTC (permalink / raw) To: Dmitry Vyukov Cc: LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller [-- Attachment #1: Type: text/plain, Size: 913 bytes --] Hi! > > Some of syzbot emails don't appear on LKML mailing lists, while they > were mailed as any other emails. Here are few examples: > > "KASAN: use-after-free Read in rds_tcp_dev_event" > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > > "general protection fault in __wake_up_common" > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > > Does anybody know how to get in contact with real people behind LKML > and/or bugzilla? Not delivering syzbot emails might be good thing? You claimed you generate so many of them that you can't even check them by hand, so what makes you think thousands of lkml readers want to delete them by hand? You may want to talk to David Miller. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:25 ` Pavel Machek @ 2018-01-04 9:38 ` Mike Galbraith 2018-01-04 9:56 ` Pavel Machek 0 siblings, 1 reply; 53+ messages in thread From: Mike Galbraith @ 2018-01-04 9:38 UTC (permalink / raw) To: Pavel Machek, Dmitry Vyukov Cc: LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: > Hi! > > > > Some of syzbot emails don't appear on LKML mailing lists, while they > > were mailed as any other emails. Here are few examples: > > > > "KASAN: use-after-free Read in rds_tcp_dev_event" > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > > > > "general protection fault in __wake_up_common" > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > > > > Does anybody know how to get in contact with real people behind LKML > > and/or bugzilla? > > Not delivering syzbot emails might be good thing? Nah, the thing is finding and reporting bugs just like a human would, it just doesn't need sleep etc, so sometimes reports more than humans can keep up with. It needs a smarter brother.. but then again, maybe not, if bots start fixing things too, a lot of meatware hackers would have to go find real jobs. > You claimed you generate so many of them that you can't even check > them by hand, so what makes you think thousands of lkml readers want > to delete them by hand? Mail filters have existed for untold ages :) -Mike ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:38 ` Mike Galbraith @ 2018-01-04 9:56 ` Pavel Machek 2018-01-04 11:09 ` Dmitry Vyukov 0 siblings, 1 reply; 53+ messages in thread From: Pavel Machek @ 2018-01-04 9:56 UTC (permalink / raw) To: Mike Galbraith Cc: Dmitry Vyukov, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller > On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: > > Hi! > > > > > > Some of syzbot emails don't appear on LKML mailing lists, while they > > > were mailed as any other emails. Here are few examples: > > > > > > "KASAN: use-after-free Read in rds_tcp_dev_event" > > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > > > > > > "general protection fault in __wake_up_common" > > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > > > > > > Does anybody know how to get in contact with real people behind LKML > > > and/or bugzilla? > > > > Not delivering syzbot emails might be good thing? > > Nah, the thing is finding and reporting bugs just like a human would, > it just doesn't need sleep etc, so sometimes reports more than humans > can keep up with. It needs a smarter brother.. but then again, maybe > not, if bots start fixing things too, a lot of meatware hackers would > have to go find real jobs. Sending random, unrepeatable Oopses to lkml is not what humans would do, and perhaps not something bots should do, either. > > You claimed you generate so many of them that you can't even check > > them by hand, so what makes you think thousands of lkml readers want > > to delete them by hand? > > Mail filters have existed for untold ages :) Yeah, well, spammers also existed for long long time :-). Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 9:56 ` Pavel Machek @ 2018-01-04 11:09 ` Dmitry Vyukov 2018-01-04 11:18 ` Pavel Machek ` (2 more replies) 0 siblings, 3 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-04 11:09 UTC (permalink / raw) To: Pavel Machek Cc: Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote: >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: >> > Hi! >> > > >> > > Some of syzbot emails don't appear on LKML mailing lists, while they >> > > were mailed as any other emails. Here are few examples: >> > > >> > > "KASAN: use-after-free Read in rds_tcp_dev_event" >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ >> > > >> > > "general protection fault in __wake_up_common" >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ >> > > >> > > Does anybody know how to get in contact with real people behind LKML >> > > and/or bugzilla? >> > >> > Not delivering syzbot emails might be good thing? >> >> Nah, the thing is finding and reporting bugs just like a human would, >> it just doesn't need sleep etc, so sometimes reports more than humans >> can keep up with. It needs a smarter brother.. but then again, maybe >> not, if bots start fixing things too, a lot of meatware hackers would >> have to go find real jobs. > > Sending random, unrepeatable Oopses to lkml is not what humans would > do, and perhaps not something bots should do, either. Hi Pavel, I've answered this question here in full detail. In short, this is useful and actionable. https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:09 ` Dmitry Vyukov @ 2018-01-04 11:18 ` Pavel Machek 2018-01-15 10:08 ` Dmitry Vyukov 2018-01-04 15:23 ` Eric W. Biederman 2018-01-04 21:38 ` Pavel Machek 2 siblings, 1 reply; 53+ messages in thread From: Pavel Machek @ 2018-01-04 11:18 UTC (permalink / raw) To: Dmitry Vyukov Cc: Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller [-- Attachment #1: Type: text/plain, Size: 2312 bytes --] On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote: > On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote: > >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: > >> > Hi! > >> > > > >> > > Some of syzbot emails don't appear on LKML mailing lists, while they > >> > > were mailed as any other emails. Here are few examples: > >> > > > >> > > "KASAN: use-after-free Read in rds_tcp_dev_event" > >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > >> > > > >> > > "general protection fault in __wake_up_common" > >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > >> > > > >> > > Does anybody know how to get in contact with real people behind LKML > >> > > and/or bugzilla? > >> > > >> > Not delivering syzbot emails might be good thing? > >> > >> Nah, the thing is finding and reporting bugs just like a human would, > >> it just doesn't need sleep etc, so sometimes reports more than humans > >> can keep up with. It needs a smarter brother.. but then again, maybe > >> not, if bots start fixing things too, a lot of meatware hackers would > >> have to go find real jobs. > > > > Sending random, unrepeatable Oopses to lkml is not what humans would > > do, and perhaps not something bots should do, either. > > > Hi Pavel, > > I've answered this question here in full detail. In short, this is > useful and actionable. > https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ I've already deleted many such reports from my lkml folder. It definitely is below quality of normal bug reports. You want all the credit while doing none of the work: This bug is generated by a dumb bot. It may contain errors. See https://goo.gl/tpsmEJ for details. Direct all questions to syzk...@googlegroups.com. Please credit me with: Reported-by: syzbot <syzk...@googlegroups.com> ... #syz invalid Note: if the crash happens again, it will cause creation of a new bug report. I'd say this belongs to separate list. Interested parties can subscribe, and you can still manually forward reports if you handle duplicities etc. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:18 ` Pavel Machek @ 2018-01-15 10:08 ` Dmitry Vyukov 2018-01-15 13:08 ` Pavel Machek 0 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-15 10:08 UTC (permalink / raw) To: Pavel Machek Cc: Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Thu, Jan 4, 2018 at 12:18 PM, Pavel Machek <pavel@ucw.cz> wrote: > On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote: >> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote: >> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: >> >> > Hi! >> >> > > >> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they >> >> > > were mailed as any other emails. Here are few examples: >> >> > > >> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event" >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ >> >> > > >> >> > > "general protection fault in __wake_up_common" >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ >> >> > > >> >> > > Does anybody know how to get in contact with real people behind LKML >> >> > > and/or bugzilla? >> >> > >> >> > Not delivering syzbot emails might be good thing? >> >> >> >> Nah, the thing is finding and reporting bugs just like a human would, >> >> it just doesn't need sleep etc, so sometimes reports more than humans >> >> can keep up with. It needs a smarter brother.. but then again, maybe >> >> not, if bots start fixing things too, a lot of meatware hackers would >> >> have to go find real jobs. >> > >> > Sending random, unrepeatable Oopses to lkml is not what humans would >> > do, and perhaps not something bots should do, either. >> >> >> Hi Pavel, >> >> I've answered this question here in full detail. In short, this is >> useful and actionable. >> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ > > I've already deleted many such reports from my lkml folder. It > definitely is below quality of normal bug reports. Pavel, if you point to exact deficiencies in the process and propose ways to improve it, we can turn this into a positive, constructive and actionable discussion. In lots of cases (~50%) quality of syzbot reports is equal to human reports, or _higher_. It provides exact kernel commit, config, compiler, stand-alone C reproducer and a nice, symbolized report even with inline frames. You don't always get all of this from human reports. In the remaining cases (no reproducer), quality of syzbot reports is the _same_ as for human reports. Say, your machine randomly crashes. You reboot it, but it crashes again after some time. You try to repeat what you did before the crash (say, opened a particular web page), but it does not reproduce the crash. But one way or another, it happened and it's a kernel bug (you did not write garbage into /dev/mem, nor loaded untested kernel modules). What do you do in such case? Right, you report what you know about the bug (kernel crash message, kernel revision and config). As I said before, in lots of cases (I would say ~2/3), kernel crash reports are actionable on its own. For example, KASAN/LOCKDEP reports contain enough context to localize a bug. Lots of WARNINGs/GPFs point to simple missed input checks or off-by-one's. So it would be wrong to hide them, keep them private and pretend that nothing happened. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 10:08 ` Dmitry Vyukov @ 2018-01-15 13:08 ` Pavel Machek 2018-01-15 13:38 ` Dmitry Vyukov 0 siblings, 1 reply; 53+ messages in thread From: Pavel Machek @ 2018-01-15 13:08 UTC (permalink / raw) To: Dmitry Vyukov Cc: Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller [-- Attachment #1: Type: text/plain, Size: 3101 bytes --] On Mon 2018-01-15 11:08:12, Dmitry Vyukov wrote: > On Thu, Jan 4, 2018 at 12:18 PM, Pavel Machek <pavel@ucw.cz> wrote: > > On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote: > >> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote: > >> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: > >> >> > Hi! > >> >> > > > >> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they > >> >> > > were mailed as any other emails. Here are few examples: > >> >> > > > >> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event" > >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > >> >> > > > >> >> > > "general protection fault in __wake_up_common" > >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > >> >> > > > >> >> > > Does anybody know how to get in contact with real people behind LKML > >> >> > > and/or bugzilla? > >> >> > > >> >> > Not delivering syzbot emails might be good thing? > >> >> > >> >> Nah, the thing is finding and reporting bugs just like a human would, > >> >> it just doesn't need sleep etc, so sometimes reports more than humans > >> >> can keep up with. It needs a smarter brother.. but then again, maybe > >> >> not, if bots start fixing things too, a lot of meatware hackers would > >> >> have to go find real jobs. > >> > > >> > Sending random, unrepeatable Oopses to lkml is not what humans would > >> > do, and perhaps not something bots should do, either. > >> > >> > >> Hi Pavel, > >> > >> I've answered this question here in full detail. In short, this is > >> useful and actionable. > >> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ > > > > I've already deleted many such reports from my lkml folder. It > > definitely is below quality of normal bug reports. > > > Pavel, if you point to exact deficiencies in the process and propose > ways to improve it, we can turn this into a positive, constructive and > actionable discussion. > > In lots of cases (~50%) quality of syzbot reports is equal to human > reports, or _higher_. > It provides exact kernel commit, config, compiler, stand-alone C > reproducer and a nice, symbolized report even with inline frames. You > don't always get all of this from human reports. > > In the remaining cases (no reproducer), quality of syzbot reports is > the _same_ as for human reports. > Say, your machine randomly crashes. You reboot it, but it crashes > again after some time. You try to repeat what you did before the crash > (say, opened a particular web page), but it does not reproduce the > crash. But one way or another, it happened and it's a kernel bug I have not seen a good quality report from syzbot, yet. Normally, humans agregate test reports, and test patches etc. Can you step between robot and lkml, and provide same services humans usually provide? Thanks, Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 13:08 ` Pavel Machek @ 2018-01-15 13:38 ` Dmitry Vyukov 2018-01-15 13:53 ` Pavel Machek 0 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-15 13:38 UTC (permalink / raw) To: Pavel Machek Cc: Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Mon, Jan 15, 2018 at 2:08 PM, Pavel Machek <pavel@ucw.cz> wrote: > On Mon 2018-01-15 11:08:12, Dmitry Vyukov wrote: >> On Thu, Jan 4, 2018 at 12:18 PM, Pavel Machek <pavel@ucw.cz> wrote: >> > On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote: >> >> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote: >> >> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: >> >> >> > Hi! >> >> >> > > >> >> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they >> >> >> > > were mailed as any other emails. Here are few examples: >> >> >> > > >> >> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event" >> >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ >> >> >> > > >> >> >> > > "general protection fault in __wake_up_common" >> >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ >> >> >> > > >> >> >> > > Does anybody know how to get in contact with real people behind LKML >> >> >> > > and/or bugzilla? >> >> >> > >> >> >> > Not delivering syzbot emails might be good thing? >> >> >> >> >> >> Nah, the thing is finding and reporting bugs just like a human would, >> >> >> it just doesn't need sleep etc, so sometimes reports more than humans >> >> >> can keep up with. It needs a smarter brother.. but then again, maybe >> >> >> not, if bots start fixing things too, a lot of meatware hackers would >> >> >> have to go find real jobs. >> >> > >> >> > Sending random, unrepeatable Oopses to lkml is not what humans would >> >> > do, and perhaps not something bots should do, either. >> >> >> >> >> >> Hi Pavel, >> >> >> >> I've answered this question here in full detail. In short, this is >> >> useful and actionable. >> >> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ >> > >> > I've already deleted many such reports from my lkml folder. It >> > definitely is below quality of normal bug reports. >> >> >> Pavel, if you point to exact deficiencies in the process and propose >> ways to improve it, we can turn this into a positive, constructive and >> actionable discussion. >> >> In lots of cases (~50%) quality of syzbot reports is equal to human >> reports, or _higher_. >> It provides exact kernel commit, config, compiler, stand-alone C >> reproducer and a nice, symbolized report even with inline frames. You >> don't always get all of this from human reports. >> >> In the remaining cases (no reproducer), quality of syzbot reports is >> the _same_ as for human reports. >> Say, your machine randomly crashes. You reboot it, but it crashes >> again after some time. You try to repeat what you did before the crash >> (say, opened a particular web page), but it does not reproduce the >> crash. But one way or another, it happened and it's a kernel bug > > I have not seen a good quality report from syzbot, yet. I don't know how many you checked, so I don't know how to interpret this. If you want to see one, grep kernel commit log for syzbot, get bug id, search for it in https://groups.google.com/forum/#!forum/syzkaller-bugs > Normally, humans agregate test reports, and test patches etc. Can you > step between robot and lkml, and provide same services humans usually > provide? syzbot does all of this already. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 13:38 ` Dmitry Vyukov @ 2018-01-15 13:53 ` Pavel Machek 0 siblings, 0 replies; 53+ messages in thread From: Pavel Machek @ 2018-01-15 13:53 UTC (permalink / raw) To: Dmitry Vyukov Cc: Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller [-- Attachment #1: Type: text/plain, Size: 2214 bytes --] Hi! > >> In lots of cases (~50%) quality of syzbot reports is equal to human > >> reports, or _higher_. > >> It provides exact kernel commit, config, compiler, stand-alone C > >> reproducer and a nice, symbolized report even with inline frames. You > >> don't always get all of this from human reports. > >> > >> In the remaining cases (no reproducer), quality of syzbot reports is > >> the _same_ as for human reports. > >> Say, your machine randomly crashes. You reboot it, but it crashes > >> again after some time. You try to repeat what you did before the crash > >> (say, opened a particular web page), but it does not reproduce the > >> crash. But one way or another, it happened and it's a kernel bug > > > > I have not seen a good quality report from syzbot, yet. > > I don't know how many you checked, so I don't know how to interpret this. > If you want to see one, grep kernel commit log for syzbot, get bug id, > search for it in > https://groups.google.com/forum/#!forum/syzkaller-bugs > > > > Normally, humans agregate test reports, and test patches etc. Can you > > step between robot and lkml, and provide same services humans usually > > provide? > > syzbot does all of this already. Does it? Because its documentation says otherwise: syzbot will keep track of this bug report. Once a fix for this bug is merged into any tree, reply to this email with: #syz fix: exact-commit-title To mark this as a duplicate of another syzbot report, please reply with: #syz dup: exact-subject-of-another-report If it's a one-off invalid bug report, please reply with: #syz invalid Note: if the crash happens again, it will cause creation of a new bug report. Note: all commands must start from beginning of the line in the email body. _You_ should be doing duplicates processing, and preferably you should also take care to translate replies into something syzbot understand. Reports should come from your email adress, and you should really handle replies that do not come in required format. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:09 ` Dmitry Vyukov 2018-01-04 11:18 ` Pavel Machek @ 2018-01-04 15:23 ` Eric W. Biederman 2018-01-15 10:54 ` Dmitry Vyukov 2018-01-04 21:38 ` Pavel Machek 2 siblings, 1 reply; 53+ messages in thread From: Eric W. Biederman @ 2018-01-04 15:23 UTC (permalink / raw) To: Dmitry Vyukov Cc: Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller Dmitry Vyukov <dvyukov@google.com> writes: > Hi Pavel, > > I've answered this question here in full detail. In short, this is > useful and actionable. > https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ *Snort* If the information to solve an issue is not in the Oops syzbot is useless. The Oops isn't even mailed in plain text so I have to save the stupid thing in a file to see the full text of the problem. Further there is no place in the syzbot process to test fixes. Then there is the issue of testing linux-next and reporting errors on who knows what code configuration against code that hasn't changed in linux-next. Which presumably any sane person would assume the errors are introduced by some other piece of new code. But syzbot goes and spams the people who wrote the function where the code is failing. Bots can work. We have all of the automatic testing infrastructure against everyone's branches on kernel.org to prove it. syzbot finds weird errors, so that makes the problem space more difficult to deal with. Still I compleltely don't see the people behind syzbot presumably you Dmitry taking responsibility for syzbot failings. Instead I see excuses like you don't completely control some part of the code that syzbot is built on so can't fix practical real world issues. Like Content-type. Bots can be the most horrible thing for a code base. If there is not someone or something going through an filtering out the false positives. If there is not a process to ensure that issues are brought to the proper peoples attention so things get fixed. Bots can be completely demoralizing or possibily desensitizing because you keep seeing issues, and nothing you do ever makes the issues go away. Given that no one seems to take any responsibility for syzbots failures of any kind. Not content-type in the emails. Not the body of the message (which has a massive disclaimer). I don't find syzbot at all useful. Tools are for people, in this case kernel programmers. syzbot has serious usability issues. That makes syzbot a bad tool. Eric ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 15:23 ` Eric W. Biederman @ 2018-01-15 10:54 ` Dmitry Vyukov 2018-01-15 12:54 ` Pavel Machek 2018-01-15 16:38 ` Eric W. Biederman 0 siblings, 2 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-15 10:54 UTC (permalink / raw) To: Eric W. Biederman Cc: Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Thu, Jan 4, 2018 at 4:23 PM, Eric W. Biederman <ebiederm@xmission.com> wrote: > Dmitry Vyukov <dvyukov@google.com> writes: > >> Hi Pavel, >> >> I've answered this question here in full detail. In short, this is >> useful and actionable. >> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ > > *Snort* > > If the information to solve an issue is not in the Oops syzbot is > useless. Hi Eric That's true. But maintainers of the subsystem is in the best position to judge that. For that they need to see the report. > The Oops isn't even mailed in plain text so I have to save the stupid > thing in a file to see the full text of the problem. Please elaborate. Take any syzbot email, oops is right there in the email, in plain text: https://groups.google.com/forum/#!topic/syzkaller-bugs/F6ImuLmyue8 > Further there is no place in the syzbot process to test fixes. Please elaborate. Kernel developer who fixes the bugs, tests it the same way as he/she does for any other bugs. There is really nothing in syzbot that prevents you from testing. > Then there is the issue of testing linux-next and reporting errors on > who knows what code configuration against code that hasn't changed in > linux-next. Which presumably any sane person would assume the errors > are introduced by some other piece of new code. But syzbot goes and > spams the people who wrote the function where the code is failing. syzbot uses get_maintainers.pl. If you have better suggestions, I am listening. And note: syzbot _always_ provides exact code configuration. > Bots can work. We have all of the automatic testing infrastructure > against everyone's branches on kernel.org to prove it. If you mean build/boot testing, than that's an order of magnitude simper problem. You can build on every commit, you can precisely pinpoint the guilty commit, etc. Please keep this in mind. > syzbot finds weird errors, so that makes the problem space more > difficult to deal with. kernel contains weird errors, that makes the problem space more difficult. > Still I compleltely don't see the people behind syzbot presumably you > Dmitry taking responsibility for syzbot failings. Instead I see excuses > like you don't completely control some part of the code that syzbot is > built on so can't fix practical real world issues. Like Content-type. As far as I understand you mean this one: https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/VSZaokajCgAJ I probably should have described the rationale in more details. It's not only about technical limitations. It's also about importance of a feature, time required to implement it, and in the end if it's the right thing to do at all or not. If that would be a major issue that is significantly affects experience, that would happen one way or another regardless of technical limitations. Also simple one-line changes generally happen even if it's low profit. But in that case, I think it's just the wrong thing to do. .txt is good, standard extension for text files. On the other hand, .syz is completely non-standard that no programs know how to deal with. That's why it did not happen. The support for Reported-by tags as discussed in "syzbot process" thread happened within a week. Hope this resolves your concerns. > Bots can be the most horrible thing for a code base. If there is not > someone or something going through an filtering out the false positives. > If there is not a process to ensure that issues are brought to the > proper peoples attention so things get fixed. Bots can be completely > demoralizing or possibily desensitizing because you keep seeing issues, > and nothing you do ever makes the issues go away. > > Given that no one seems to take any responsibility for syzbots failures > of any kind. Not content-type in the emails. Not the body of the > message (which has a massive disclaimer). I don't find syzbot at all > useful. > > Tools are for people, in this case kernel programmers. syzbot has > serious usability issues. That makes syzbot a bad tool. First of all, none of syzbot reports are false positives in the main sense of this term. Everything it reports happened on unmodified kernel, running user-space workload, without loading custom modules, writing to /dev/mem, etc. These _all_ are kernel bugs. In some cases kernel is bad at explaining precisely what went wrong. But it's expected from complex, concurrent, non-deterministic system written in an unsafe language. We need to deal with it. You get the same reports from humans as well. Say, there is an invalid free in pcrypt which corrupts memory, but kernel crashes in selinux later. You will get report about selinux from a human. syzbot actually makes situation a bit better to the degree possible as it enables almost all debugging configs. So instead of a random corruption reports, it provides a KASAN report about the exact location. Instead of a dead kernel, you get LOCKDEP report about exact lock inversion, etc. Now there are duplicates, induced bugs, unexplainable crashes, reports mailed to wrong people, etc. There are hundreds of subsystems in kernel. And answering any of these questions requires expertise in a particular subsystem. Say, this crash is also a possible way how that bug could manifest. Or, the crash happened in this subsystem, but the root cause is actually in the upper-level subsystem that misuses this subsystem. The right people to deal with this are maintainers of particular subsystems. Not a single person that does not work on any of these hundreds of subsystems. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 10:54 ` Dmitry Vyukov @ 2018-01-15 12:54 ` Pavel Machek 2018-01-15 13:02 ` Dmitry Vyukov 2018-01-15 16:38 ` Eric W. Biederman 1 sibling, 1 reply; 53+ messages in thread From: Pavel Machek @ 2018-01-15 12:54 UTC (permalink / raw) To: Dmitry Vyukov Cc: Eric W. Biederman, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller [-- Attachment #1: Type: text/plain, Size: 878 bytes --] Hi! > > *Snort* > > > > If the information to solve an issue is not in the Oops syzbot is > > useless. > > Hi Eric > > That's true. But maintainers of the subsystem is in the best position > to judge that. For that they need to see the report. Unless they are already overloaded by better quality reports. > > Further there is no place in the syzbot process to test fixes. > > Please elaborate. > Kernel developer who fixes the bugs, tests it the same way as he/she > does for any other bugs. There is really nothing in syzbot that > prevents you from testing. Well, normally people are interested in the bugs they report, and thus willing to test the patches. Your bot.. is not interested. Pavel -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 12:54 ` Pavel Machek @ 2018-01-15 13:02 ` Dmitry Vyukov 2018-01-15 16:16 ` Eric W. Biederman 0 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-15 13:02 UTC (permalink / raw) To: Pavel Machek Cc: Eric W. Biederman, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Mon, Jan 15, 2018 at 1:54 PM, Pavel Machek <pavel@ucw.cz> wrote: > Hi! > >> > *Snort* >> > >> > If the information to solve an issue is not in the Oops syzbot is >> > useless. >> >> Hi Eric >> >> That's true. But maintainers of the subsystem is in the best position >> to judge that. For that they need to see the report. > > Unless they are already overloaded by better quality reports. > >> > Further there is no place in the syzbot process to test fixes. >> >> Please elaborate. >> Kernel developer who fixes the bugs, tests it the same way as he/she >> does for any other bugs. There is really nothing in syzbot that >> prevents you from testing. > > Well, normally people are interested in the bugs they report, and thus > willing to test the patches. Your bot.. is not interested. Not true. syzbot is very much interested in bugs it reports, keeps careful track of them and tests patches. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 13:02 ` Dmitry Vyukov @ 2018-01-15 16:16 ` Eric W. Biederman 2018-01-16 18:01 ` Dmitry Vyukov 0 siblings, 1 reply; 53+ messages in thread From: Eric W. Biederman @ 2018-01-15 16:16 UTC (permalink / raw) To: Dmitry Vyukov Cc: Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller Dmitry Vyukov <dvyukov@google.com> writes: 2> On Mon, Jan 15, 2018 at 1:54 PM, Pavel Machek <pavel@ucw.cz> wrote: >> Hi! >> >>> > *Snort* >>> > >>> > If the information to solve an issue is not in the Oops syzbot is >>> > useless. >>> >>> Hi Eric >>> >>> That's true. But maintainers of the subsystem is in the best position >>> to judge that. For that they need to see the report. >> >> Unless they are already overloaded by better quality reports. >> >>> > Further there is no place in the syzbot process to test fixes. >>> >>> Please elaborate. >>> Kernel developer who fixes the bugs, tests it the same way as he/she >>> does for any other bugs. There is really nothing in syzbot that >>> prevents you from testing. >> >> Well, normally people are interested in the bugs they report, and thus >> willing to test the patches. Your bot.. is not interested. > > Not true. syzbot is very much interested in bugs it reports, keeps > careful track of them and tests patches. It offers to test fixes not to add more information so that the bug can be tracked down better. Having asked explicitly for some additional testing to track down and issue and been told the process was happy to test fixes I know that this has been most definitely the case. Modifying the kernel and testing is important as sometimes it can be extremely difficult to figure out what causes an issue. Especially when it is an interaction of factors like running a system on the edge of OOM. So it requires small kmallocs to fail (which does not usually happen). Eric ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 16:16 ` Eric W. Biederman @ 2018-01-16 18:01 ` Dmitry Vyukov 0 siblings, 0 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-16 18:01 UTC (permalink / raw) To: Eric W. Biederman Cc: Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Mon, Jan 15, 2018 at 5:16 PM, Eric W. Biederman <ebiederm@xmission.com> wrote: > Dmitry Vyukov <dvyukov@google.com> writes: > > 2> On Mon, Jan 15, 2018 at 1:54 PM, Pavel Machek <pavel@ucw.cz> wrote: >>> Hi! >>> >>>> > *Snort* >>>> > >>>> > If the information to solve an issue is not in the Oops syzbot is >>>> > useless. >>>> >>>> Hi Eric >>>> >>>> That's true. But maintainers of the subsystem is in the best position >>>> to judge that. For that they need to see the report. >>> >>> Unless they are already overloaded by better quality reports. >>> >>>> > Further there is no place in the syzbot process to test fixes. >>>> >>>> Please elaborate. >>>> Kernel developer who fixes the bugs, tests it the same way as he/she >>>> does for any other bugs. There is really nothing in syzbot that >>>> prevents you from testing. >>> >>> Well, normally people are interested in the bugs they report, and thus >>> willing to test the patches. Your bot.. is not interested. >> >> Not true. syzbot is very much interested in bugs it reports, keeps >> careful track of them and tests patches. > > It offers to test fixes not to add more information so that the bug can > be tracked down better. Having asked explicitly for some additional > testing to track down and issue and been told the process was happy to > test fixes I know that this has been most definitely the case. > > Modifying the kernel and testing is important as sometimes it can be > extremely difficult to figure out what causes an issue. Especially when > it is an interaction of factors like running a system on the edge of > OOM. So it requires small kmallocs to fail (which does not usually > happen). Patch testing can be used for debugging as well. It just runs your patch (maybe with additional WARNINGs) and then gives you console output if kernel crashes. I've clarified docs regarding this: https://github.com/google/syzkaller/commit/9aaf64b3742d1d4f744e22cad567906cebb201a2 9 out of 10 of these bugs can also be reproduced locally without any special setup other than using the provided config. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 10:54 ` Dmitry Vyukov 2018-01-15 12:54 ` Pavel Machek @ 2018-01-15 16:38 ` Eric W. Biederman 2018-01-16 7:12 ` Theodore Ts'o ` (2 more replies) 1 sibling, 3 replies; 53+ messages in thread From: Eric W. Biederman @ 2018-01-15 16:38 UTC (permalink / raw) To: Dmitry Vyukov Cc: Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller Dmitry Vyukov <dvyukov@google.com> writes: > On Thu, Jan 4, 2018 at 4:23 PM, Eric W. Biederman <ebiederm@xmission.com> wrote: >> Dmitry Vyukov <dvyukov@google.com> writes: >> >>> Hi Pavel, >>> >>> I've answered this question here in full detail. In short, this is >>> useful and actionable. >>> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ >> >> *Snort* >> >> If the information to solve an issue is not in the Oops syzbot is >> useless. > > Hi Eric > > That's true. But maintainers of the subsystem is in the best position > to judge that. For that they need to see the report. >> Then there is the issue of testing linux-next and reporting errors on >> who knows what code configuration against code that hasn't changed in >> linux-next. Which presumably any sane person would assume the errors >> are introduced by some other piece of new code. But syzbot goes and >> spams the people who wrote the function where the code is failing. > > syzbot uses get_maintainers.pl. If you have better suggestions, I am listening. > And note: syzbot _always_ provides exact code configuration. If you are testing linux-next you should really report it to whomevers branch in linux-next you are testing. Ideally the tests would be run on mainline and see nothing and then on linux-next so you know it is a newly introduced error. Sometimes the branches on linux-next are experimental crap. If someone adds an experimental memory allocator to linux-next before discovering it causes all kinds of problems I don't want bug reports about my code not being able to allocate memory because the memory allocator was bad. If you don't have the resources to test the individual branches of linux-next please just test Linus's tree. That will be much more meaningful and productive. >> Bots can work. We have all of the automatic testing infrastructure >> against everyone's branches on kernel.org to prove it. > > If you mean build/boot testing, than that's an order of magnitude > simper problem. You can build on every commit, you can precisely > pinpoint the guilty commit, etc. Please keep this in mind. The difference is not what is being tested. The difference is how the interface to human beings is constructed. If it doesn't feel like there is a human being willing to work with you on the other end of a bug report it is not a good situation. >> syzbot finds weird errors, so that makes the problem space more >> difficult to deal with. > > kernel contains weird errors, that makes the problem space more difficult. > > >> Still I compleltely don't see the people behind syzbot presumably you >> Dmitry taking responsibility for syzbot failings. Instead I see excuses >> like you don't completely control some part of the code that syzbot is >> built on so can't fix practical real world issues. Like Content-type. > > As far as I understand you mean this one: > https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/VSZaokajCgAJ > > I probably should have described the rationale in more details. > It's not only about technical limitations. It's also about importance > of a feature, time required to implement it, and in the end if it's > the right thing to do at all or not. If that would be a major issue > that is significantly affects experience, that would happen one way or > another regardless of technical limitations. Also simple one-line > changes generally happen even if it's low profit. But in that case, I > think it's just the wrong thing to do. .txt is good, standard > extension for text files. On the other hand, .syz is completely > non-standard that no programs know how to deal with. That's why it did > not happen. > The support for Reported-by tags as discussed in "syzbot process" > thread happened within a week. When I made the complaint it came to me and to messages on lkml as .log. With Content-Type: Application/Octent-stream. That is a bloody mess that wastes peoples time. If it is fixed good, it certainly was not fixed at that point. But it is much more than any single issue. You get defensive when people critisize syzbot. Instead of recognizing it's failings. It is fine to say I would like to do xyz but it will take awhile before we can get to it. > Hope this resolves your concerns. This email just intensifies them, as it feels again like you are trying to shift the blame. >> Bots can be the most horrible thing for a code base. If there is not >> someone or something going through an filtering out the false positives. >> If there is not a process to ensure that issues are brought to the >> proper peoples attention so things get fixed. Bots can be completely >> demoralizing or possibily desensitizing because you keep seeing issues, >> and nothing you do ever makes the issues go away. >> >> Given that no one seems to take any responsibility for syzbots failures >> of any kind. Not content-type in the emails. Not the body of the >> message (which has a massive disclaimer). I don't find syzbot at all >> useful. >> >> Tools are for people, in this case kernel programmers. syzbot has >> serious usability issues. That makes syzbot a bad tool. > > First of all, none of syzbot reports are false positives in the main > sense of this term. *Snort* You are testing linux-next. Who knows what pieces of that are going to go into a stable kernel. You are not reporting failures against individual maintainers who changed linux-next you are reporting to random people in get_maintainers.pl who may have no connection with the code change. So it is a very low quality bug report, on some random mutation of the linux-kernel. You might as well be testing some random distro kernel and using get_maintainers.pl to tell us something is wrong. > You get the same reports from humans as well. Say, there is an invalid > free in pcrypt which corrupts memory, but kernel crashes in selinux > later. You will get report about selinux from a human. > syzbot actually makes situation a bit better to the degree possible as > it enables almost all debugging configs. So instead of a random > corruption reports, it provides a KASAN report about the exact > location. Instead of a dead kernel, you get LOCKDEP report about exact > lock inversion, etc. But I can ask the human what their configuration was and what they were doing when the error happened. Further things can be prioritized by how badly the errors affect real people. In practice if bugs don't affect people more than once, they don't care. That conversation can not be had with syzbot. Outside of the bugs being considered as considered as security issues, the bugs syzbot finds are generally things that don't affect anyone in practice. So are very low on the priority of things to get fixed. > Now there are duplicates, induced bugs, unexplainable crashes, reports > mailed to wrong people, etc. > There are hundreds of subsystems in kernel. And answering any of these > questions requires expertise in a particular subsystem. Say, this > crash is also a possible way how that bug could manifest. Or, the > crash happened in this subsystem, but the root cause is actually in > the upper-level subsystem that misuses this subsystem. > The right people to deal with this are maintainers of particular > subsystems. Not a single person that does not work on any of these > hundreds of subsystems. I am definitely not asking for expertise in the kernel. I am asking for a human who wants to help track down bugs in the kernel. Not a poorly backed up accusation that I did something wrong. Because that is how syzbot feels today. Eric ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 16:38 ` Eric W. Biederman @ 2018-01-16 7:12 ` Theodore Ts'o 2018-01-16 7:51 ` Dmitry Vyukov ` (3 more replies) 2018-01-16 18:20 ` Dmitry Vyukov 2018-01-17 8:13 ` Dmitry Vyukov 2 siblings, 4 replies; 53+ messages in thread From: Theodore Ts'o @ 2018-01-16 7:12 UTC (permalink / raw) To: Eric W. Biederman Cc: Dmitry Vyukov, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote: > > Sometimes the branches on linux-next are experimental crap. If someone > adds an experimental memory allocator to linux-next before discovering > it causes all kinds of problems I don't want bug reports about my code > not being able to allocate memory because the memory allocator was bad. > > If you don't have the resources to test the individual branches of > linux-next please just test Linus's tree. That will be much more > meaningful and productive. I have to agree with Eric here, the reason why Fengguang Wu's 0-day testing robot is much better received by developers is that he does not test linux-net, but rather individual subsystem git trees and branches. His test automation also does an automatic bisection search, and can point at a specific commit --- at which point e-mail goes out to owner of the subsystem git tree, and to the people who authored and/or reviewed the guilty commit. Dmitry, perhaps you could collaborate with Intel's 0-day testing folks? They have code which does all of this, and perhaps it can be leveraged. > > When I made the complaint it came to me and to messages on lkml as > .log. With Content-Type: Application/Octent-stream. > > That is a bloody mess that wastes peoples time. If it is fixed good, > it certainly was not fixed at that point. I just checked a recent report from the Syzbot, and it's not fixed. The raw.log file still uses a Content-Type of Application/Octet-stream. Worse the reproducer C source file has a content type of Application/Octet-stream instead of the much more sane Application/text. <Face palm> Hint to Googlers --- many kernel developers do not use Gmail because it does unspeakable things to whitespaces, which results in mangled patches, or because they want real mail threading. The Content-Type really matters, because for many text-based Mail User Agents, if it is Application/octet-stream, the MUA will assume that it can't be displayed as text, and require that it be saved to a file, which the developer then has to manually read by firing up a pager or an editor. When you are getting hundreds or thousands of messages a day, having critical data which darn well *could* be displayed as text require manual processing adds friction and destroys developer productivity. So *you* might think the Content-type is trivial, but for developers who live in their MUA's (that's why many prefer to review patches in their mail reader, and not have to switch to web browser to use Gerrit), screwing up the Content-type is going to be a big deal to them. > Outside of the bugs being considered as considered as security issues, > the bugs syzbot finds are generally things that don't affect anyone in > practice. So are very low on the priority of things to get fixed. The real danger is that people will start auto-filing Syzbot reports to "file 13" (e.g., the trash can) because they are too annoying.... But that's Dmitry and the Syzkaller team's problem, not the kernel developers. :-) - Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 7:12 ` Theodore Ts'o @ 2018-01-16 7:51 ` Dmitry Vyukov 2018-01-16 9:52 ` Guenter Roeck 2018-01-16 7:59 ` Dmitry Vyukov ` (2 subsequent siblings) 3 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-16 7:51 UTC (permalink / raw) To: Theodore Ts'o, Eric W. Biederman, Dmitry Vyukov, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller, Guenter Roeck On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: > On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote: >> >> Sometimes the branches on linux-next are experimental crap. If someone >> adds an experimental memory allocator to linux-next before discovering >> it causes all kinds of problems I don't want bug reports about my code >> not being able to allocate memory because the memory allocator was bad. >> >> If you don't have the resources to test the individual branches of >> linux-next please just test Linus's tree. That will be much more >> meaningful and productive. > > I have to agree with Eric here, the reason why Fengguang Wu's 0-day > testing robot is much better received by developers is that he does > not test linux-net, I will remove linux-next if there is a general agreement that it's not useful. Though, I've heard different opinions from kernel developers as well. I will write a separate email asking what branches should be tested. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 7:51 ` Dmitry Vyukov @ 2018-01-16 9:52 ` Guenter Roeck 2018-01-16 9:56 ` Dmitry Vyukov 0 siblings, 1 reply; 53+ messages in thread From: Guenter Roeck @ 2018-01-16 9:52 UTC (permalink / raw) To: Dmitry Vyukov Cc: Theodore Ts'o, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Mon, Jan 15, 2018 at 11:51 PM, Dmitry Vyukov <dvyukov@google.com> wrote: > On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: >> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote: >>> >>> Sometimes the branches on linux-next are experimental crap. If someone >>> adds an experimental memory allocator to linux-next before discovering >>> it causes all kinds of problems I don't want bug reports about my code >>> not being able to allocate memory because the memory allocator was bad. >>> >>> If you don't have the resources to test the individual branches of >>> linux-next please just test Linus's tree. That will be much more >>> meaningful and productive. >> >> I have to agree with Eric here, the reason why Fengguang Wu's 0-day >> testing robot is much better received by developers is that he does >> not test linux-net, > Interesting. Assuming that refers to linux-next, not linux-net, that may explain why linux-next tends to deteriorate. I wonder if I should drop it from my testing as well. I'll be happy to follow whatever the result of this exchange is and do the same. Guenter > I will remove linux-next if there is a general agreement that it's not > useful. Though, I've heard different opinions from kernel developers > as well. I will write a separate email asking what branches should be > tested. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 9:52 ` Guenter Roeck @ 2018-01-16 9:56 ` Dmitry Vyukov 2018-01-16 9:58 ` Guenter Roeck 0 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-16 9:56 UTC (permalink / raw) To: Guenter Roeck Cc: Theodore Ts'o, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Tue, Jan 16, 2018 at 10:52 AM, Guenter Roeck <groeck@google.com> wrote: > On Mon, Jan 15, 2018 at 11:51 PM, Dmitry Vyukov <dvyukov@google.com> wrote: >> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: >>> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote: >>>> >>>> Sometimes the branches on linux-next are experimental crap. If someone >>>> adds an experimental memory allocator to linux-next before discovering >>>> it causes all kinds of problems I don't want bug reports about my code >>>> not being able to allocate memory because the memory allocator was bad. >>>> >>>> If you don't have the resources to test the individual branches of >>>> linux-next please just test Linus's tree. That will be much more >>>> meaningful and productive. >>> >>> I have to agree with Eric here, the reason why Fengguang Wu's 0-day >>> testing robot is much better received by developers is that he does >>> not test linux-net, >> > > Interesting. Assuming that refers to linux-next, not linux-net, that > may explain why linux-next tends to deteriorate. I wonder if I should > drop it from my testing as well. I'll be happy to follow whatever the > result of this exchange is and do the same. > > Guenter > >> I will remove linux-next if there is a general agreement that it's not >> useful. Though, I've heard different opinions from kernel developers >> as well. I will write a separate email asking what branches should be >> tested. Let's please move discussion of this topic to "what trees/branches to test on syzbot" thread. This thread is now about too many things. Hope you don't mind if I repost your last email there. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 9:56 ` Dmitry Vyukov @ 2018-01-16 9:58 ` Guenter Roeck 0 siblings, 0 replies; 53+ messages in thread From: Guenter Roeck @ 2018-01-16 9:58 UTC (permalink / raw) To: Dmitry Vyukov Cc: Theodore Ts'o, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Tue, Jan 16, 2018 at 1:56 AM, Dmitry Vyukov <dvyukov@google.com> wrote: > On Tue, Jan 16, 2018 at 10:52 AM, Guenter Roeck <groeck@google.com> wrote: >> On Mon, Jan 15, 2018 at 11:51 PM, Dmitry Vyukov <dvyukov@google.com> wrote: >>> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: >>>> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote: >>>>> >>>>> Sometimes the branches on linux-next are experimental crap. If someone >>>>> adds an experimental memory allocator to linux-next before discovering >>>>> it causes all kinds of problems I don't want bug reports about my code >>>>> not being able to allocate memory because the memory allocator was bad. >>>>> >>>>> If you don't have the resources to test the individual branches of >>>>> linux-next please just test Linus's tree. That will be much more >>>>> meaningful and productive. >>>> >>>> I have to agree with Eric here, the reason why Fengguang Wu's 0-day >>>> testing robot is much better received by developers is that he does >>>> not test linux-net, >>> >> >> Interesting. Assuming that refers to linux-next, not linux-net, that >> may explain why linux-next tends to deteriorate. I wonder if I should >> drop it from my testing as well. I'll be happy to follow whatever the >> result of this exchange is and do the same. >> >> Guenter >> >>> I will remove linux-next if there is a general agreement that it's not >>> useful. Though, I've heard different opinions from kernel developers >>> as well. I will write a separate email asking what branches should be >>> tested. > > Let's please move discussion of this topic to "what trees/branches to > test on syzbot" thread. This thread is now about too many things. > Hope you don't mind if I repost your last email there. Sure, go ahead. Guenter ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 7:12 ` Theodore Ts'o 2018-01-16 7:51 ` Dmitry Vyukov @ 2018-01-16 7:59 ` Dmitry Vyukov 2018-01-16 11:19 ` Fengguang Wu 2018-01-16 8:31 ` Dmitry Vyukov 2018-01-16 8:34 ` Dmitry Vyukov 3 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-16 7:59 UTC (permalink / raw) To: Theodore Ts'o, Eric W. Biederman, Dmitry Vyukov, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller, Fengguang Wu On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: > On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote: >> >> Sometimes the branches on linux-next are experimental crap. If someone >> adds an experimental memory allocator to linux-next before discovering >> it causes all kinds of problems I don't want bug reports about my code >> not being able to allocate memory because the memory allocator was bad. >> >> If you don't have the resources to test the individual branches of >> linux-next please just test Linus's tree. That will be much more >> meaningful and productive. > > I have to agree with Eric here, the reason why Fengguang Wu's 0-day > testing robot is much better received by developers is that he does > not test linux-net, but rather individual subsystem git trees and > branches. His test automation also does an automatic bisection > search, and can point at a specific commit --- at which point e-mail > goes out to owner of the subsystem git tree, and to the people who > authored and/or reviewed the guilty commit. > > Dmitry, perhaps you could collaborate with Intel's 0-day testing > folks? They have code which does all of this, and perhaps it can be > leveraged. +Fengguang Please note that in most cases 0-day solves an order of magnitude simpler problem. Build/sparse errors are much faster to find, always possible to precisely bisect and attribute. Yes, for that you just test every commit, bisect and send targeted emails. syzbot only finds runtime bugs, lots of them are related to races and can't be reliably reproduced, bisected, etc. Lots of them are old (e.g. predate KASAN that detects them). But they still can be fixed. In ~half of cases developers fix them looking only at the oops report. The last time I checked 0-day infrastructure was closed source. Fengguang, what do you do with trinity crashes that happen episodically, but you can't reliably reproduce, bisect and attribute? Thanks ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 7:59 ` Dmitry Vyukov @ 2018-01-16 11:19 ` Fengguang Wu 0 siblings, 0 replies; 53+ messages in thread From: Fengguang Wu @ 2018-01-16 11:19 UTC (permalink / raw) To: Dmitry Vyukov Cc: Theodore Ts'o, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller, Ye, Xiaolong, lkp On Tue, Jan 16, 2018 at 08:59:36AM +0100, Dmitry Vyukov wrote: >On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: >> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote: >>> >>> Sometimes the branches on linux-next are experimental crap. If someone >>> adds an experimental memory allocator to linux-next before discovering >>> it causes all kinds of problems I don't want bug reports about my code >>> not being able to allocate memory because the memory allocator was bad. >>> >>> If you don't have the resources to test the individual branches of >>> linux-next please just test Linus's tree. That will be much more >>> meaningful and productive. >> >> I have to agree with Eric here, the reason why Fengguang Wu's 0-day >> testing robot is much better received by developers is that he does >> not test linux-net, but rather individual subsystem git trees and >> branches. His test automation also does an automatic bisection >> search, and can point at a specific commit --- at which point e-mail >> goes out to owner of the subsystem git tree, and to the people who >> authored and/or reviewed the guilty commit. >> >> Dmitry, perhaps you could collaborate with Intel's 0-day testing >> folks? They have code which does all of this, and perhaps it can be >> leveraged. > >+Fengguang > >Please note that in most cases 0-day solves an order of magnitude >simpler problem. Build/sparse errors are much faster to find, always >possible to precisely bisect and attribute. Yes, for that you just >test every commit, bisect and send targeted emails. syzbot only finds >runtime bugs, lots of them are related to races and can't be reliably >reproduced, bisected, etc. Lots of them are old (e.g. predate KASAN >that detects them). But they still can be fixed. In ~half of cases >developers fix them looking only at the oops report. > >The last time I checked 0-day infrastructure was closed source. > >Fengguang, what do you do with trinity crashes that happen >episodically, but you can't reliably reproduce, bisect and attribute? 0-day runs most trinity tests in QEMU machines, which can run massively in parallel. Which means we can afford to bisect them by running up to 1000 boots in each bisect step. Ditto for KASAN errors. Since Xiaolong (CCed) has enabled syzkaller in 0-day, it could in theory utilize the same auto bisect infrastructure. If we can make sure syzkaller runs effectively in the 0-day test farm. Thanks, Fengguang ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 7:12 ` Theodore Ts'o 2018-01-16 7:51 ` Dmitry Vyukov 2018-01-16 7:59 ` Dmitry Vyukov @ 2018-01-16 8:31 ` Dmitry Vyukov 2018-01-16 23:13 ` Theodore Ts'o 2018-01-16 8:34 ` Dmitry Vyukov 3 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-16 8:31 UTC (permalink / raw) To: Theodore Ts'o, Eric W. Biederman, Dmitry Vyukov, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: > I just checked a recent report from the Syzbot, and it's not fixed. > The raw.log file still uses a Content-Type of > Application/Octet-stream. Worse the reproducer C source file has a > content type of Application/Octet-stream instead of the much more sane > Application/text. <Face palm> I will look into using a different mailing system which allows more control over email contents. A quick fix for raw.log will be to rename it to raw.log.txt. Not sure if text/plain repro.c.txt is better than application/octet-stream repro.c... ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 8:31 ` Dmitry Vyukov @ 2018-01-16 23:13 ` Theodore Ts'o 2018-01-17 8:06 ` Dmitry Vyukov 0 siblings, 1 reply; 53+ messages in thread From: Theodore Ts'o @ 2018-01-16 23:13 UTC (permalink / raw) To: Dmitry Vyukov Cc: Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Tue, Jan 16, 2018 at 09:31:26AM +0100, Dmitry Vyukov wrote: > On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: > > I just checked a recent report from the Syzbot, and it's not fixed. > > The raw.log file still uses a Content-Type of > > Application/Octet-stream. Worse the reproducer C source file has a > > content type of Application/Octet-stream instead of the much more sane > > Application/text. <Face palm> > > > I will look into using a different mailing system which allows more > control over email contents. > A quick fix for raw.log will be to rename it to raw.log.txt. Not sure > if text/plain repro.c.txt is better than application/octet-stream > repro.c... My personal opinion is that if there is no way to force the content type except by using magic extensions --- which is super-surprising to me; that seems like a broken API and a feature request bug should be filed against the relevant API --- using repro.c.txt would be the best of bad alternatives. Someone is going to have to exit to a shell to compile the repro, and renaming the filename isn't a big deal. The Mail User Agent I use (mutt) allows me to specify the directory to save the file, and gives me the opportunity to edit the filename, before I save it. So at least for me, it really isn't a big deal for you to use repro.c.txt. Cheers, - Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 23:13 ` Theodore Ts'o @ 2018-01-17 8:06 ` Dmitry Vyukov 2018-01-17 16:53 ` Theodore Ts'o 0 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-17 8:06 UTC (permalink / raw) To: Theodore Ts'o, Dmitry Vyukov, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Wed, Jan 17, 2018 at 12:13 AM, Theodore Ts'o <tytso@mit.edu> wrote: > On Tue, Jan 16, 2018 at 09:31:26AM +0100, Dmitry Vyukov wrote: >> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: >> > I just checked a recent report from the Syzbot, and it's not fixed. >> > The raw.log file still uses a Content-Type of >> > Application/Octet-stream. Worse the reproducer C source file has a >> > content type of Application/Octet-stream instead of the much more sane >> > Application/text. <Face palm> >> >> >> I will look into using a different mailing system which allows more >> control over email contents. >> A quick fix for raw.log will be to rename it to raw.log.txt. Not sure >> if text/plain repro.c.txt is better than application/octet-stream >> repro.c... > > My personal opinion is that if there is no way to force the content > type except by using magic extensions --- which is super-surprising to > me; that seems like a broken API and a feature request bug should be > filed against the relevant API --- using repro.c.txt would be the best > of bad alternatives. Someone is going to have to exit to a shell to > compile the repro, and renaming the filename isn't a big deal. The > Mail User Agent I use (mutt) allows me to specify the directory to > save the file, and gives me the opportunity to edit the filename, > before I save it. So at least for me, it really isn't a big deal for > you to use repro.c.txt. Good. I've made such change, it's now raw.log.txt and repro.c.txt: https://github.com/google/syzkaller/commit/afcb994770d7e0f4b88c623bec76fbdce57d3910 I've also mailed a change to appengine that supports *.c, *.log and *.patch as text/plain extensions. There does not seem to be any major objects to it, but it will probably take some time to be deployed in prod. After that I will rename them back. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-17 8:06 ` Dmitry Vyukov @ 2018-01-17 16:53 ` Theodore Ts'o 2018-01-17 17:16 ` Dmitry Vyukov 0 siblings, 1 reply; 53+ messages in thread From: Theodore Ts'o @ 2018-01-17 16:53 UTC (permalink / raw) To: Dmitry Vyukov Cc: Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Wed, Jan 17, 2018 at 09:06:54AM +0100, Dmitry Vyukov wrote: > I've also mailed a change to appengine that supports *.c, *.log and > *.patch as text/plain extensions. There does not seem to be any major > objects to it, but it will probably take some time to be deployed in > prod. After that I will rename them back. I would think the ideal change would be one where one could specify an optional parameter to the appengine API which would override the default content/type, but adding some additional defaults for various magic extentions is probably a simpler change. - Ted ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-17 16:53 ` Theodore Ts'o @ 2018-01-17 17:16 ` Dmitry Vyukov 2018-01-17 22:27 ` Bernd Petrovitsch 0 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-17 17:16 UTC (permalink / raw) To: Theodore Ts'o, Dmitry Vyukov, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Wed, Jan 17, 2018 at 5:53 PM, Theodore Ts'o <tytso@mit.edu> wrote: > On Wed, Jan 17, 2018 at 09:06:54AM +0100, Dmitry Vyukov wrote: >> I've also mailed a change to appengine that supports *.c, *.log and >> *.patch as text/plain extensions. There does not seem to be any major >> objects to it, but it will probably take some time to be deployed in >> prod. After that I will rename them back. > > I would think the ideal change would be one where one could specify an > optional parameter to the appengine API which would override the > default content/type, but adding some additional defaults > for various magic extentions is probably a simpler change. Agree, but as far as I understand it is specifically restricted. You can see docs here: https://cloud.google.com/appengine/docs/standard/php/mail/mail-with-headers-attachments It also does not allow sending .exe, .com, .bat, etc at all. The support for log/patch/c will at least solve all problems for syzbot. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-17 17:16 ` Dmitry Vyukov @ 2018-01-17 22:27 ` Bernd Petrovitsch 0 siblings, 0 replies; 53+ messages in thread From: Bernd Petrovitsch @ 2018-01-17 22:27 UTC (permalink / raw) To: Dmitry Vyukov, Theodore Ts'o, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Wed, 2018-01-17 at 18:16 +0100, Dmitry Vyukov wrote: [...] > Agree, but as far as I understand it is specifically restricted. You > can see docs here: > https://cloud.google.com/appengine/docs/standard/php/mail/mail-with-headers-attachments ROTFL - as if almost no one at Google know that there are source files of any kind. > It also does not allow sending .exe, .com, .bat, etc at all. > The support for log/patch/c will at least solve all problems for syzbot. Yup, but they should better also add .h, .hpp., .cpp and a lot more from the source code world too. I'm just wondering if it wouldn't be even more safe to use text/plain (instead of application/octet-stream) as the default MIME type if one wants to avoid to be misused to send viruses etc. MfG, Bernd PS: Sry, for somewhat semi-off-topic .... -- Bernd Petrovitsch Email : bernd@petrovitsch.priv.at LUGA : http://www.luga.at ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 7:12 ` Theodore Ts'o ` (2 preceding siblings ...) 2018-01-16 8:31 ` Dmitry Vyukov @ 2018-01-16 8:34 ` Dmitry Vyukov 2018-01-24 16:04 ` Alan Cox 3 siblings, 1 reply; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-16 8:34 UTC (permalink / raw) To: Theodore Ts'o, Eric W. Biederman, Dmitry Vyukov, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: >> Outside of the bugs being considered as considered as security issues, >> the bugs syzbot finds are generally things that don't affect anyone in >> practice. So are very low on the priority of things to get fixed. Not sure why are you saying this, but syzbot has found lots of hundreds of use-after-free's, out-of-bounds, information leaks, deadlocks, vm escapes, etc. They have very direct stability and security impact. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-16 8:34 ` Dmitry Vyukov @ 2018-01-24 16:04 ` Alan Cox 2018-01-24 17:06 ` Eric W. Biederman 0 siblings, 1 reply; 53+ messages in thread From: Alan Cox @ 2018-01-24 16:04 UTC (permalink / raw) To: Dmitry Vyukov Cc: Theodore Ts'o, Eric W. Biederman, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Tue, 16 Jan 2018 09:34:01 +0100 Dmitry Vyukov <dvyukov@google.com> wrote: > On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: > >> Outside of the bugs being considered as considered as security issues, > >> the bugs syzbot finds are generally things that don't affect anyone in > >> practice. So are very low on the priority of things to get fixed. > > Not sure why are you saying this, but syzbot has found lots of > hundreds of use-after-free's, out-of-bounds, information leaks, > deadlocks, vm escapes, etc. They have very direct stability and > security impact. Agreed - there may be some UI and presentation issues but it's found some really nasty little bugs. Alan ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-24 16:04 ` Alan Cox @ 2018-01-24 17:06 ` Eric W. Biederman 2018-01-24 17:32 ` Greg Kroah-Hartman 0 siblings, 1 reply; 53+ messages in thread From: Eric W. Biederman @ 2018-01-24 17:06 UTC (permalink / raw) To: Alan Cox Cc: Dmitry Vyukov, Theodore Ts'o, Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller Alan Cox <gnomes@lxorguk.ukuu.org.uk> writes: > On Tue, 16 Jan 2018 09:34:01 +0100 > Dmitry Vyukov <dvyukov@google.com> wrote: > >> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: >> >> Outside of the bugs being considered as considered as security issues, >> >> the bugs syzbot finds are generally things that don't affect anyone in >> >> practice. So are very low on the priority of things to get fixed. >> >> Not sure why are you saying this, but syzbot has found lots of >> hundreds of use-after-free's, out-of-bounds, information leaks, >> deadlocks, vm escapes, etc. They have very direct stability and >> security impact. > > Agreed - there may be some UI and presentation issues but it's found some > really nasty little bugs. I am not certain it has always really found the bugs it hits. My experience tends towards a bug report with too little information in the Oops to guess what went wrong, that I can not reproduce the issue locally, that the no can reproduce, that was produced on a weird tree, and with a reporter telling you they are only interested in testing fixes. Which is a long way of saying if the UI issues are bad enough the issue can not be identified in the code I am not certain we have actually found a bug. So while I can see lots of potential in syzbot. I can't say if the it is greater potential to get bugs fixed or to annoy developers with complaints they can't do anything about. Eric ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-24 17:06 ` Eric W. Biederman @ 2018-01-24 17:32 ` Greg Kroah-Hartman 0 siblings, 0 replies; 53+ messages in thread From: Greg Kroah-Hartman @ 2018-01-24 17:32 UTC (permalink / raw) To: Eric W. Biederman Cc: Alan Cox, Dmitry Vyukov, Theodore Ts'o, Pavel Machek, Mike Galbraith, LKML, Andrew Morton, Linus Torvalds, syzkaller On Wed, Jan 24, 2018 at 11:06:02AM -0600, Eric W. Biederman wrote: > Alan Cox <gnomes@lxorguk.ukuu.org.uk> writes: > > > On Tue, 16 Jan 2018 09:34:01 +0100 > > Dmitry Vyukov <dvyukov@google.com> wrote: > > > >> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote: > >> >> Outside of the bugs being considered as considered as security issues, > >> >> the bugs syzbot finds are generally things that don't affect anyone in > >> >> practice. So are very low on the priority of things to get fixed. > >> > >> Not sure why are you saying this, but syzbot has found lots of > >> hundreds of use-after-free's, out-of-bounds, information leaks, > >> deadlocks, vm escapes, etc. They have very direct stability and > >> security impact. > > > > Agreed - there may be some UI and presentation issues but it's found some > > really nasty little bugs. > > I am not certain it has always really found the bugs it hits. > > My experience tends towards a bug report with too little information > in the Oops to guess what went wrong, that I can not reproduce the > issue locally, that the no can reproduce, that was produced on a weird > tree, and with a reporter telling you they are only interested in > testing fixes. > > Which is a long way of saying if the UI issues are bad enough the issue > can not be identified in the code I am not certain we have actually > found a bug. > > So while I can see lots of potential in syzbot. I can't say if the it > is greater potential to get bugs fixed or to annoy developers with > complaints they can't do anything about. I'm with Alan here, syzbot has found lots of nasty bugs in the areas of the kernel I maintain. Many of which are still on my TODO list to fix :) So yes, it's annoying to me at times as well, but it is good work here, and I hope to see it continue. greg k-h ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 16:38 ` Eric W. Biederman 2018-01-16 7:12 ` Theodore Ts'o @ 2018-01-16 18:20 ` Dmitry Vyukov 2018-01-17 8:13 ` Dmitry Vyukov 2 siblings, 0 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-16 18:20 UTC (permalink / raw) To: Eric W. Biederman Cc: Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Mon, Jan 15, 2018 at 5:38 PM, Eric W. Biederman <ebiederm@xmission.com> wrote: > When I made the complaint it came to me and to messages on lkml as > .log. With Content-Type: Application/Octent-stream. Where was that? If I am not mistaken you actually didn't. I triple-checked my inbox and searched internet. The only references to Content-Type from you I can find are in this thread. And the are "Like Content-type" and "Not content-type in the emails". I simply was not aware of the problem, and I did not understand these comment because they don't explain the problem. The only discussion that happened around Content-Type is this: https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/7DjzSA2jCgAJ This is proposal to give files .syz extension, which does not look right to me and it a different topic anyway. > That is a bloody mess that wastes peoples time. If it is fixed good, > it certainly was not fixed at that point. > > But it is much more than any single issue. You get defensive when > people critisize syzbot. Instead of recognizing it's failings. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-15 16:38 ` Eric W. Biederman 2018-01-16 7:12 ` Theodore Ts'o 2018-01-16 18:20 ` Dmitry Vyukov @ 2018-01-17 8:13 ` Dmitry Vyukov 2 siblings, 0 replies; 53+ messages in thread From: Dmitry Vyukov @ 2018-01-17 8:13 UTC (permalink / raw) To: Eric W. Biederman Cc: Pavel Machek, Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller On Mon, Jan 15, 2018 at 5:38 PM, Eric W. Biederman <ebiederm@xmission.com> wrote: > I am definitely not asking for expertise in the kernel. I am asking > for a human who wants to help track down bugs in the kernel. What is that additional information that you need and that syzbot currently does not provide? ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: LKML admins (syzbot emails are not delivered) 2018-01-04 11:09 ` Dmitry Vyukov 2018-01-04 11:18 ` Pavel Machek 2018-01-04 15:23 ` Eric W. Biederman @ 2018-01-04 21:38 ` Pavel Machek 2 siblings, 0 replies; 53+ messages in thread From: Pavel Machek @ 2018-01-04 21:38 UTC (permalink / raw) To: Dmitry Vyukov Cc: Mike Galbraith, LKML, Greg Kroah-Hartman, Andrew Morton, Linus Torvalds, syzkaller [-- Attachment #1: Type: text/plain, Size: 2321 bytes --] On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote: > On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote: > >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote: > >> > Hi! > >> > > > >> > > Some of syzbot emails don't appear on LKML mailing lists, while they > >> > > were mailed as any other emails. Here are few examples: > >> > > > >> > > "KASAN: use-after-free Read in rds_tcp_dev_event" > >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ > >> > > > >> > > "general protection fault in __wake_up_common" > >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ > >> > > > >> > > Does anybody know how to get in contact with real people behind LKML > >> > > and/or bugzilla? > >> > > >> > Not delivering syzbot emails might be good thing? > >> > >> Nah, the thing is finding and reporting bugs just like a human would, > >> it just doesn't need sleep etc, so sometimes reports more than humans > >> can keep up with. It needs a smarter brother.. but then again, maybe > >> not, if bots start fixing things too, a lot of meatware hackers would > >> have to go find real jobs. > > > > Sending random, unrepeatable Oopses to lkml is not what humans would > > do, and perhaps not something bots should do, either. > > > Hi Pavel, > > I've answered this question here in full detail. In short, this is > useful and actionable. > https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ Unfortantely, some of the reports are making it to the lkml. 200KB each! Due to attached .config files. I believe you should not be spamming lkml in the first place. If you insist on sending to lkml and list admins are okay with that, please gzip the config file. Pavel > do_syscall_32_irqs_on arch/x86/entry/common.c:327 [inline] > do_fast_syscall_32+0x3ee/0xf9d arch/x86/entry/common.c:389 > entry_SYSENTER_compat+0x54/0x63 arch/x86/entry/entry_64_compat.S:129 Looks like the same bug I sent out a fix for yesterday. #syz fix: capabilities: fix buffer overread on very short xattr https://marc.info/?l=linux-kernel&m=151488700301705 -- (english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 181 bytes --] ^ permalink raw reply [flat|nested] 53+ messages in thread
end of thread, other threads:[~2018-03-12 14:03 UTC | newest] Thread overview: 53+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-01-04 9:09 LKML admins (syzbot emails are not delivered) Dmitry Vyukov 2018-01-04 9:23 ` Greg Kroah-Hartman 2018-01-04 9:56 ` Ozgur 2018-01-04 15:31 ` David Miller 2018-01-04 19:11 ` Ozgur 2018-01-04 11:03 ` Dmitry Vyukov 2018-01-04 11:04 ` Dmitry Vyukov 2018-01-04 11:20 ` Greg Kroah-Hartman 2018-01-04 15:35 ` David Miller 2018-01-15 9:43 ` Dmitry Vyukov 2018-03-01 16:22 ` Dmitry Vyukov 2018-03-08 20:03 ` David Miller 2018-03-12 8:56 ` Dmitry Vyukov 2018-03-12 14:03 ` David Miller 2018-01-04 23:50 ` Theodore Ts'o 2018-01-08 12:01 ` Dmitry Vyukov 2018-01-08 12:11 ` Dmitry Vyukov 2018-01-04 9:25 ` Pavel Machek 2018-01-04 9:38 ` Mike Galbraith 2018-01-04 9:56 ` Pavel Machek 2018-01-04 11:09 ` Dmitry Vyukov 2018-01-04 11:18 ` Pavel Machek 2018-01-15 10:08 ` Dmitry Vyukov 2018-01-15 13:08 ` Pavel Machek 2018-01-15 13:38 ` Dmitry Vyukov 2018-01-15 13:53 ` Pavel Machek 2018-01-04 15:23 ` Eric W. Biederman 2018-01-15 10:54 ` Dmitry Vyukov 2018-01-15 12:54 ` Pavel Machek 2018-01-15 13:02 ` Dmitry Vyukov 2018-01-15 16:16 ` Eric W. Biederman 2018-01-16 18:01 ` Dmitry Vyukov 2018-01-15 16:38 ` Eric W. Biederman 2018-01-16 7:12 ` Theodore Ts'o 2018-01-16 7:51 ` Dmitry Vyukov 2018-01-16 9:52 ` Guenter Roeck 2018-01-16 9:56 ` Dmitry Vyukov 2018-01-16 9:58 ` Guenter Roeck 2018-01-16 7:59 ` Dmitry Vyukov 2018-01-16 11:19 ` Fengguang Wu 2018-01-16 8:31 ` Dmitry Vyukov 2018-01-16 23:13 ` Theodore Ts'o 2018-01-17 8:06 ` Dmitry Vyukov 2018-01-17 16:53 ` Theodore Ts'o 2018-01-17 17:16 ` Dmitry Vyukov 2018-01-17 22:27 ` Bernd Petrovitsch 2018-01-16 8:34 ` Dmitry Vyukov 2018-01-24 16:04 ` Alan Cox 2018-01-24 17:06 ` Eric W. Biederman 2018-01-24 17:32 ` Greg Kroah-Hartman 2018-01-16 18:20 ` Dmitry Vyukov 2018-01-17 8:13 ` Dmitry Vyukov 2018-01-04 21:38 ` Pavel Machek
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.