LKML admins (syzbot emails are not delivered)

LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

Hello,

Some of syzbot emails don't appear on LKML mailing lists, while they
were mailed as any other emails. Here are few examples:

"KASAN: use-after-free Read in rds_tcp_dev_event"
https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ

"general protection fault in __wake_up_common"
https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ

Does anybody know how to get in contact with real people behind LKML
and/or bugzilla?

I am trying to understand why this happens, but failed so far (it does
not do any obviously prohibited stuff, and replies to these emails are
delivered). I tried to use autoanswer@vger.kernel.org (which is
referenced from http://vger.kernel.org/majordomo-info.html), but it
now always return:

554 5.0.0 Hi [209.85.192.182], unresolvable address:
<autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org

I failed to find any admin email referenced anywhere.

On a related note, I also tried to contact bugzilla admins via
rt.linuxfoundation.org. But there is complete silence. Does anybody
know how to get it touch with these people?

Thanks in advance

Re: LKML admins (syzbot emails are not delivered)

[Greg Kroah-Hartman]

On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote:
> Hello,
> 
> Some of syzbot emails don't appear on LKML mailing lists, while they
> were mailed as any other emails. Here are few examples:
> 
> "KASAN: use-after-free Read in rds_tcp_dev_event"
> https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> 
> "general protection fault in __wake_up_common"
> https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> 
> Does anybody know how to get in contact with real people behind LKML
> and/or bugzilla?
> 
> I am trying to understand why this happens, but failed so far (it does
> not do any obviously prohibited stuff, and replies to these emails are
> delivered).

You should get a bounce notice from vger if the email is being rejected.
If not, you might be on the spam filter list, which is listed on vger,
did you check that?

> I tried to use autoanswer@vger.kernel.org (which is
> referenced from http://vger.kernel.org/majordomo-info.html), but it
> now always return:
> 
> 554 5.0.0 Hi [209.85.192.182], unresolvable address:
> <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org

autoanswer is not for admin requests.

> I failed to find any admin email referenced anywhere.

Look a bit harder, like at the bottom of this page:
	http://vger.kernel.org/majordomo-info.html

:)

> On a related note, I also tried to contact bugzilla admins via
> rt.linuxfoundation.org. But there is complete silence. Does anybody
> know how to get it touch with these people?

Did you get an answer back from the rt system?  If not, it did not go
through, and the help address might have changed.  I can dig it up if
so...

thanks,

greg k-h

Re: LKML admins (syzbot emails are not delivered)

[Ozgur]

04.01.2018, 12:23, "Greg Kroah-Hartman" <gregkh@linuxfoundation.org>:
> On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote:
>>  Hello,
>>
>>  Some of syzbot emails don't appear on LKML mailing lists, while they
>>  were mailed as any other emails. Here are few examples:
>>
>>  "KASAN: use-after-free Read in rds_tcp_dev_event"
>>  https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
>>
>>  "general protection fault in __wake_up_common"
>>  https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
>>
>>  Does anybody know how to get in contact with real people behind LKML
>>  and/or bugzilla?
>>
>>  I am trying to understand why this happens, but failed so far (it does
>>  not do any obviously prohibited stuff, and replies to these emails are
>>  delivered).
>
> You should get a bounce notice from vger if the email is being rejected.
> If not, you might be on the spam filter list, which is listed on vger,
> did you check that?
>
>>  I tried to use autoanswer@vger.kernel.org (which is
>>  referenced from http://vger.kernel.org/majordomo-info.html), but it
>>  now always return:
>>
>>  554 5.0.0 Hi [209.85.192.182], unresolvable address:
>>  <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org

Hello,

autoanswer is just automated reply address that the lmkl system works and your email arrives.
LKML e-mail list implemented SPF, DKIM and DMARC and I think some domains were forbid.
For example yandex.ru or mail.ru

I think should add David (Miller), I added.

> autoanswer is not for admin requests.
>
>>  I failed to find any admin email referenced anywhere.
>
> Look a bit harder, like at the bottom of this page:
>         http://vger.kernel.org/majordomo-info.html
>
> :)
>
>>  On a related note, I also tried to contact bugzilla admins via
>>  rt.linuxfoundation.org. But there is complete silence. Does anybody
>>  know how to get it touch with these people?

And please check MX:

http://vger.kernel.org/mxverify.html

Ozgur

> Did you get an answer back from the rt system? If not, it did not go
> through, and the help address might have changed. I can dig it up if
> so...
>
> thanks,
>
> greg k-h

Re: LKML admins (syzbot emails are not delivered)

[David Miller]

From: Ozgur <ozgur@goosey.org>
Date: Thu, 04 Jan 2018 12:56:56 +0300

> autoanswer is just automated reply address that the lmkl system works and your email arrives.
> LKML e-mail list implemented SPF, DKIM and DMARC and I think some domains were forbid.
> For example yandex.ru or mail.ru

If I am given specific examples of postings that don't arrive from this point forward
I can look into them.

But I have to be alerted very quickly after the face rather than a day or so later.

I also only process vger.kernel.org bounces and postmaster email about once, maybe
twice per day.  So please keep this in mind in your expectations.

Thank you.

Re: LKML admins (syzbot emails are not delivered)

[Ozgur]

04.01.2018, 18:31, "David Miller" <davem@davemloft.net>:
> From: Ozgur <ozgur@goosey.org>
> Date: Thu, 04 Jan 2018 12:56:56 +0300

Dvyukov,

I think you will set a bot sensitive and syzbot send e-mail that it doesn't disturb list members :)
David is sometimes nervous.

Ozgur

>>  autoanswer is just automated reply address that the lmkl system works and your email arrives.
>>  LKML e-mail list implemented SPF, DKIM and DMARC and I think some domains were forbid.
>>  For example yandex.ru or mail.ru
>
> If I am given specific examples of postings that don't arrive from this point forward
> I can look into them.
>
> But I have to be alerted very quickly after the face rather than a day or so later.
>
> I also only process vger.kernel.org bounces and postmaster email about once, maybe
> twice per day. So please keep this in mind in your expectations.
>
> Thank you.

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

[-- Attachment #1: Type: text/plain, Size: 2871 bytes --]

On Thu, Jan 4, 2018 at 10:23 AM, Greg Kroah-Hartman <
gregkh@linuxfoundation.org> wrote:

> On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote:
> > Hello,
> >
> > Some of syzbot emails don't appear on LKML mailing lists, while they
> > were mailed as any other emails. Here are few examples:
> >
> > "KASAN: use-after-free Read in rds_tcp_dev_event"
> > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> >
> > "general protection fault in __wake_up_common"
> > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> >
> > Does anybody know how to get in contact with real people behind LKML
> > and/or bugzilla?
> >
> > I am trying to understand why this happens, but failed so far (it does
> > not do any obviously prohibited stuff, and replies to these emails are
> > delivered).
>
> You should get a bounce notice from vger if the email is being rejected.
>

The problem is that it's not _me_, it's a computer program which uses some
mail delivery system which I don't have full visibility into. I don't know
if it even gets bounce emails (as far as I understand it's not LKML that
generates them, LKML SMTP server just returns some error code and then it's
a responsibility of somebody else to represent it by a reply email). If the
only way to probe the behavior is to send actual emails to LKML (which have
high chances to be actually delivered to all subscribers), it makes testing
somewhat problematic.




> If not, you might be on the spam filter list, which is listed on vger,
> did you check that?


I've looked at it manually and did not find any matches. Again, a reply
quoting full original text was delivered.




> > I tried to use autoanswer@vger.kernel.org (which is
> > referenced from http://vger.kernel.org/majordomo-info.html), but it
> > now always return:
> >
> > 554 5.0.0 Hi [209.85.192.182], unresolvable address:
> > <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org
>
> autoanswer is not for admin requests.
>


Yes, I understand. But that would be a perfect way to probe behavior and
look at bounce messages. The problem is that it does not seem to work at
all.



> > I failed to find any admin email referenced anywhere.
>
> Look a bit harder, like at the bottom of this page:
>         http://vger.kernel.org/majordomo-info.html
>
> :)



I will try that one. Thanks.



>
> > On a related note, I also tried to contact bugzilla admins via
> > rt.linuxfoundation.org. But there is complete silence. Does anybody
> > know how to get it touch with these people?
>
> Did you get an answer back from the rt system?  If not, it did not go
> through, and the help address might have changed.  I can dig it up if
> so...
>


Yes, the ticket was filed as:
https://rt.linuxfoundation.org/SelfService/Display.html?id=50245
(so emailing helpdesk@kernel.org part worked fine)

[-- Attachment #2: Type: text/html, Size: 5631 bytes --]

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Thu, Jan 4, 2018 at 10:23 AM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote:
> > Hello,
> >
> > Some of syzbot emails don't appear on LKML mailing lists, while they
> > were mailed as any other emails. Here are few examples:
> >
> > "KASAN: use-after-free Read in rds_tcp_dev_event"
> > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> >
> > "general protection fault in __wake_up_common"
> > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> >
> > Does anybody know how to get in contact with real people behind LKML
> > and/or bugzilla?
> >
> > I am trying to understand why this happens, but failed so far (it does
> > not do any obviously prohibited stuff, and replies to these emails are
> > delivered).
>
> You should get a bounce notice from vger if the email is being rejected.


The problem is that it's not _me_, it's a computer program which uses
some mail delivery system which I don't have full visibility into. I
don't know if it even gets bounce emails (as far as I understand it's
not LKML that generates them, LKML SMTP server just returns some error
code and then it's a responsibility of somebody else to represent it
by a reply email). If the only way to probe the behavior is to send
actual emails to LKML (which have high chances to be actually
delivered to all subscribers), it makes testing somewhat problematic.



>
> If not, you might be on the spam filter list, which is listed on vger,
> did you check that?


I've looked at it manually and did not find any matches. Again, a
reply quoting full original text was delivered.



>
> > I tried to use autoanswer@vger.kernel.org (which is
> > referenced from http://vger.kernel.org/majordomo-info.html), but it
> > now always return:
> >
> > 554 5.0.0 Hi [209.85.192.182], unresolvable address:
> > <autoanswer@vger.kernel.org>; nosuchuser; autoanswer@vger.kernel.org
>
> autoanswer is not for admin requests.



Yes, I understand. But that would be a perfect way to probe behavior
and look at bounce messages. The problem is that it does not seem to
work at all.


>
> > I failed to find any admin email referenced anywhere.
>
> Look a bit harder, like at the bottom of this page:
>         http://vger.kernel.org/majordomo-info.html
>
> :)



I will try that one. Thanks.


>
>
> > On a related note, I also tried to contact bugzilla admins via
> > rt.linuxfoundation.org. But there is complete silence. Does anybody
> > know how to get it touch with these people?
>
> Did you get an answer back from the rt system?  If not, it did not go
> through, and the help address might have changed.  I can dig it up if
> so...



Yes, the ticket was filed as:
https://rt.linuxfoundation.org/SelfService/Display.html?id=50245
(so emailing helpdesk@kernel.org part worked fine)

Re: LKML admins (syzbot emails are not delivered)

[Greg Kroah-Hartman]

On Thu, Jan 04, 2018 at 12:04:34PM +0100, Dmitry Vyukov wrote:
> On Thu, Jan 4, 2018 at 10:23 AM, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote:
> > > Hello,
> > >
> > > Some of syzbot emails don't appear on LKML mailing lists, while they
> > > were mailed as any other emails. Here are few examples:
> > >
> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> > >
> > > "general protection fault in __wake_up_common"
> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> > >
> > > Does anybody know how to get in contact with real people behind LKML
> > > and/or bugzilla?
> > >
> > > I am trying to understand why this happens, but failed so far (it does
> > > not do any obviously prohibited stuff, and replies to these emails are
> > > delivered).
> >
> > You should get a bounce notice from vger if the email is being rejected.
> 
> 
> The problem is that it's not _me_, it's a computer program which uses
> some mail delivery system which I don't have full visibility into.

But you, or someone, should have access to that email address to see any
responses sent to it.  If no one can, then it's just a spam bot and if I
were vger's postmaster, I would blacklist it as well :)

Good luck!

greg k-h

Re: LKML admins (syzbot emails are not delivered)

[David Miller]

From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date: Thu, 4 Jan 2018 12:20:28 +0100

> If no one can, then it's just a spam bot and if I were vger's
> postmaster, I would blacklist it as well :)

+1

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Thu, Jan 4, 2018 at 4:35 PM, David Miller <davem@davemloft.net> wrote:
> From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> Date: Thu, 4 Jan 2018 12:20:28 +0100
>
>> If no one can, then it's just a spam bot and if I were vger's
>> postmaster, I would blacklist it as well :)
>
> +1


Just to make it clear, humans were reading all normal communication
related to bugs from day 1. It was just bounce emails. Now we are
receiving bounce emails as well. So this is resolved.

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Thu, Jan 4, 2018 at 12:20 PM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>> > On Thu, Jan 04, 2018 at 10:09:16AM +0100, Dmitry Vyukov wrote:
>> > > Hello,
>> > >
>> > > Some of syzbot emails don't appear on LKML mailing lists, while they
>> > > were mailed as any other emails. Here are few examples:
>> > >
>> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
>> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
>> > >
>> > > "general protection fault in __wake_up_common"
>> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
>> > >
>> > > Does anybody know how to get in contact with real people behind LKML
>> > > and/or bugzilla?
>> > >
>> > > I am trying to understand why this happens, but failed so far (it does
>> > > not do any obviously prohibited stuff, and replies to these emails are
>> > > delivered).
>> >
>> > You should get a bounce notice from vger if the email is being rejected.
>>
>>
>> The problem is that it's not _me_, it's a computer program which uses
>> some mail delivery system which I don't have full visibility into.
>
> But you, or someone, should have access to that email address to see any
> responses sent to it.  If no one can, then it's just a spam bot and if I
> were vger's postmaster, I would blacklist it as well :)
>
> Good luck!
>
> greg k-h


The problem strikes back.
Again people complain that a email does not appear on LKML:
https://www.spinics.net/lists/kernel/msg2735905.html
Though, it appeared on syzkaller-bugs group:
https://groups.google.com/d/msg/syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ
https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ
syzbot now receives and logs bounce notifications, but there were none...

FWIW I've filed support ticket
https://rt.linuxfoundation.org/SelfService/Display.html?id=53105

Re: LKML admins (syzbot emails are not delivered)

[David Miller]

From: Dmitry Vyukov <dvyukov@google.com>
Date: Thu, 1 Mar 2018 17:22:54 +0100

> The problem strikes back.
> Again people complain that a email does not appear on LKML:
> https://www.spinics.net/lists/kernel/msg2735905.html
> Though, it appeared on syzkaller-bugs group:
> https://groups.google.com/d/msg/syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ
> https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ
> syzbot now receives and logs bounce notifications, but there were none...

The problem is that the name 'kernelci.orgbot' has a special character
in it, namely "."

And when such special characters occur in email headers, the string
in question must be surrounded by double quotes.

People have to do this when they use my name "David S. Miller" as
well, for example.

So I think if you put double quotes around kernelci.orgbot when it
is provided in email headers, the problem will go away.

Please also make sure that multiple From: headers are not being
generated or anything like that.

Thanks.

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Thu, Mar 8, 2018 at 9:03 PM, David Miller <davem@davemloft.net> wrote:
> From: Dmitry Vyukov <dvyukov@google.com>
> Date: Thu, 1 Mar 2018 17:22:54 +0100
>
>> The problem strikes back.
>> Again people complain that a email does not appear on LKML:
>> https://www.spinics.net/lists/kernel/msg2735905.html
>> Though, it appeared on syzkaller-bugs group:
>> https://groups.google.com/d/msg/syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ
>> https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ
>> syzbot now receives and logs bounce notifications, but there were none...
>
> The problem is that the name 'kernelci.orgbot' has a special character
> in it, namely "."
>
> And when such special characters occur in email headers, the string
> in question must be surrounded by double quotes.
>
> People have to do this when they use my name "David S. Miller" as
> well, for example.
>
> So I think if you put double quotes around kernelci.orgbot when it
> is provided in email headers, the problem will go away.
>
> Please also make sure that multiple From: headers are not being
> generated or anything like that.


Thanks for looking into this, David!

But I don't see kernelci.orgbot referenced anywhere in the emails.
E.g. in the one that was linked above:
https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ

Which one do you mean? syzbot takes emails from get_maintainers.pl, I
guess kernelci.orgbot is not usually there. And syzbot does not
includes names, just email addresses.
Could a dot in email address produce the same result? E.g.
To: foo.bar@something.com
? I see such addresses in some of the emails (but not in the one
referenced above).

Is it possible to revive autoanswer@vger.kernel.org? It would
hopefully allow us to debug this without bothering you.


Possibly related, when I am searching for "syzkaller" I am finding
entries like the following titled "This message generated a parse
failure. Raw output follows here. Please use 'back' to navigate.".

https://lkml.org/lkml/2018/3/10/73
https://lkml.org/lkml/2018/2/27/581
https://lkml.org/lkml/2018/2/22/774

Could this "parse failure" explain the undelivered emails? How can I
figure out what exactly caused the failure?

Thanks again

Re: LKML admins (syzbot emails are not delivered)

[David Miller]

From: Dmitry Vyukov <dvyukov@google.com>
Date: Mon, 12 Mar 2018 09:56:42 +0100

> But I don't see kernelci.orgbot referenced anywhere in the emails.
> E.g. in the one that was linked above:
> https://groups.google.com/forum/message/raw?msg=syzkaller-bugs/OkwLGoS6UQA/hHAtCdnNBgAJ

I'm not referencing that one above, I'm simply reporting what I've
seen in the vger bounce logs.

> Could a dot in email address produce the same result? E.g.
> To: foo.bar@something.com

No.

> Is it possible to revive autoanswer@vger.kernel.org? It would
> hopefully allow us to debug this without bothering you.

Sorry, no engineering resources will be put into the current
setup as we prepare a new system based upon mailman.

Re: LKML admins (syzbot emails are not delivered)

[Theodore Ts'o]

On Thu, Jan 04, 2018 at 12:04:34PM +0100, Dmitry Vyukov wrote:
> 
> The problem is that it's not _me_, it's a computer program which uses
> some mail delivery system which I don't have full visibility into. I
> don't know if it even gets bounce emails (as far as I understand it's
> not LKML that generates them, LKML SMTP server just returns some error
> code and then it's a responsibility of somebody else to represent it
> by a reply email). If the only way to probe the behavior is to send
> actual emails to LKML (which have high chances to be actually
> delivered to all subscribers), it makes testing somewhat problematic.

It looks like you're using the App Engine Mail API.  You can configure
it to get bounce e-mails[1].  From what I can tell looking at the mail
headers, the mail gets sent via an intermediate SMTP host, such as
mail-it0-f69.google.com before it is delievered to vger.  So if vger's
mailer bounces it via an SMTP error, it would be
mail-it0-f69.google.com's responsibility to generate a bounce message,
which you then should be able to catch.

[1] https://cloud.google.com/appengine/docs/standard/go/mail/bounce

You should be able to test this by having your app-engine send a
message to "invalid-address@vger.kernel.org".  I've verified that this
will cause an immediate SMTP error:

554 5.0.0 Hi [74.207.234.97], unresolvable address: <invalid-address@vger.kernel.org>; nosuchuser; invalid-address@vger.kernel.org

If it doesn't, you should file an internal bug report since that's
clearly a bug in the App Engine's mail infrastructure.  If you can't
get satisfaction that way, my recommendation would be to set up an
Linode server (you can't use GCE because GCE blocks outgoing SMTP
connections) to be your mail gateway, and send the notifications from
a domain such as syzkaller.org, where you have full control of your
mail infrastructure, and you don't have to deal with nonsense such as
DMARC.


It also seems to me, looking at other complaints on this thread, that
there is the opportunity for the syzbot to do much more.  For example,
you can see if it repro's on the last released mainline kernel (such
as 4.14) and if so, have the syzbot automatically do a bisection
search, so you can make sure the report goes to the best set of
developers to fix it, a pointer to the guilty commit.

Cheers,

					- Ted

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Fri, Jan 5, 2018 at 12:50 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Thu, Jan 04, 2018 at 12:04:34PM +0100, Dmitry Vyukov wrote:
>>
>> The problem is that it's not _me_, it's a computer program which uses
>> some mail delivery system which I don't have full visibility into. I
>> don't know if it even gets bounce emails (as far as I understand it's
>> not LKML that generates them, LKML SMTP server just returns some error
>> code and then it's a responsibility of somebody else to represent it
>> by a reply email). If the only way to probe the behavior is to send
>> actual emails to LKML (which have high chances to be actually
>> delivered to all subscribers), it makes testing somewhat problematic.
>
> It looks like you're using the App Engine Mail API.  You can configure
> it to get bounce e-mails[1].  From what I can tell looking at the mail
> headers, the mail gets sent via an intermediate SMTP host, such as
> mail-it0-f69.google.com before it is delievered to vger.  So if vger's
> mailer bounces it via an SMTP error, it would be
> mail-it0-f69.google.com's responsibility to generate a bounce message,
> which you then should be able to catch.
>
> [1] https://cloud.google.com/appengine/docs/standard/go/mail/bounce
>
> You should be able to test this by having your app-engine send a
> message to "invalid-address@vger.kernel.org".  I've verified that this
> will cause an immediate SMTP error:
>
> 554 5.0.0 Hi [74.207.234.97], unresolvable address: <invalid-address@vger.kernel.org>; nosuchuser; invalid-address@vger.kernel.org
>
> If it doesn't, you should file an internal bug report since that's
> clearly a bug in the App Engine's mail infrastructure.  If you can't
> get satisfaction that way, my recommendation would be to set up an
> Linode server (you can't use GCE because GCE blocks outgoing SMTP
> connections) to be your mail gateway, and send the notifications from
> a domain such as syzkaller.org, where you have full control of your
> mail infrastructure, and you don't have to deal with nonsense such as
> DMARC.


Thanks, Ted! This is helpful. I've added handler for bounce emails:
https://github.com/google/syzkaller/commit/19c05fffcb1860b2dcf17989b40ca16ed259fdea
And I do see them after sending to invalid-address@vger.kernel.org. So
now hopefully it will sched some light when we get another bounce.

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Fri, Jan 5, 2018 at 12:50 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> It also seems to me, looking at other complaints on this thread, that
> there is the opportunity for the syzbot to do much more.  For example,
> you can see if it repro's on the last released mainline kernel (such
> as 4.14) and if so, have the syzbot automatically do a bisection
> search, so you can make sure the report goes to the best set of
> developers to fix it, a pointer to the guilty commit.

Hi Ted,

I've filed https://github.com/google/syzkaller/issues/501 for this as
this come up several times. There is bunch of problems, though:

 - unreliable reproducers
 - triggering of unrelated bugs (happens frequently)
 - flakes (bugs in GCE, crashes in tty on first ssh connection, etc)
 - bugs (races) that manifest in multiple different ways
 - bugs that will be attributed to tools improvements (e.g. KASAN,
fault injection improvements)
 - reproducers that need slight changes on different kernel revisions

Not sure what quality of bisection we can achieve. And kernel
developers tend to be negative to any kind of bot gaffe, so we can
lose both ways.

Re: LKML admins (syzbot emails are not delivered)

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

Hi!
> 
> Some of syzbot emails don't appear on LKML mailing lists, while they
> were mailed as any other emails. Here are few examples:
> 
> "KASAN: use-after-free Read in rds_tcp_dev_event"
> https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> 
> "general protection fault in __wake_up_common"
> https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> 
> Does anybody know how to get in contact with real people behind LKML
> and/or bugzilla?

Not delivering syzbot emails might be good thing?

You claimed you generate so many of them that you can't even check
them by hand, so what makes you think thousands of lkml readers want
to delete them by hand?

You may want to talk to David Miller.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: LKML admins (syzbot emails are not delivered)

[Mike Galbraith]

On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
> Hi!
> > 
> > Some of syzbot emails don't appear on LKML mailing lists, while they
> > were mailed as any other emails. Here are few examples:
> > 
> > "KASAN: use-after-free Read in rds_tcp_dev_event"
> > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> > 
> > "general protection fault in __wake_up_common"
> > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> > 
> > Does anybody know how to get in contact with real people behind LKML
> > and/or bugzilla?
> 
> Not delivering syzbot emails might be good thing?

Nah, the thing is finding and reporting bugs just like a human would,
it just doesn't need sleep etc, so sometimes reports more than humans
can keep up with.  It needs a smarter brother.. but then again, maybe
not, if bots start fixing things too, a lot of meatware hackers would
have to go find real jobs.

> You claimed you generate so many of them that you can't even check
> them by hand, so what makes you think thousands of lkml readers want
> to delete them by hand?

Mail filters have existed for untold ages :)

	-Mike

Re: LKML admins (syzbot emails are not delivered)

[Pavel Machek]

> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
> > Hi!
> > > 
> > > Some of syzbot emails don't appear on LKML mailing lists, while they
> > > were mailed as any other emails. Here are few examples:
> > > 
> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> > > 
> > > "general protection fault in __wake_up_common"
> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> > > 
> > > Does anybody know how to get in contact with real people behind LKML
> > > and/or bugzilla?
> > 
> > Not delivering syzbot emails might be good thing?
> 
> Nah, the thing is finding and reporting bugs just like a human would,
> it just doesn't need sleep etc, so sometimes reports more than humans
> can keep up with.  It needs a smarter brother.. but then again, maybe
> not, if bots start fixing things too, a lot of meatware hackers would
> have to go find real jobs.

Sending random, unrepeatable Oopses to lkml is not what humans would
do, and perhaps not something bots should do, either.

> > You claimed you generate so many of them that you can't even check
> > them by hand, so what makes you think thousands of lkml readers want
> > to delete them by hand?
> 
> Mail filters have existed for untold ages :)

Yeah, well, spammers also existed for long long time :-).

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote:
>> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
>> > Hi!
>> > >
>> > > Some of syzbot emails don't appear on LKML mailing lists, while they
>> > > were mailed as any other emails. Here are few examples:
>> > >
>> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
>> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
>> > >
>> > > "general protection fault in __wake_up_common"
>> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
>> > >
>> > > Does anybody know how to get in contact with real people behind LKML
>> > > and/or bugzilla?
>> >
>> > Not delivering syzbot emails might be good thing?
>>
>> Nah, the thing is finding and reporting bugs just like a human would,
>> it just doesn't need sleep etc, so sometimes reports more than humans
>> can keep up with.  It needs a smarter brother.. but then again, maybe
>> not, if bots start fixing things too, a lot of meatware hackers would
>> have to go find real jobs.
>
> Sending random, unrepeatable Oopses to lkml is not what humans would
> do, and perhaps not something bots should do, either.


Hi Pavel,

I've answered this question here in full detail. In short, this is
useful and actionable.
https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ

Re: LKML admins (syzbot emails are not delivered)

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2312 bytes --]

On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote:
> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote:
> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
> >> > Hi!
> >> > >
> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they
> >> > > were mailed as any other emails. Here are few examples:
> >> > >
> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> >> > >
> >> > > "general protection fault in __wake_up_common"
> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> >> > >
> >> > > Does anybody know how to get in contact with real people behind LKML
> >> > > and/or bugzilla?
> >> >
> >> > Not delivering syzbot emails might be good thing?
> >>
> >> Nah, the thing is finding and reporting bugs just like a human would,
> >> it just doesn't need sleep etc, so sometimes reports more than humans
> >> can keep up with.  It needs a smarter brother.. but then again, maybe
> >> not, if bots start fixing things too, a lot of meatware hackers would
> >> have to go find real jobs.
> >
> > Sending random, unrepeatable Oopses to lkml is not what humans would
> > do, and perhaps not something bots should do, either.
> 
> 
> Hi Pavel,
> 
> I've answered this question here in full detail. In short, this is
> useful and actionable.
> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ

I've already deleted many such reports from my lkml folder. It
definitely is below quality of normal bug reports.

You want all the credit while doing none of the work:

This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzk...@googlegroups.com.
Please credit me with: Reported-by: syzbot <syzk...@googlegroups.com>
...
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug
report.

I'd say this belongs to separate list. Interested parties can
subscribe, and you can still manually forward reports if you handle
duplicities etc.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Thu, Jan 4, 2018 at 12:18 PM, Pavel Machek <pavel@ucw.cz> wrote:
> On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote:
>> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote:
>> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
>> >> > Hi!
>> >> > >
>> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they
>> >> > > were mailed as any other emails. Here are few examples:
>> >> > >
>> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
>> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
>> >> > >
>> >> > > "general protection fault in __wake_up_common"
>> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
>> >> > >
>> >> > > Does anybody know how to get in contact with real people behind LKML
>> >> > > and/or bugzilla?
>> >> >
>> >> > Not delivering syzbot emails might be good thing?
>> >>
>> >> Nah, the thing is finding and reporting bugs just like a human would,
>> >> it just doesn't need sleep etc, so sometimes reports more than humans
>> >> can keep up with.  It needs a smarter brother.. but then again, maybe
>> >> not, if bots start fixing things too, a lot of meatware hackers would
>> >> have to go find real jobs.
>> >
>> > Sending random, unrepeatable Oopses to lkml is not what humans would
>> > do, and perhaps not something bots should do, either.
>>
>>
>> Hi Pavel,
>>
>> I've answered this question here in full detail. In short, this is
>> useful and actionable.
>> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ
>
> I've already deleted many such reports from my lkml folder. It
> definitely is below quality of normal bug reports.


Pavel, if you point to exact deficiencies in the process and propose
ways to improve it, we can turn this into a positive, constructive and
actionable discussion.

In lots of cases (~50%) quality of syzbot reports is equal to human
reports, or _higher_.
It provides exact kernel commit, config, compiler, stand-alone C
reproducer and a nice, symbolized report even with inline frames. You
don't always get all of this from human reports.

In the remaining cases (no reproducer), quality of syzbot reports is
the _same_ as for human reports.
Say, your machine randomly crashes. You reboot it, but it crashes
again after some time. You try to repeat what you did before the crash
(say, opened a particular web page), but it does not reproduce the
crash. But one way or another, it happened and it's a kernel bug (you
did not write garbage into /dev/mem, nor loaded untested kernel
modules). What do you do in such case? Right, you report what you know
about the bug (kernel crash message, kernel revision and config).
As I said before, in lots of cases (I would say ~2/3), kernel crash
reports are actionable on its own. For example, KASAN/LOCKDEP reports
contain enough context to localize a bug. Lots of WARNINGs/GPFs point
to simple missed input checks or off-by-one's. So it would be wrong to
hide them, keep them private and pretend that nothing happened.

Re: LKML admins (syzbot emails are not delivered)

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 3101 bytes --]

On Mon 2018-01-15 11:08:12, Dmitry Vyukov wrote:
> On Thu, Jan 4, 2018 at 12:18 PM, Pavel Machek <pavel@ucw.cz> wrote:
> > On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote:
> >> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote:
> >> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
> >> >> > Hi!
> >> >> > >
> >> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they
> >> >> > > were mailed as any other emails. Here are few examples:
> >> >> > >
> >> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
> >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> >> >> > >
> >> >> > > "general protection fault in __wake_up_common"
> >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> >> >> > >
> >> >> > > Does anybody know how to get in contact with real people behind LKML
> >> >> > > and/or bugzilla?
> >> >> >
> >> >> > Not delivering syzbot emails might be good thing?
> >> >>
> >> >> Nah, the thing is finding and reporting bugs just like a human would,
> >> >> it just doesn't need sleep etc, so sometimes reports more than humans
> >> >> can keep up with.  It needs a smarter brother.. but then again, maybe
> >> >> not, if bots start fixing things too, a lot of meatware hackers would
> >> >> have to go find real jobs.
> >> >
> >> > Sending random, unrepeatable Oopses to lkml is not what humans would
> >> > do, and perhaps not something bots should do, either.
> >>
> >>
> >> Hi Pavel,
> >>
> >> I've answered this question here in full detail. In short, this is
> >> useful and actionable.
> >> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ
> >
> > I've already deleted many such reports from my lkml folder. It
> > definitely is below quality of normal bug reports.
> 
> 
> Pavel, if you point to exact deficiencies in the process and propose
> ways to improve it, we can turn this into a positive, constructive and
> actionable discussion.
> 
> In lots of cases (~50%) quality of syzbot reports is equal to human
> reports, or _higher_.
> It provides exact kernel commit, config, compiler, stand-alone C
> reproducer and a nice, symbolized report even with inline frames. You
> don't always get all of this from human reports.
> 
> In the remaining cases (no reproducer), quality of syzbot reports is
> the _same_ as for human reports.
> Say, your machine randomly crashes. You reboot it, but it crashes
> again after some time. You try to repeat what you did before the crash
> (say, opened a particular web page), but it does not reproduce the
> crash. But one way or another, it happened and it's a kernel bug

I have not seen a good quality report from syzbot, yet.

Normally, humans agregate test reports, and test patches etc. Can you
step between robot and lkml, and provide same services humans usually
provide?

Thanks,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Mon, Jan 15, 2018 at 2:08 PM, Pavel Machek <pavel@ucw.cz> wrote:
> On Mon 2018-01-15 11:08:12, Dmitry Vyukov wrote:
>> On Thu, Jan 4, 2018 at 12:18 PM, Pavel Machek <pavel@ucw.cz> wrote:
>> > On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote:
>> >> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote:
>> >> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
>> >> >> > Hi!
>> >> >> > >
>> >> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they
>> >> >> > > were mailed as any other emails. Here are few examples:
>> >> >> > >
>> >> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
>> >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
>> >> >> > >
>> >> >> > > "general protection fault in __wake_up_common"
>> >> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
>> >> >> > >
>> >> >> > > Does anybody know how to get in contact with real people behind LKML
>> >> >> > > and/or bugzilla?
>> >> >> >
>> >> >> > Not delivering syzbot emails might be good thing?
>> >> >>
>> >> >> Nah, the thing is finding and reporting bugs just like a human would,
>> >> >> it just doesn't need sleep etc, so sometimes reports more than humans
>> >> >> can keep up with.  It needs a smarter brother.. but then again, maybe
>> >> >> not, if bots start fixing things too, a lot of meatware hackers would
>> >> >> have to go find real jobs.
>> >> >
>> >> > Sending random, unrepeatable Oopses to lkml is not what humans would
>> >> > do, and perhaps not something bots should do, either.
>> >>
>> >>
>> >> Hi Pavel,
>> >>
>> >> I've answered this question here in full detail. In short, this is
>> >> useful and actionable.
>> >> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ
>> >
>> > I've already deleted many such reports from my lkml folder. It
>> > definitely is below quality of normal bug reports.
>>
>>
>> Pavel, if you point to exact deficiencies in the process and propose
>> ways to improve it, we can turn this into a positive, constructive and
>> actionable discussion.
>>
>> In lots of cases (~50%) quality of syzbot reports is equal to human
>> reports, or _higher_.
>> It provides exact kernel commit, config, compiler, stand-alone C
>> reproducer and a nice, symbolized report even with inline frames. You
>> don't always get all of this from human reports.
>>
>> In the remaining cases (no reproducer), quality of syzbot reports is
>> the _same_ as for human reports.
>> Say, your machine randomly crashes. You reboot it, but it crashes
>> again after some time. You try to repeat what you did before the crash
>> (say, opened a particular web page), but it does not reproduce the
>> crash. But one way or another, it happened and it's a kernel bug
>
> I have not seen a good quality report from syzbot, yet.

I don't know how many you checked, so I don't know how to interpret this.
If you want to see one, grep kernel commit log for syzbot, get bug id,
search for it in
https://groups.google.com/forum/#!forum/syzkaller-bugs


> Normally, humans agregate test reports, and test patches etc. Can you
> step between robot and lkml, and provide same services humans usually
> provide?

syzbot does all of this already.

Re: LKML admins (syzbot emails are not delivered)

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

Hi!

> >> In lots of cases (~50%) quality of syzbot reports is equal to human
> >> reports, or _higher_.
> >> It provides exact kernel commit, config, compiler, stand-alone C
> >> reproducer and a nice, symbolized report even with inline frames. You
> >> don't always get all of this from human reports.
> >>
> >> In the remaining cases (no reproducer), quality of syzbot reports is
> >> the _same_ as for human reports.
> >> Say, your machine randomly crashes. You reboot it, but it crashes
> >> again after some time. You try to repeat what you did before the crash
> >> (say, opened a particular web page), but it does not reproduce the
> >> crash. But one way or another, it happened and it's a kernel bug
> >
> > I have not seen a good quality report from syzbot, yet.
> 
> I don't know how many you checked, so I don't know how to interpret this.
> If you want to see one, grep kernel commit log for syzbot, get bug id,
> search for it in
> https://groups.google.com/forum/#!forum/syzkaller-bugs
> 
> 
> > Normally, humans agregate test reports, and test patches etc. Can you
> > step between robot and lkml, and provide same services humans usually
> > provide?
> 
> syzbot does all of this already.

Does it? Because its documentation says otherwise:

syzbot will keep track of this bug report.
Once a fix for this bug is merged into any tree, reply to this email
with:
#syz fix: exact-commit-title
To mark this as a duplicate of another syzbot report, please reply
with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug
report.
Note: all commands must start from beginning of the line in the email
body.

_You_ should be doing duplicates processing, and preferably you should
also take care to translate replies into something syzbot
understand. Reports should come from your email adress, and you should
really handle replies that do not come in required format.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: LKML admins (syzbot emails are not delivered)

[Eric W. Biederman]

Dmitry Vyukov <dvyukov@google.com> writes:

> Hi Pavel,
>
> I've answered this question here in full detail. In short, this is
> useful and actionable.
> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ

*Snort*

If the information to solve an issue is not in the Oops syzbot is
useless.

The Oops isn't even mailed in plain text so I have to save the stupid
thing in a file to see the full text of the problem.

Further there is no place in the syzbot process to test fixes.

Then there is the issue of testing linux-next and reporting errors on
who knows what code configuration against code that hasn't changed in
linux-next.   Which presumably any sane person would assume the errors
are introduced by some other piece of new code.  But syzbot goes and
spams the people who wrote the function where the code is failing.

Bots can work.  We have all of the automatic testing infrastructure
against everyone's branches on kernel.org to prove it.

syzbot finds weird errors, so that makes the problem space more
difficult to deal with.

Still I compleltely don't see the people behind syzbot presumably you
Dmitry taking responsibility for syzbot failings.  Instead I see excuses
like you don't completely control some part of the code that syzbot is
built on so can't fix practical real world issues.  Like Content-type.

Bots can be the most horrible thing for a code base.  If there is not
someone or something going through an filtering out the false positives.
If there is not a process to ensure that issues are brought to the
proper peoples attention so things get fixed.  Bots can be completely
demoralizing or possibily desensitizing because you keep seeing issues,
and nothing you do ever makes the issues go away.

Given that no one seems to take any responsibility for syzbots failures
of any kind.  Not content-type in the emails.  Not the body of the
message (which has a massive disclaimer).  I don't find syzbot at all
useful.

Tools are for people, in this case kernel programmers.  syzbot has
serious usability issues.  That makes syzbot a bad tool.

Eric

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Thu, Jan 4, 2018 at 4:23 PM, Eric W. Biederman <ebiederm@xmission.com> wrote:
> Dmitry Vyukov <dvyukov@google.com> writes:
>
>> Hi Pavel,
>>
>> I've answered this question here in full detail. In short, this is
>> useful and actionable.
>> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ
>
> *Snort*
>
> If the information to solve an issue is not in the Oops syzbot is
> useless.

Hi Eric

That's true. But maintainers of the subsystem is in the best position
to judge that. For that they need to see the report.


> The Oops isn't even mailed in plain text so I have to save the stupid
> thing in a file to see the full text of the problem.

Please elaborate.
Take any syzbot email, oops is right there in the email, in plain text:
https://groups.google.com/forum/#!topic/syzkaller-bugs/F6ImuLmyue8


> Further there is no place in the syzbot process to test fixes.

Please elaborate.
Kernel developer who fixes the bugs, tests it the same way as he/she
does for any other bugs. There is really nothing in syzbot that
prevents you from testing.


> Then there is the issue of testing linux-next and reporting errors on
> who knows what code configuration against code that hasn't changed in
> linux-next.   Which presumably any sane person would assume the errors
> are introduced by some other piece of new code.  But syzbot goes and
> spams the people who wrote the function where the code is failing.

syzbot uses get_maintainers.pl. If you have better suggestions, I am listening.
And note: syzbot _always_ provides exact code configuration.


> Bots can work.  We have all of the automatic testing infrastructure
> against everyone's branches on kernel.org to prove it.

If you mean build/boot testing, than that's an order of magnitude
simper problem. You can build on every commit, you can precisely
pinpoint the guilty commit, etc. Please keep this in mind.


> syzbot finds weird errors, so that makes the problem space more
> difficult to deal with.

kernel contains weird errors, that makes the problem space more difficult.


> Still I compleltely don't see the people behind syzbot presumably you
> Dmitry taking responsibility for syzbot failings.  Instead I see excuses
> like you don't completely control some part of the code that syzbot is
> built on so can't fix practical real world issues.  Like Content-type.

As far as I understand you mean this one:
https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/VSZaokajCgAJ

I probably should have described the rationale in more details.
It's not only about technical limitations. It's also about importance
of a feature, time required to implement it, and in the end if it's
the right thing to do at all or not. If that would be a major issue
that is significantly affects experience, that would happen one way or
another regardless of technical limitations. Also simple one-line
changes generally happen even if it's low profit. But in that case, I
think it's just the wrong thing to do. .txt is good, standard
extension for text files. On the other hand, .syz is completely
non-standard that no programs know how to deal with. That's why it did
not happen.
The support for Reported-by tags as discussed in "syzbot process"
thread happened within a week.

Hope this resolves your concerns.


> Bots can be the most horrible thing for a code base.  If there is not
> someone or something going through an filtering out the false positives.
> If there is not a process to ensure that issues are brought to the
> proper peoples attention so things get fixed.  Bots can be completely
> demoralizing or possibily desensitizing because you keep seeing issues,
> and nothing you do ever makes the issues go away.
>
> Given that no one seems to take any responsibility for syzbots failures
> of any kind.  Not content-type in the emails.  Not the body of the
> message (which has a massive disclaimer).  I don't find syzbot at all
> useful.
>
> Tools are for people, in this case kernel programmers.  syzbot has
> serious usability issues.  That makes syzbot a bad tool.

First of all, none of syzbot reports are false positives in the main
sense of this term. Everything it reports happened on unmodified
kernel, running user-space workload, without loading custom modules,
writing to /dev/mem, etc. These _all_ are kernel bugs. In some cases
kernel is bad at explaining precisely what went wrong. But it's
expected from complex, concurrent, non-deterministic system written in
an unsafe language. We need to deal with it.
You get the same reports from humans as well. Say, there is an invalid
free in pcrypt which corrupts memory, but kernel crashes in selinux
later. You will get report about selinux from a human.
syzbot actually makes situation a bit better to the degree possible as
it enables almost all debugging configs. So instead of a random
corruption reports, it provides a KASAN report about the exact
location. Instead of a dead kernel, you get LOCKDEP report about exact
lock inversion, etc.

Now there are duplicates, induced bugs, unexplainable crashes, reports
mailed to wrong people, etc.
There are hundreds of subsystems in kernel. And answering any of these
questions requires expertise in a particular subsystem. Say, this
crash is also a possible way how that bug could manifest. Or, the
crash happened in this subsystem, but the root cause is actually in
the upper-level subsystem that misuses this subsystem.
The right people to deal with this are maintainers of particular
subsystems. Not a single person that does not work on any of these
hundreds of subsystems.

Re: LKML admins (syzbot emails are not delivered)

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

Hi!

> > *Snort*
> >
> > If the information to solve an issue is not in the Oops syzbot is
> > useless.
> 
> Hi Eric
> 
> That's true. But maintainers of the subsystem is in the best position
> to judge that. For that they need to see the report.

Unless they are already overloaded by better quality reports.

> > Further there is no place in the syzbot process to test fixes.
> 
> Please elaborate.
> Kernel developer who fixes the bugs, tests it the same way as he/she
> does for any other bugs. There is really nothing in syzbot that
> prevents you from testing.

Well, normally people are interested in the bugs they report, and thus
willing to test the patches. Your bot.. is not interested.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Mon, Jan 15, 2018 at 1:54 PM, Pavel Machek <pavel@ucw.cz> wrote:
> Hi!
>
>> > *Snort*
>> >
>> > If the information to solve an issue is not in the Oops syzbot is
>> > useless.
>>
>> Hi Eric
>>
>> That's true. But maintainers of the subsystem is in the best position
>> to judge that. For that they need to see the report.
>
> Unless they are already overloaded by better quality reports.
>
>> > Further there is no place in the syzbot process to test fixes.
>>
>> Please elaborate.
>> Kernel developer who fixes the bugs, tests it the same way as he/she
>> does for any other bugs. There is really nothing in syzbot that
>> prevents you from testing.
>
> Well, normally people are interested in the bugs they report, and thus
> willing to test the patches. Your bot.. is not interested.

Not true. syzbot is very much interested in bugs it reports, keeps
careful track of them and tests patches.

Re: LKML admins (syzbot emails are not delivered)

[Eric W. Biederman]

Dmitry Vyukov <dvyukov@google.com> writes:

2> On Mon, Jan 15, 2018 at 1:54 PM, Pavel Machek <pavel@ucw.cz> wrote:
>> Hi!
>>
>>> > *Snort*
>>> >
>>> > If the information to solve an issue is not in the Oops syzbot is
>>> > useless.
>>>
>>> Hi Eric
>>>
>>> That's true. But maintainers of the subsystem is in the best position
>>> to judge that. For that they need to see the report.
>>
>> Unless they are already overloaded by better quality reports.
>>
>>> > Further there is no place in the syzbot process to test fixes.
>>>
>>> Please elaborate.
>>> Kernel developer who fixes the bugs, tests it the same way as he/she
>>> does for any other bugs. There is really nothing in syzbot that
>>> prevents you from testing.
>>
>> Well, normally people are interested in the bugs they report, and thus
>> willing to test the patches. Your bot.. is not interested.
>
> Not true. syzbot is very much interested in bugs it reports, keeps
> careful track of them and tests patches.

It offers to test fixes not to add more information so that the bug can
be tracked down better.  Having asked explicitly for some additional
testing to track down and issue and been told the process was happy to
test fixes I know that this has been most definitely the case.

Modifying the kernel and testing is important as sometimes it can be
extremely difficult to figure out what causes an issue.  Especially when
it is an interaction of factors like running a system on the edge of
OOM.  So it requires small kmallocs to fail (which does not usually
happen).

Eric

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Mon, Jan 15, 2018 at 5:16 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> Dmitry Vyukov <dvyukov@google.com> writes:
>
> 2> On Mon, Jan 15, 2018 at 1:54 PM, Pavel Machek <pavel@ucw.cz> wrote:
>>> Hi!
>>>
>>>> > *Snort*
>>>> >
>>>> > If the information to solve an issue is not in the Oops syzbot is
>>>> > useless.
>>>>
>>>> Hi Eric
>>>>
>>>> That's true. But maintainers of the subsystem is in the best position
>>>> to judge that. For that they need to see the report.
>>>
>>> Unless they are already overloaded by better quality reports.
>>>
>>>> > Further there is no place in the syzbot process to test fixes.
>>>>
>>>> Please elaborate.
>>>> Kernel developer who fixes the bugs, tests it the same way as he/she
>>>> does for any other bugs. There is really nothing in syzbot that
>>>> prevents you from testing.
>>>
>>> Well, normally people are interested in the bugs they report, and thus
>>> willing to test the patches. Your bot.. is not interested.
>>
>> Not true. syzbot is very much interested in bugs it reports, keeps
>> careful track of them and tests patches.
>
> It offers to test fixes not to add more information so that the bug can
> be tracked down better.  Having asked explicitly for some additional
> testing to track down and issue and been told the process was happy to
> test fixes I know that this has been most definitely the case.
>
> Modifying the kernel and testing is important as sometimes it can be
> extremely difficult to figure out what causes an issue.  Especially when
> it is an interaction of factors like running a system on the edge of
> OOM.  So it requires small kmallocs to fail (which does not usually
> happen).

Patch testing can be used for debugging as well. It just runs your
patch (maybe with additional WARNINGs) and then gives you console
output if kernel crashes. I've clarified docs regarding this:
https://github.com/google/syzkaller/commit/9aaf64b3742d1d4f744e22cad567906cebb201a2
9 out of 10 of these bugs can also be reproduced locally without any
special setup other than using the provided config.

Re: LKML admins (syzbot emails are not delivered)

[Eric W. Biederman]

Dmitry Vyukov <dvyukov@google.com> writes:

> On Thu, Jan 4, 2018 at 4:23 PM, Eric W. Biederman <ebiederm@xmission.com> wrote:
>> Dmitry Vyukov <dvyukov@google.com> writes:
>>
>>> Hi Pavel,
>>>
>>> I've answered this question here in full detail. In short, this is
>>> useful and actionable.
>>> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ
>>
>> *Snort*
>>
>> If the information to solve an issue is not in the Oops syzbot is
>> useless.
>
> Hi Eric
>
> That's true. But maintainers of the subsystem is in the best position
> to judge that. For that they need to see the report.



>> Then there is the issue of testing linux-next and reporting errors on
>> who knows what code configuration against code that hasn't changed in
>> linux-next.   Which presumably any sane person would assume the errors
>> are introduced by some other piece of new code.  But syzbot goes and
>> spams the people who wrote the function where the code is failing.
>
> syzbot uses get_maintainers.pl. If you have better suggestions, I am listening.
> And note: syzbot _always_ provides exact code configuration.

If you are testing linux-next you should really report it to whomevers
branch in linux-next you are testing.

Ideally the tests would be run on mainline and see nothing and then
on linux-next so you know it is a newly introduced error.

Sometimes the branches on linux-next are experimental crap.  If someone
adds an experimental memory allocator to linux-next before discovering
it causes all kinds of problems I don't want bug reports about my code
not being able to allocate memory because the memory allocator was bad.

If you don't have the resources to test the individual branches of
linux-next please just test Linus's tree.   That will be much more
meaningful and productive.

>> Bots can work.  We have all of the automatic testing infrastructure
>> against everyone's branches on kernel.org to prove it.
>
> If you mean build/boot testing, than that's an order of magnitude
> simper problem. You can build on every commit, you can precisely
> pinpoint the guilty commit, etc. Please keep this in mind.

The difference is not what is being tested.  The difference is how the
interface to human beings is constructed.  If it doesn't feel like
there is a human being willing to work with you on the other end of a
bug report it is not a good situation.

>> syzbot finds weird errors, so that makes the problem space more
>> difficult to deal with.
>
> kernel contains weird errors, that makes the problem space more difficult.
>
>
>> Still I compleltely don't see the people behind syzbot presumably you
>> Dmitry taking responsibility for syzbot failings.  Instead I see excuses
>> like you don't completely control some part of the code that syzbot is
>> built on so can't fix practical real world issues.  Like Content-type.
>
> As far as I understand you mean this one:
> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/VSZaokajCgAJ
>
> I probably should have described the rationale in more details.
> It's not only about technical limitations. It's also about importance
> of a feature, time required to implement it, and in the end if it's
> the right thing to do at all or not. If that would be a major issue
> that is significantly affects experience, that would happen one way or
> another regardless of technical limitations. Also simple one-line
> changes generally happen even if it's low profit. But in that case, I
> think it's just the wrong thing to do. .txt is good, standard
> extension for text files. On the other hand, .syz is completely
> non-standard that no programs know how to deal with. That's why it did
> not happen.
> The support for Reported-by tags as discussed in "syzbot process"
> thread happened within a week.

When I made the complaint it came to me and to messages on lkml as
.log.  With Content-Type: Application/Octent-stream.

That is a bloody mess that wastes peoples time.  If it is fixed good,
it certainly was not fixed at that point.

But it is much more than any single issue.  You get defensive when
people critisize syzbot.  Instead of recognizing it's failings.

It is fine to say I would like to do xyz but it will take awhile before
we can get to it.  


> Hope this resolves your concerns.

This email just intensifies them, as it feels again like you are trying
to shift the blame.

>> Bots can be the most horrible thing for a code base.  If there is not
>> someone or something going through an filtering out the false positives.
>> If there is not a process to ensure that issues are brought to the
>> proper peoples attention so things get fixed.  Bots can be completely
>> demoralizing or possibily desensitizing because you keep seeing issues,
>> and nothing you do ever makes the issues go away.
>>
>> Given that no one seems to take any responsibility for syzbots failures
>> of any kind.  Not content-type in the emails.  Not the body of the
>> message (which has a massive disclaimer).  I don't find syzbot at all
>> useful.
>>
>> Tools are for people, in this case kernel programmers.  syzbot has
>> serious usability issues.  That makes syzbot a bad tool.
>
> First of all, none of syzbot reports are false positives in the main
> sense of this term.

*Snort*  You are testing linux-next.  Who knows what pieces of that are
going to go into a stable kernel.  You are not reporting failures
against individual maintainers who changed linux-next you are
reporting to random people in get_maintainers.pl who may have no
connection with the code change.

So it is a very low quality bug report, on some random mutation of the
linux-kernel.  You might as well be testing some random distro kernel
and using get_maintainers.pl to tell us something is wrong.

> You get the same reports from humans as well. Say, there is an invalid
> free in pcrypt which corrupts memory, but kernel crashes in selinux
> later. You will get report about selinux from a human.
> syzbot actually makes situation a bit better to the degree possible as
> it enables almost all debugging configs. So instead of a random
> corruption reports, it provides a KASAN report about the exact
> location. Instead of a dead kernel, you get LOCKDEP report about exact
> lock inversion, etc.

But I can ask the human what their configuration was and what they were
doing when the error happened.  Further things can be prioritized by how
badly the errors affect real people.  In practice if bugs don't affect
people more than once, they don't care.

That conversation can not be had with syzbot.

Outside of the bugs being considered as considered as security issues,
the bugs syzbot finds are generally things that don't affect anyone in
practice.  So are very low on the priority of things to get fixed.

> Now there are duplicates, induced bugs, unexplainable crashes, reports
> mailed to wrong people, etc.
> There are hundreds of subsystems in kernel. And answering any of these
> questions requires expertise in a particular subsystem. Say, this
> crash is also a possible way how that bug could manifest. Or, the
> crash happened in this subsystem, but the root cause is actually in
> the upper-level subsystem that misuses this subsystem.
> The right people to deal with this are maintainers of particular
> subsystems. Not a single person that does not work on any of these
> hundreds of subsystems.

I am definitely not asking for expertise in the kernel.  I am asking
for a human who wants to help track down bugs in the kernel.  Not a
poorly backed up accusation that I did something wrong.

Because that is how syzbot feels today.

Eric

Re: LKML admins (syzbot emails are not delivered)

[Theodore Ts'o]

On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote:
> 
> Sometimes the branches on linux-next are experimental crap.  If someone
> adds an experimental memory allocator to linux-next before discovering
> it causes all kinds of problems I don't want bug reports about my code
> not being able to allocate memory because the memory allocator was bad.
> 
> If you don't have the resources to test the individual branches of
> linux-next please just test Linus's tree.   That will be much more
> meaningful and productive.

I have to agree with Eric here, the reason why Fengguang Wu's 0-day
testing robot is much better received by developers is that he does
not test linux-net, but rather individual subsystem git trees and
branches.  His test automation also does an automatic bisection
search, and can point at a specific commit --- at which point e-mail
goes out to owner of the subsystem git tree, and to the people who
authored and/or reviewed the guilty commit.

Dmitry, perhaps you could collaborate with Intel's 0-day testing
folks?  They have code which does all of this, and perhaps it can be
leveraged.

> 
> When I made the complaint it came to me and to messages on lkml as
> .log.  With Content-Type: Application/Octent-stream.
> 
> That is a bloody mess that wastes peoples time.  If it is fixed good,
> it certainly was not fixed at that point.

I just checked a recent report from the Syzbot, and it's not fixed.
The raw.log file still uses a Content-Type of
Application/Octet-stream.  Worse the reproducer C source file has a
content type of Application/Octet-stream instead of the much more sane
Application/text.  <Face palm>

Hint to Googlers --- many kernel developers do not use Gmail because
it does unspeakable things to whitespaces, which results in mangled
patches, or because they want real mail threading.  The Content-Type
really matters, because for many text-based Mail User Agents, if it is
Application/octet-stream, the MUA will assume that it can't be
displayed as text, and require that it be saved to a file, which the
developer then has to manually read by firing up a pager or an editor.
When you are getting hundreds or thousands of messages a day, having
critical data which darn well *could* be displayed as text require
manual processing adds friction and destroys developer productivity.
So *you* might think the Content-type is trivial, but for developers
who live in their MUA's (that's why many prefer to review patches in
their mail reader, and not have to switch to web browser to use
Gerrit), screwing up the Content-type is going to be a big deal to
them.

> Outside of the bugs being considered as considered as security issues,
> the bugs syzbot finds are generally things that don't affect anyone in
> practice.  So are very low on the priority of things to get fixed.

The real danger is that people will start auto-filing Syzbot reports
to "file 13" (e.g., the trash can) because they are too annoying....
But that's Dmitry and the Syzkaller team's problem, not the kernel
developers.   :-)

						- Ted

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote:
>>
>> Sometimes the branches on linux-next are experimental crap.  If someone
>> adds an experimental memory allocator to linux-next before discovering
>> it causes all kinds of problems I don't want bug reports about my code
>> not being able to allocate memory because the memory allocator was bad.
>>
>> If you don't have the resources to test the individual branches of
>> linux-next please just test Linus's tree.   That will be much more
>> meaningful and productive.
>
> I have to agree with Eric here, the reason why Fengguang Wu's 0-day
> testing robot is much better received by developers is that he does
> not test linux-net,

I will remove linux-next if there is a general agreement that it's not
useful. Though, I've heard different opinions from kernel developers
as well. I will write a separate email asking what branches should be
tested.

Re: LKML admins (syzbot emails are not delivered)

[Guenter Roeck]

On Mon, Jan 15, 2018 at 11:51 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote:
>>>
>>> Sometimes the branches on linux-next are experimental crap.  If someone
>>> adds an experimental memory allocator to linux-next before discovering
>>> it causes all kinds of problems I don't want bug reports about my code
>>> not being able to allocate memory because the memory allocator was bad.
>>>
>>> If you don't have the resources to test the individual branches of
>>> linux-next please just test Linus's tree.   That will be much more
>>> meaningful and productive.
>>
>> I have to agree with Eric here, the reason why Fengguang Wu's 0-day
>> testing robot is much better received by developers is that he does
>> not test linux-net,
>

Interesting. Assuming that refers to linux-next, not linux-net, that
may explain why linux-next tends to deteriorate. I wonder if I should
drop it from my testing as well. I'll be happy to follow whatever the
result of this exchange is and do the same.

Guenter

> I will remove linux-next if there is a general agreement that it's not
> useful. Though, I've heard different opinions from kernel developers
> as well. I will write a separate email asking what branches should be
> tested.

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Tue, Jan 16, 2018 at 10:52 AM, Guenter Roeck <groeck@google.com> wrote:
> On Mon, Jan 15, 2018 at 11:51 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>>> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote:
>>>>
>>>> Sometimes the branches on linux-next are experimental crap.  If someone
>>>> adds an experimental memory allocator to linux-next before discovering
>>>> it causes all kinds of problems I don't want bug reports about my code
>>>> not being able to allocate memory because the memory allocator was bad.
>>>>
>>>> If you don't have the resources to test the individual branches of
>>>> linux-next please just test Linus's tree.   That will be much more
>>>> meaningful and productive.
>>>
>>> I have to agree with Eric here, the reason why Fengguang Wu's 0-day
>>> testing robot is much better received by developers is that he does
>>> not test linux-net,
>>
>
> Interesting. Assuming that refers to linux-next, not linux-net, that
> may explain why linux-next tends to deteriorate. I wonder if I should
> drop it from my testing as well. I'll be happy to follow whatever the
> result of this exchange is and do the same.
>
> Guenter
>
>> I will remove linux-next if there is a general agreement that it's not
>> useful. Though, I've heard different opinions from kernel developers
>> as well. I will write a separate email asking what branches should be
>> tested.

Let's please move discussion of this topic to "what trees/branches to
test on syzbot" thread. This thread is now about too many things.
Hope you don't mind if I repost your last email there.

Re: LKML admins (syzbot emails are not delivered)

[Guenter Roeck]

On Tue, Jan 16, 2018 at 1:56 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Tue, Jan 16, 2018 at 10:52 AM, Guenter Roeck <groeck@google.com> wrote:
>> On Mon, Jan 15, 2018 at 11:51 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>>> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>>>> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote:
>>>>>
>>>>> Sometimes the branches on linux-next are experimental crap.  If someone
>>>>> adds an experimental memory allocator to linux-next before discovering
>>>>> it causes all kinds of problems I don't want bug reports about my code
>>>>> not being able to allocate memory because the memory allocator was bad.
>>>>>
>>>>> If you don't have the resources to test the individual branches of
>>>>> linux-next please just test Linus's tree.   That will be much more
>>>>> meaningful and productive.
>>>>
>>>> I have to agree with Eric here, the reason why Fengguang Wu's 0-day
>>>> testing robot is much better received by developers is that he does
>>>> not test linux-net,
>>>
>>
>> Interesting. Assuming that refers to linux-next, not linux-net, that
>> may explain why linux-next tends to deteriorate. I wonder if I should
>> drop it from my testing as well. I'll be happy to follow whatever the
>> result of this exchange is and do the same.
>>
>> Guenter
>>
>>> I will remove linux-next if there is a general agreement that it's not
>>> useful. Though, I've heard different opinions from kernel developers
>>> as well. I will write a separate email asking what branches should be
>>> tested.
>
> Let's please move discussion of this topic to "what trees/branches to
> test on syzbot" thread. This thread is now about too many things.
> Hope you don't mind if I repost your last email there.

Sure, go ahead.

Guenter

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote:
>>
>> Sometimes the branches on linux-next are experimental crap.  If someone
>> adds an experimental memory allocator to linux-next before discovering
>> it causes all kinds of problems I don't want bug reports about my code
>> not being able to allocate memory because the memory allocator was bad.
>>
>> If you don't have the resources to test the individual branches of
>> linux-next please just test Linus's tree.   That will be much more
>> meaningful and productive.
>
> I have to agree with Eric here, the reason why Fengguang Wu's 0-day
> testing robot is much better received by developers is that he does
> not test linux-net, but rather individual subsystem git trees and
> branches.  His test automation also does an automatic bisection
> search, and can point at a specific commit --- at which point e-mail
> goes out to owner of the subsystem git tree, and to the people who
> authored and/or reviewed the guilty commit.
>
> Dmitry, perhaps you could collaborate with Intel's 0-day testing
> folks?  They have code which does all of this, and perhaps it can be
> leveraged.

+Fengguang

Please note that in most cases 0-day solves an order of magnitude
simpler problem. Build/sparse errors are much faster to find, always
possible to precisely bisect and attribute. Yes, for that you just
test every commit, bisect and send targeted emails. syzbot only finds
runtime bugs, lots of them are related to races and can't be reliably
reproduced, bisected, etc. Lots of them are old (e.g. predate KASAN
that detects them). But they still can be fixed. In ~half of cases
developers fix them looking only at the oops report.

The last time I checked 0-day infrastructure was closed source.

Fengguang, what do you do with trinity crashes that happen
episodically, but you can't reliably reproduce, bisect and attribute?

Thanks

Re: LKML admins (syzbot emails are not delivered)

[Fengguang Wu]

On Tue, Jan 16, 2018 at 08:59:36AM +0100, Dmitry Vyukov wrote:
>On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>> On Mon, Jan 15, 2018 at 10:38:42AM -0600, Eric W. Biederman wrote:
>>>
>>> Sometimes the branches on linux-next are experimental crap.  If someone
>>> adds an experimental memory allocator to linux-next before discovering
>>> it causes all kinds of problems I don't want bug reports about my code
>>> not being able to allocate memory because the memory allocator was bad.
>>>
>>> If you don't have the resources to test the individual branches of
>>> linux-next please just test Linus's tree.   That will be much more
>>> meaningful and productive.
>>
>> I have to agree with Eric here, the reason why Fengguang Wu's 0-day
>> testing robot is much better received by developers is that he does
>> not test linux-net, but rather individual subsystem git trees and
>> branches.  His test automation also does an automatic bisection
>> search, and can point at a specific commit --- at which point e-mail
>> goes out to owner of the subsystem git tree, and to the people who
>> authored and/or reviewed the guilty commit.
>>
>> Dmitry, perhaps you could collaborate with Intel's 0-day testing
>> folks?  They have code which does all of this, and perhaps it can be
>> leveraged.
>
>+Fengguang
>
>Please note that in most cases 0-day solves an order of magnitude
>simpler problem. Build/sparse errors are much faster to find, always
>possible to precisely bisect and attribute. Yes, for that you just
>test every commit, bisect and send targeted emails. syzbot only finds
>runtime bugs, lots of them are related to races and can't be reliably
>reproduced, bisected, etc. Lots of them are old (e.g. predate KASAN
>that detects them). But they still can be fixed. In ~half of cases
>developers fix them looking only at the oops report.
>
>The last time I checked 0-day infrastructure was closed source.
>
>Fengguang, what do you do with trinity crashes that happen
>episodically, but you can't reliably reproduce, bisect and attribute?

0-day runs most trinity tests in QEMU machines, which can run
massively in parallel. Which means we can afford to bisect them
by running up to 1000 boots in each bisect step. Ditto for KASAN
errors.

Since Xiaolong (CCed) has enabled syzkaller in 0-day, it could in
theory utilize the same auto bisect infrastructure. If we can make
sure syzkaller runs effectively in the 0-day test farm.

Thanks,
Fengguang

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> I just checked a recent report from the Syzbot, and it's not fixed.
> The raw.log file still uses a Content-Type of
> Application/Octet-stream.  Worse the reproducer C source file has a
> content type of Application/Octet-stream instead of the much more sane
> Application/text.  <Face palm>


I will look into using a different mailing system which allows more
control over email contents.
A quick fix for raw.log will be to rename it to raw.log.txt. Not sure
if text/plain repro.c.txt is better than application/octet-stream
repro.c...

Re: LKML admins (syzbot emails are not delivered)

[Theodore Ts'o]

On Tue, Jan 16, 2018 at 09:31:26AM +0100, Dmitry Vyukov wrote:
> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> > I just checked a recent report from the Syzbot, and it's not fixed.
> > The raw.log file still uses a Content-Type of
> > Application/Octet-stream.  Worse the reproducer C source file has a
> > content type of Application/Octet-stream instead of the much more sane
> > Application/text.  <Face palm>
> 
> 
> I will look into using a different mailing system which allows more
> control over email contents.
> A quick fix for raw.log will be to rename it to raw.log.txt. Not sure
> if text/plain repro.c.txt is better than application/octet-stream
> repro.c...

My personal opinion is that if there is no way to force the content
type except by using magic extensions --- which is super-surprising to
me; that seems like a broken API and a feature request bug should be
filed against the relevant API --- using repro.c.txt would be the best
of bad alternatives.  Someone is going to have to exit to a shell to
compile the repro, and renaming the filename isn't a big deal.  The
Mail User Agent I use (mutt) allows me to specify the directory to
save the file, and gives me the opportunity to edit the filename,
before I save it.  So at least for me, it really isn't a big deal for
you to use repro.c.txt.

Cheers,

					- Ted

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Wed, Jan 17, 2018 at 12:13 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Tue, Jan 16, 2018 at 09:31:26AM +0100, Dmitry Vyukov wrote:
>> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>> > I just checked a recent report from the Syzbot, and it's not fixed.
>> > The raw.log file still uses a Content-Type of
>> > Application/Octet-stream.  Worse the reproducer C source file has a
>> > content type of Application/Octet-stream instead of the much more sane
>> > Application/text.  <Face palm>
>>
>>
>> I will look into using a different mailing system which allows more
>> control over email contents.
>> A quick fix for raw.log will be to rename it to raw.log.txt. Not sure
>> if text/plain repro.c.txt is better than application/octet-stream
>> repro.c...
>
> My personal opinion is that if there is no way to force the content
> type except by using magic extensions --- which is super-surprising to
> me; that seems like a broken API and a feature request bug should be
> filed against the relevant API --- using repro.c.txt would be the best
> of bad alternatives.  Someone is going to have to exit to a shell to
> compile the repro, and renaming the filename isn't a big deal.  The
> Mail User Agent I use (mutt) allows me to specify the directory to
> save the file, and gives me the opportunity to edit the filename,
> before I save it.  So at least for me, it really isn't a big deal for
> you to use repro.c.txt.

Good. I've made such change, it's now raw.log.txt and repro.c.txt:
https://github.com/google/syzkaller/commit/afcb994770d7e0f4b88c623bec76fbdce57d3910

I've also mailed a change to appengine that supports *.c, *.log and
*.patch as text/plain extensions. There does not seem to be any major
objects to it, but it will probably take some time to be deployed in
prod. After that I will rename them back.

Re: LKML admins (syzbot emails are not delivered)

[Theodore Ts'o]

On Wed, Jan 17, 2018 at 09:06:54AM +0100, Dmitry Vyukov wrote:
> I've also mailed a change to appengine that supports *.c, *.log and
> *.patch as text/plain extensions. There does not seem to be any major
> objects to it, but it will probably take some time to be deployed in
> prod. After that I will rename them back.

I would think the ideal change would be one where one could specify an
optional parameter to the appengine API which would override the
default content/type, but adding some additional defaults
for various magic extentions is probably a simpler change.

    	    	  	     		   - Ted

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Wed, Jan 17, 2018 at 5:53 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Wed, Jan 17, 2018 at 09:06:54AM +0100, Dmitry Vyukov wrote:
>> I've also mailed a change to appengine that supports *.c, *.log and
>> *.patch as text/plain extensions. There does not seem to be any major
>> objects to it, but it will probably take some time to be deployed in
>> prod. After that I will rename them back.
>
> I would think the ideal change would be one where one could specify an
> optional parameter to the appengine API which would override the
> default content/type, but adding some additional defaults
> for various magic extentions is probably a simpler change.

Agree, but as far as I understand it is specifically restricted. You
can see docs here:
https://cloud.google.com/appengine/docs/standard/php/mail/mail-with-headers-attachments
It also does not allow sending .exe, .com, .bat, etc at all.
The support for log/patch/c will at least solve all problems for syzbot.

Re: LKML admins (syzbot emails are not delivered)

[Bernd Petrovitsch]

On Wed, 2018-01-17 at 18:16 +0100, Dmitry Vyukov wrote:
[...]
> Agree, but as far as I understand it is specifically restricted. You
> can see docs here:
> https://cloud.google.com/appengine/docs/standard/php/mail/mail-with-headers-attachments

ROTFL - as if almost no one at Google know that there are source files
of any kind.

> It also does not allow sending .exe, .com, .bat, etc at all.
> The support for log/patch/c will at least solve all problems for syzbot.

Yup, but they should better also add .h, .hpp., .cpp and a lot more
from the source code world too.

I'm just wondering if it wouldn't be even more safe to use text/plain
(instead of application/octet-stream) as the default MIME type if one
wants to avoid to be misused to send viruses etc.

MfG,
	Bernd

PS: Sry, for somewhat semi-off-topic ....
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
                     LUGA : http://www.luga.at

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>> Outside of the bugs being considered as considered as security issues,
>> the bugs syzbot finds are generally things that don't affect anyone in
>> practice.  So are very low on the priority of things to get fixed.

Not sure why are you saying this, but syzbot has found lots of
hundreds of use-after-free's, out-of-bounds, information leaks,
deadlocks, vm escapes, etc. They have very direct stability and
security impact.

Re: LKML admins (syzbot emails are not delivered)

[Alan Cox]

On Tue, 16 Jan 2018 09:34:01 +0100
Dmitry Vyukov <dvyukov@google.com> wrote:

> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> >> Outside of the bugs being considered as considered as security issues,
> >> the bugs syzbot finds are generally things that don't affect anyone in
> >> practice.  So are very low on the priority of things to get fixed.  
> 
> Not sure why are you saying this, but syzbot has found lots of
> hundreds of use-after-free's, out-of-bounds, information leaks,
> deadlocks, vm escapes, etc. They have very direct stability and
> security impact.

Agreed - there may be some UI and presentation issues but it's found some
really nasty little bugs.

Alan

Re: LKML admins (syzbot emails are not delivered)

[Eric W. Biederman]

Alan Cox <gnomes@lxorguk.ukuu.org.uk> writes:

> On Tue, 16 Jan 2018 09:34:01 +0100
> Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
>> >> Outside of the bugs being considered as considered as security issues,
>> >> the bugs syzbot finds are generally things that don't affect anyone in
>> >> practice.  So are very low on the priority of things to get fixed.  
>> 
>> Not sure why are you saying this, but syzbot has found lots of
>> hundreds of use-after-free's, out-of-bounds, information leaks,
>> deadlocks, vm escapes, etc. They have very direct stability and
>> security impact.
>
> Agreed - there may be some UI and presentation issues but it's found some
> really nasty little bugs.

I am not certain it has always really found the bugs it hits.

My experience tends towards a bug report with too little information
in the Oops to guess what went wrong, that I can not reproduce the
issue locally, that the no can reproduce, that was produced on a weird
tree, and with a reporter telling you they are only interested in
testing fixes.

Which is a long way of saying if the UI issues are bad enough the issue
can not be identified in the code I am not certain we have actually
found a bug.

So while I can see lots of potential in syzbot.  I can't say if the it
is greater potential to get bugs fixed or to annoy developers with
complaints they can't do anything about.

Eric

Re: LKML admins (syzbot emails are not delivered)

[Greg Kroah-Hartman]

On Wed, Jan 24, 2018 at 11:06:02AM -0600, Eric W. Biederman wrote:
> Alan Cox <gnomes@lxorguk.ukuu.org.uk> writes:
> 
> > On Tue, 16 Jan 2018 09:34:01 +0100
> > Dmitry Vyukov <dvyukov@google.com> wrote:
> >
> >> On Tue, Jan 16, 2018 at 8:12 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> >> >> Outside of the bugs being considered as considered as security issues,
> >> >> the bugs syzbot finds are generally things that don't affect anyone in
> >> >> practice.  So are very low on the priority of things to get fixed.  
> >> 
> >> Not sure why are you saying this, but syzbot has found lots of
> >> hundreds of use-after-free's, out-of-bounds, information leaks,
> >> deadlocks, vm escapes, etc. They have very direct stability and
> >> security impact.
> >
> > Agreed - there may be some UI and presentation issues but it's found some
> > really nasty little bugs.
> 
> I am not certain it has always really found the bugs it hits.
> 
> My experience tends towards a bug report with too little information
> in the Oops to guess what went wrong, that I can not reproduce the
> issue locally, that the no can reproduce, that was produced on a weird
> tree, and with a reporter telling you they are only interested in
> testing fixes.
> 
> Which is a long way of saying if the UI issues are bad enough the issue
> can not be identified in the code I am not certain we have actually
> found a bug.
> 
> So while I can see lots of potential in syzbot.  I can't say if the it
> is greater potential to get bugs fixed or to annoy developers with
> complaints they can't do anything about.

I'm with Alan here, syzbot has found lots of nasty bugs in the areas of
the kernel I maintain.  Many of which are still on my TODO list to fix :)

So yes, it's annoying to me at times as well, but it is good work here,
and I hope to see it continue.

greg k-h

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Mon, Jan 15, 2018 at 5:38 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> When I made the complaint it came to me and to messages on lkml as
> .log.  With Content-Type: Application/Octent-stream.


Where was that? If I am not mistaken you actually didn't. I
triple-checked my inbox and searched internet. The only references to
Content-Type from you I can find are in this thread. And the are "Like
Content-type" and "Not content-type in the emails". I simply was not
aware of the problem, and I did not understand these comment because
they don't explain the problem. The only discussion that happened
around Content-Type is this:
https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/7DjzSA2jCgAJ
This is proposal to give files .syz extension, which does not look
right to me and it a different topic anyway.


> That is a bloody mess that wastes peoples time.  If it is fixed good,
> it certainly was not fixed at that point.
>
> But it is much more than any single issue.  You get defensive when
> people critisize syzbot.  Instead of recognizing it's failings.

Re: LKML admins (syzbot emails are not delivered)

[Dmitry Vyukov]

On Mon, Jan 15, 2018 at 5:38 PM, Eric W. Biederman
<ebiederm@xmission.com> wrote:
> I am definitely not asking for expertise in the kernel.  I am asking
> for a human who wants to help track down bugs in the kernel.

What is that additional information that you need and that syzbot
currently does not provide?

Re: LKML admins (syzbot emails are not delivered)

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]

On Thu 2018-01-04 12:09:26, Dmitry Vyukov wrote:
> On Thu, Jan 4, 2018 at 10:56 AM, Pavel Machek <pavel@ucw.cz> wrote:
> >> On Thu, 2018-01-04 at 10:25 +0100, Pavel Machek wrote:
> >> > Hi!
> >> > >
> >> > > Some of syzbot emails don't appear on LKML mailing lists, while they
> >> > > were mailed as any other emails. Here are few examples:
> >> > >
> >> > > "KASAN: use-after-free Read in rds_tcp_dev_event"
> >> > > https://groups.google.com/d/msg/syzkaller-bugs/nEeIAsNLWL4/1GzamOmRAwAJ
> >> > >
> >> > > "general protection fault in __wake_up_common"
> >> > > https://groups.google.com/d/msg/syzkaller-bugs/4TrrZ0bIViw/rBcYLUJHAgAJ
> >> > >
> >> > > Does anybody know how to get in contact with real people behind LKML
> >> > > and/or bugzilla?
> >> >
> >> > Not delivering syzbot emails might be good thing?
> >>
> >> Nah, the thing is finding and reporting bugs just like a human would,
> >> it just doesn't need sleep etc, so sometimes reports more than humans
> >> can keep up with.  It needs a smarter brother.. but then again, maybe
> >> not, if bots start fixing things too, a lot of meatware hackers would
> >> have to go find real jobs.
> >
> > Sending random, unrepeatable Oopses to lkml is not what humans would
> > do, and perhaps not something bots should do, either.
> 
> 
> Hi Pavel,
> 
> I've answered this question here in full detail. In short, this is
> useful and actionable.
> https://groups.google.com/d/msg/syzkaller/2nVn_XkVhEE/GjjfISejCgAJ

Unfortantely, some of the reports are making it to the lkml.

200KB each! Due to attached .config files. I believe you should not be
spamming lkml in the first place. If you insist on sending to lkml and
list admins are okay with that, please gzip the config file.

								Pavel


>  do_syscall_32_irqs_on arch/x86/entry/common.c:327 [inline]
>  do_fast_syscall_32+0x3ee/0xf9d arch/x86/entry/common.c:389
>  entry_SYSENTER_compat+0x54/0x63
arch/x86/entry/entry_64_compat.S:129

Looks like the same bug I sent out a fix for yesterday.

#syz fix: capabilities: fix buffer overread on very short xattr

https://marc.info/?l=linux-kernel&m=151488700301705


-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]