Re: INFO: task hung in __do_page_fault (2)

Re: INFO: task hung in __do_page_fault (2)

[syzbot]

syzbot has bisected this bug to:

commit 0161028b7c8aebef64194d3d73e43bc3b53b5c66
Author: Andy Lutomirski <luto@kernel.org>
Date:   Mon May 9 22:48:51 2016 +0000

     perf/core: Change the default paranoia level to 2

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=15910e86e00000
start commit:   18d0eae3 Merge tag 'char-misc-4.20-rc1' of git://git.kerne..
git tree:       upstream
final crash:    https://syzkaller.appspot.com/x/report.txt?x=17910e86e00000
console output: https://syzkaller.appspot.com/x/log.txt?x=13910e86e00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=342f43de913c81b9
dashboard link: https://syzkaller.appspot.com/bug?extid=6b074f741adbd93d2df5
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12482713400000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=158fd4a3400000

Reported-by: syzbot+6b074f741adbd93d2df5@syzkaller.appspotmail.com
Fixes: 0161028b7c8a ("perf/core: Change the default paranoia level to 2")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

Re: INFO: task hung in __do_page_fault (2)

[Andy Lutomirski]

On Wed, Nov 20, 2019 at 11:52 AM syzbot
<syzbot+6b074f741adbd93d2df5@syzkaller.appspotmail.com> wrote:
>
> syzbot has bisected this bug to:
>
> commit 0161028b7c8aebef64194d3d73e43bc3b53b5c66
> Author: Andy Lutomirski <luto@kernel.org>
> Date:   Mon May 9 22:48:51 2016 +0000
>
>      perf/core: Change the default paranoia level to 2
>
> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=15910e86e00000
> start commit:   18d0eae3 Merge tag 'char-misc-4.20-rc1' of git://git.kerne..
> git tree:       upstream
> final crash:    https://syzkaller.appspot.com/x/report.txt?x=17910e86e00000
> console output: https://syzkaller.appspot.com/x/log.txt?x=13910e86e00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=342f43de913c81b9
> dashboard link: https://syzkaller.appspot.com/bug?extid=6b074f741adbd93d2df5
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12482713400000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=158fd4a3400000
>
> Reported-by: syzbot+6b074f741adbd93d2df5@syzkaller.appspotmail.com
> Fixes: 0161028b7c8a ("perf/core: Change the default paranoia level to 2")
>
> For information about bisection process see: https://goo.gl/tpsmEJ#bisection

Hi syzbot-

I'm not quite sure how to tell you this in syzbotese, but I'm pretty
sure you've bisected this wrong.  The blamed patch makes no sense.

--Andy

Re: INFO: task hung in __do_page_fault (2)

[Dmitry Vyukov]

On Thu, Nov 21, 2019 at 7:01 PM Andy Lutomirski <luto@kernel.org> wrote:
>
> On Wed, Nov 20, 2019 at 11:52 AM syzbot
> <syzbot+6b074f741adbd93d2df5@syzkaller.appspotmail.com> wrote:
> >
> > syzbot has bisected this bug to:
> >
> > commit 0161028b7c8aebef64194d3d73e43bc3b53b5c66
> > Author: Andy Lutomirski <luto@kernel.org>
> > Date:   Mon May 9 22:48:51 2016 +0000
> >
> >      perf/core: Change the default paranoia level to 2
> >
> > bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=15910e86e00000
> > start commit:   18d0eae3 Merge tag 'char-misc-4.20-rc1' of git://git.kerne..
> > git tree:       upstream
> > final crash:    https://syzkaller.appspot.com/x/report.txt?x=17910e86e00000
> > console output: https://syzkaller.appspot.com/x/log.txt?x=13910e86e00000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=342f43de913c81b9
> > dashboard link: https://syzkaller.appspot.com/bug?extid=6b074f741adbd93d2df5
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12482713400000
> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=158fd4a3400000
> >
> > Reported-by: syzbot+6b074f741adbd93d2df5@syzkaller.appspotmail.com
> > Fixes: 0161028b7c8a ("perf/core: Change the default paranoia level to 2")
> >
> > For information about bisection process see: https://goo.gl/tpsmEJ#bisection
>
> Hi syzbot-
>
> I'm not quite sure how to tell you this in syzbotese, but I'm pretty
> sure you've bisected this wrong.  The blamed patch makes no sense.


Hi Andy,

Three is no way to tell syzbot about this, it does not have any way to
use this information.
You can tell this to other recipients, though, and for the record on
the bug report email thread. For this you can use any free form.

But what makes you think this is wrong?
From everything I see this looks like amazingly precise bisection.
The reproducer contains perf_event_open which seems to cause the hang
(there is a number of reports where perf_event_open hangs kernel dead
IIRC) _and_ it contains setresuid. Which makes good match for
"perf/core: Change the default paranoia level to 2" (for unpriv
users).
The bisection log also looks perfectly correct to me: no unrelated
kernel bugs were hit along the way; the crash was always reproduced
100% reliably in all 10 runs; nothing else suspicious.
I can totally imagine that your patch unmasked some latent bug, but
it's not 100% obvious to me and in either case syzbot did the job as
well as a robot could possibly do.

Re: INFO: task hung in __do_page_fault (2)

[Eric W. Biederman]

Dmitry Vyukov <dvyukov@google.com> writes:

> On Thu, Nov 21, 2019 at 7:01 PM Andy Lutomirski <luto@kernel.org> wrote:
>>
>> On Wed, Nov 20, 2019 at 11:52 AM syzbot
>> <syzbot+6b074f741adbd93d2df5@syzkaller.appspotmail.com> wrote:
>> >
>> > syzbot has bisected this bug to:
>> >
>> > commit 0161028b7c8aebef64194d3d73e43bc3b53b5c66
>> > Author: Andy Lutomirski <luto@kernel.org>
>> > Date:   Mon May 9 22:48:51 2016 +0000
>> >
>> >      perf/core: Change the default paranoia level to 2
>> >
>> > bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=15910e86e00000
>> > start commit:   18d0eae3 Merge tag 'char-misc-4.20-rc1' of git://git.kerne..
>> > git tree:       upstream
>> > final crash:    https://syzkaller.appspot.com/x/report.txt?x=17910e86e00000
>> > console output: https://syzkaller.appspot.com/x/log.txt?x=13910e86e00000
>> > kernel config:  https://syzkaller.appspot.com/x/.config?x=342f43de913c81b9
>> > dashboard link: https://syzkaller.appspot.com/bug?extid=6b074f741adbd93d2df5
>> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12482713400000
>> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=158fd4a3400000
>> >
>> > Reported-by: syzbot+6b074f741adbd93d2df5@syzkaller.appspotmail.com
>> > Fixes: 0161028b7c8a ("perf/core: Change the default paranoia level to 2")
>> >
>> > For information about bisection process see: https://goo.gl/tpsmEJ#bisection
>>
>> Hi syzbot-
>>
>> I'm not quite sure how to tell you this in syzbotese, but I'm pretty
>> sure you've bisected this wrong.  The blamed patch makes no sense.
>
>
> Hi Andy,
>
> Three is no way to tell syzbot about this, it does not have any way to
> use this information.
> You can tell this to other recipients, though, and for the record on
> the bug report email thread. For this you can use any free form.
>
> But what makes you think this is wrong?
> From everything I see this looks like amazingly precise bisection.
> The reproducer contains perf_event_open which seems to cause the hang
> (there is a number of reports where perf_event_open hangs kernel dead
> IIRC) _and_ it contains setresuid. Which makes good match for
> "perf/core: Change the default paranoia level to 2" (for unpriv
> users).
> The bisection log also looks perfectly correct to me: no unrelated
> kernel bugs were hit along the way; the crash was always reproduced
> 100% reliably in all 10 runs; nothing else suspicious.
> I can totally imagine that your patch unmasked some latent bug, but
> it's not 100% obvious to me and in either case syzbot did the job as
> well as a robot could possibly do.

All Andy's patch did was change the default value of
sysctl_perf_event_paranoid.  Which a quick skim of the code can only
cause perf_event_open to fail.

So if perf is running as non-root aka unprivileged it might have
been affected.

That said the most likely effect that would cause a hang is for perf to
not be started and therefore it's NMI's did not happen and so something
else was free to hang.

The other possibility is something in perf_event_open goes haywire
when it attempts to start and gets permission denied.  That seems
unlikely.  Assuming that was the case Andy's change did not
touch any of the perf_event_open code.  So at most it is highlighting
a path that was broken in earlier kernels and Andy's change to
the default caused the syzbot code to take a path that was broken
much earlier.


The common sense operation to perform at this point is to realize
that the setting of sysctl_perf_event_open matters to the test and
to modify the test to set sysctl_perf_event_open before it does
more things, and then syzbot or it's keepers can track down a likely
cause for the hang.


Certainly pointing at Andy's patch gives no one any real information of
why the kernel was hanging.  It is literally changing an default value
of 1 to a default value of 2.

Eric