From: Dmitry Vyukov <dvyukov@google.com>
To: Borislav Petkov <bp@alien8.de>
Cc: Nick Desaulniers <ndesaulniers@google.com>,
Josh Poimboeuf <jpoimboe@redhat.com>,
syzbot <syzbot+ce179bc99e64377c24bc@syzkaller.appspotmail.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Alexander Shishkin <alexander.shishkin@linux.intel.com>,
"H. Peter Anvin" <hpa@zytor.com>, Jiri Olsa <jolsa@redhat.com>,
LKML <linux-kernel@vger.kernel.org>,
Mark Rutland <mark.rutland@arm.com>,
Ingo Molnar <mingo@redhat.com>,
Namhyung Kim <namhyung@kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
Thomas Gleixner <tglx@linutronix.de>,
"the arch/x86 maintainers" <x86@kernel.org>,
clang-built-linux <clang-built-linux@googlegroups.com>
Subject: Re: general protection fault in perf_misc_flags
Date: Wed, 23 Sep 2020 11:24:48 +0200 [thread overview]
Message-ID: <CACT4Y+Y4-vqdv01ebyzhUoggUCUyvbhjut7Wvj=r4dBfyxLeng@mail.gmail.com> (raw)
In-Reply-To: <20200923090336.GD28545@zn.tnic>
On Wed, Sep 23, 2020 at 11:03 AM Borislav Petkov <bp@alien8.de> wrote:
>
> On Tue, Sep 22, 2020 at 11:56:04AM -0700, Nick Desaulniers wrote:
> > So I think there's an issue with "deterministically reproducible."
> > The syzcaller report has:
> > > > Unfortunately, I don't have any reproducer for this issue yet.
>
> Yeah, Dmitry gave two other links of similar reports, the first one
> works for me:
>
> https://syzkaller.appspot.com/bug?extid=1dccfcb049726389379c
>
> and that one doesn't have a reproducer either. The bytes look familiar
> though:
>
> Code: c1 e8 03 42 80 3c 20 00 74 05 e8 79 7a a7 00 49 8b 47 10 48 89 05 f6 d8 ef 09 49 8d 7f 08 48 89 f8 48 c1 e8 03 42 80 3c 00 00 <00> 00 e8 57 7a a7 00 49 8b 47 08 48 89 05 dc d8 ef 09 49 8d 7f 18
> All code
> ========
> 0: c1 e8 03 shr $0x3,%eax
> 3: 42 80 3c 20 00 cmpb $0x0,(%rax,%r12,1)
> 8: 74 05 je 0xf
> a: e8 79 7a a7 00 callq 0xa77a88
> f: 49 8b 47 10 mov 0x10(%r15),%rax
> 13: 48 89 05 f6 d8 ef 09 mov %rax,0x9efd8f6(%rip) # 0x9efd910
> 1a: 49 8d 7f 08 lea 0x8(%r15),%rdi
> 1e: 48 89 f8 mov %rdi,%rax
> 21: 48 c1 e8 03 shr $0x3,%rax
> 25: 42 80 3c 00 00 cmpb $0x0,(%rax,%r8,1)
> 2a:* 00 00 add %al,(%rax) <-- trapping instruction
> 2c: e8 57 7a a7 00 callq 0xa77a88
> 31: 49 8b 47 08 mov 0x8(%r15),%rax
> 35: 48 89 05 dc d8 ef 09 mov %rax,0x9efd8dc(%rip) # 0x9efd918
> 3c: 49 8d 7f 18 lea 0x18(%r15),%rdi
>
> 4 zero bytes again. And that .config has kasan stuff enabled too so
> could the failure be related to having kasan stuff enabled and it
> messing up offsets?
>
> That is, provided this is the mechanism how it would happen. We still
> don't know what and when wrote those zeroes in there. Not having a
> reproducer is nasty but looking at those reports above and if I'm
> reading this correctly, rIP points to
>
> RIP: 0010:update_pvclock_gtod arch/x86/kvm/x86.c:1743 [inline]
>
> each time and the URL says they're 9 crashes total. And each have
> happened at that rIP. So all we'd need is set a watchpoint when that
> address is being written and dump stuff.
>
> Dmitry, can the syzkaller do debugging stuff like that?
syzbot does not have direct support for such things.
It uses CONFIG_DEBUG_AID_FOR_SYZBOT=y:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#no-custom-patches
But that's generally useful for linux-next only and the clang build is
on the upstream tree...
Options I see:
1. Add stricter debug checks for code that overwrites code. Then maybe
we can catch it red handed.
2. Setup clang instance on linux-next
3. Run syzkaller locally with custom patches.
> > Following my hypothesis about having a bad address calculation; the
> > tricky part is I'd need to look through the relocations and try to see
> > if any could resolve to the address that was accidentally modified. I
> > suspect objtool could be leveraged for that;
>
> If you can find this at compile time...
>
> > maybe it could check whether each `struct jump_entry`'s `target`
> > member referred to either a NOP or a CMP, and error otherwise? (Do we
> > have other non-NOP or CMP targets? IDK)
>
> Follow jump_label_transform() - it does verify what it is going to
> patch. And while I'm looking at this, I realize that the jump labels
> patch 5 bytes but the above zeroes are 4 bytes. In the other opcode
> bytes I decoded it is 4 bytes too. So this might not be caused by the
> jump labels patching...
>
> > This hypothesis might also be incorrect, and thus would be chasing a
> > red herring...not really sure how else to pursue debugging this.
>
> Yeah, this one is tricky to debug.
>
> --
> Regards/Gruss,
> Boris.
>
> https://people.kernel.org/tglx/notes-about-netiquette
next prev parent reply other threads:[~2020-09-23 9:25 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-19 8:32 general protection fault in perf_misc_flags syzbot
2020-09-19 11:08 ` Borislav Petkov
2020-09-21 5:54 ` Dmitry Vyukov
2020-09-21 8:08 ` Dmitry Vyukov
2020-09-21 20:59 ` Nick Desaulniers
2020-09-21 22:13 ` Borislav Petkov
2020-09-22 18:56 ` Nick Desaulniers
2020-09-22 19:29 ` Borislav Petkov
2020-09-23 9:03 ` Borislav Petkov
2020-09-23 9:24 ` Dmitry Vyukov [this message]
2020-09-23 10:34 ` Borislav Petkov
2020-09-23 15:20 ` Dmitry Vyukov
2020-09-25 12:22 ` Dmitry Vyukov
2020-09-26 0:32 ` Nick Desaulniers
2020-09-26 6:46 ` Dmitry Vyukov
2020-09-26 17:14 ` Borislav Petkov
2020-09-26 11:21 ` Borislav Petkov
2020-09-26 12:08 ` Dmitry Vyukov
2020-09-22 5:15 ` Dmitry Vyukov
2020-09-22 5:16 ` Dmitry Vyukov
2020-09-27 14:57 ` Borislav Petkov
2020-09-28 5:18 ` Dmitry Vyukov
2020-09-28 6:06 ` Dmitry Vyukov
2020-09-28 8:38 ` Borislav Petkov
2020-09-28 8:40 ` Dmitry Vyukov
2020-09-28 8:54 ` Borislav Petkov
2020-09-28 10:33 ` Dmitry Vyukov
2020-09-28 20:23 ` Borislav Petkov
2020-09-29 8:33 ` Borislav Petkov
2020-09-29 13:29 ` Dmitry Vyukov
2020-09-30 16:17 ` Borislav Petkov
2020-09-30 16:23 ` Dmitry Vyukov
2020-09-30 16:29 ` Dmitry Vyukov
2020-09-30 16:31 ` Borislav Petkov
2020-10-01 10:23 ` Dmitry Vyukov
2020-10-01 11:05 ` Borislav Petkov
2020-09-28 20:51 ` Nick Desaulniers
2020-09-28 21:19 ` Andy Lutomirski
2020-09-28 7:25 ` Marco Elver
2020-09-28 20:32 ` Nick Desaulniers
2020-09-29 13:27 ` Dmitry Vyukov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CACT4Y+Y4-vqdv01ebyzhUoggUCUyvbhjut7Wvj=r4dBfyxLeng@mail.gmail.com' \
--to=dvyukov@google.com \
--cc=acme@kernel.org \
--cc=alexander.shishkin@linux.intel.com \
--cc=bp@alien8.de \
--cc=clang-built-linux@googlegroups.com \
--cc=hpa@zytor.com \
--cc=jolsa@redhat.com \
--cc=jpoimboe@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=ndesaulniers@google.com \
--cc=peterz@infradead.org \
--cc=syzbot+ce179bc99e64377c24bc@syzkaller.appspotmail.com \
--cc=syzkaller-bugs@googlegroups.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).