WARNING in vcpu_enter_guest

WARNING in vcpu_enter_guest

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    ad062195 Merge tag 'platform-drivers-x86-v5.4-1' of git://..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=154910ad600000
kernel config:  https://syzkaller.appspot.com/x/.config?x=f9fc16a6374d5fd0
dashboard link: https://syzkaller.appspot.com/bug?extid=00be5da1d75f1cc95f6b
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+00be5da1d75f1cc95f6b@syzkaller.appspotmail.com

WARNING: CPU: 1 PID: 15173 at arch/x86/kvm/x86.c:8007  
vcpu_enter_guest+0x4b29/0x5e90 arch/x86/kvm/x86.c:8007
Kernel panic - not syncing: panic_on_warn set ...
CPU: 1 PID: 15173 Comm: syz-executor.5 Not tainted 5.3.0+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x172/0x1f0 lib/dump_stack.c:113
  panic+0x2dc/0x755 kernel/panic.c:219
  __warn.cold+0x20/0x4c kernel/panic.c:576
  report_bug+0x263/0x2b0 lib/bug.c:186
  fixup_bug arch/x86/kernel/traps.c:179 [inline]
  fixup_bug arch/x86/kernel/traps.c:174 [inline]
  do_error_trap+0x11b/0x200 arch/x86/kernel/traps.c:272
  do_invalid_op+0x37/0x50 arch/x86/kernel/traps.c:291
  invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1028
RIP: 0010:vcpu_enter_guest+0x4b29/0x5e90 arch/x86/kvm/x86.c:8007
Code: 00 fc ff df 48 c1 ea 03 0f b6 04 02 84 c0 74 08 3c 03 0f 8e e9 11 00  
00 41 83 a5 20 27 00 00 fb e9 5c bf ff ff e8 97 9e 65 00 <0f> 0b e9 98 be  
ff ff e8 8b 9e 65 00 31 ff be 07 00 00 00 e8 ef 97
RSP: 0018:ffff8880572bfa10 EFLAGS: 00010046
RAX: 0000000000040000 RBX: 0000000000000000 RCX: ffffc900109ce000
RDX: 0000000000040000 RSI: ffffffff810d3599 RDI: 0000000000000007
RBP: ffff8880572bfb20 R08: ffff8880576b62c0 R09: ffffed100aed6c59
R10: ffffed100aed6c58 R11: ffff8880576b62c7 R12: ffff88805fa9866c
R13: ffff88805fa98640 R14: 0000000000000001 R15: ffff88805fa98670
  vcpu_run arch/x86/kvm/x86.c:8159 [inline]
  kvm_arch_vcpu_ioctl_run+0x464/0x1750 arch/x86/kvm/x86.c:8367
  kvm_vcpu_ioctl+0x4dc/0xfd0 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2765
  vfs_ioctl fs/ioctl.c:46 [inline]
  file_ioctl fs/ioctl.c:509 [inline]
  do_vfs_ioctl+0xdb6/0x13e0 fs/ioctl.c:696
  ksys_ioctl+0xab/0xd0 fs/ioctl.c:713
  __do_sys_ioctl fs/ioctl.c:720 [inline]
  __se_sys_ioctl fs/ioctl.c:718 [inline]
  __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:718
  do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
  entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4598e9
Code: fd b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7  
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff  
ff 0f 83 cb b7 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f84f3c93c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004598e9
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000005
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f84f3c946d4
R13: 00000000004c2c68 R14: 00000000004d6330 R15: 00000000ffffffff
Kernel Offset: disabled
Rebooting in 86400 seconds..


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

Re: WARNING in vcpu_enter_guest

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit:    5076190d mm: slub: be more careful about the double cmpxch..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=143ca61de00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
dashboard link: https://syzkaller.appspot.com/bug?extid=00be5da1d75f1cc95f6b
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10bb4023e00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+00be5da1d75f1cc95f6b@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 0 PID: 9833 at arch/x86/kvm/x86.c:2447 kvm_guest_time_update arch/x86/kvm/x86.c:2447 [inline]
WARNING: CPU: 0 PID: 9833 at arch/x86/kvm/x86.c:2447 vcpu_enter_guest+0x3cf3/0x6120 arch/x86/kvm/x86.c:8175
Kernel panic - not syncing: panic_on_warn set ...
CPU: 0 PID: 9833 Comm: syz-executor.0 Not tainted 5.6.0-rc6-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x188/0x20d lib/dump_stack.c:118
 panic+0x2e3/0x75c kernel/panic.c:221
 __warn.cold+0x2f/0x35 kernel/panic.c:582
 report_bug+0x27b/0x2f0 lib/bug.c:195
 fixup_bug arch/x86/kernel/traps.c:174 [inline]
 fixup_bug arch/x86/kernel/traps.c:169 [inline]
 do_error_trap+0x12b/0x220 arch/x86/kernel/traps.c:267
 do_invalid_op+0x32/0x40 arch/x86/kernel/traps.c:286
 invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027
RIP: 0010:kvm_guest_time_update arch/x86/kvm/x86.c:2447 [inline]
RIP: 0010:vcpu_enter_guest+0x3cf3/0x6120 arch/x86/kvm/x86.c:8175
Code: f3 7e 0f 94 c3 31 ff 89 de e8 d9 03 64 00 84 db 0f 84 62 ea ff ff e8 9c 02 64 00 e8 fb 43 f2 ff e9 53 ea ff ff e8 8d 02 64 00 <0f> 0b e9 e7 dc ff ff e8 81 02 64 00 bf 00 94 35 77 45 31 e4 4c 69
RSP: 0018:ffffc900024afb50 EFLAGS: 00010293
RAX: ffff888097b88040 RBX: fffffffffffff8d2 RCX: ffffffff810dff78
RDX: 0000000000000000 RSI: ffffffff810e2293 RDI: 0000000000000007
RBP: ffffc900024afcc0 R08: ffff888097b88040 R09: fffffbfff180e58f
R10: fffffbfff180e58e R11: ffffffff8c072c77 R12: 0000000000000000
R13: ffffc90002521000 R14: ffff88808e620378 R15: ffff88808e620340
 vcpu_run arch/x86/kvm/x86.c:8513 [inline]
 kvm_arch_vcpu_ioctl_run+0x41c/0x1790 arch/x86/kvm/x86.c:8735
 kvm_vcpu_ioctl+0x493/0xe60 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2932
 vfs_ioctl fs/ioctl.c:47 [inline]
 ksys_ioctl+0x11a/0x180 fs/ioctl.c:763
 __do_sys_ioctl fs/ioctl.c:772 [inline]
 __se_sys_ioctl fs/ioctl.c:770 [inline]
 __x64_sys_ioctl+0x6f/0xb0 fs/ioctl.c:770
 do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45c849
Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f5029f2ec78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f5029f2f6d4 RCX: 000000000045c849
RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000005
RBP: 000000000076bf00 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00000000000003be R14: 00000000004c647e R15: 000000000076bf0c
Kernel Offset: disabled
Rebooting in 86400 seconds..

Re: WARNING in vcpu_enter_guest

[Sean Christopherson]

On Thu, Mar 19, 2020 at 03:35:16AM -0700, syzbot wrote:
> syzbot has found a reproducer for the following crash on:
> 
> HEAD commit:    5076190d mm: slub: be more careful about the double cmpxch..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=143ca61de00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
> dashboard link: https://syzkaller.appspot.com/bug?extid=00be5da1d75f1cc95f6b
> compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10bb4023e00000
> 
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+00be5da1d75f1cc95f6b@syzkaller.appspotmail.com

Reproduced with a little tweaking of the reproducer, debug in progress.

Re: WARNING in vcpu_enter_guest

[Paolo Bonzini]

On 19/03/20 15:49, Sean Christopherson wrote:
> On Thu, Mar 19, 2020 at 03:35:16AM -0700, syzbot wrote:
>> syzbot has found a reproducer for the following crash on:
>>
>> HEAD commit:    5076190d mm: slub: be more careful about the double cmpxch..
>> git tree:       upstream
>> console output: https://syzkaller.appspot.com/x/log.txt?x=143ca61de00000
>> kernel config:  https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
>> dashboard link: https://syzkaller.appspot.com/bug?extid=00be5da1d75f1cc95f6b
>> compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
>> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10bb4023e00000
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: syzbot+00be5da1d75f1cc95f6b@syzkaller.appspotmail.com
> Reproduced with a little tweaking of the reproducer, debug in progress.
> 

I think the WARN_ON at x86.c:2447 is just bogus.  You can always get it
to trigger if garbage is passed to KVM_SET_CLOCK.

Paolo

Re: WARNING in vcpu_enter_guest

[Sean Christopherson]

On Thu, Mar 19, 2020 at 04:14:55PM +0100, Paolo Bonzini wrote:
> On 19/03/20 15:49, Sean Christopherson wrote:
> > On Thu, Mar 19, 2020 at 03:35:16AM -0700, syzbot wrote:
> >> syzbot has found a reproducer for the following crash on:
> >>
> >> HEAD commit:    5076190d mm: slub: be more careful about the double cmpxch..
> >> git tree:       upstream
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=143ca61de00000
> >> kernel config:  https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=00be5da1d75f1cc95f6b
> >> compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> >> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10bb4023e00000
> >>
> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> >> Reported-by: syzbot+00be5da1d75f1cc95f6b@syzkaller.appspotmail.com
> > Reproduced with a little tweaking of the reproducer, debug in progress.
> > 
> 
> I think the WARN_ON at x86.c:2447 is just bogus.  You can always get it
> to trigger if garbage is passed to KVM_SET_CLOCK.

Yep.  I worked through logic/math, mostly to gain a wee bit of knowledge
about the clock stuff, and it's sound.  The KVM_SET_CLOCK from syzkaller
is simply making time go backwards.

Re: WARNING in vcpu_enter_guest

[Sean Christopherson]

On Thu, Mar 19, 2020 at 10:35:49AM -0700, Sean Christopherson wrote:
> On Thu, Mar 19, 2020 at 04:14:55PM +0100, Paolo Bonzini wrote:
> > On 19/03/20 15:49, Sean Christopherson wrote:
> > > On Thu, Mar 19, 2020 at 03:35:16AM -0700, syzbot wrote:
> > >> syzbot has found a reproducer for the following crash on:
> > >>
> > >> HEAD commit:    5076190d mm: slub: be more careful about the double cmpxch..
> > >> git tree:       upstream
> > >> console output: https://syzkaller.appspot.com/x/log.txt?x=143ca61de00000
> > >> kernel config:  https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
> > >> dashboard link: https://syzkaller.appspot.com/bug?extid=00be5da1d75f1cc95f6b
> > >> compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> > >> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10bb4023e00000
> > >>
> > >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > >> Reported-by: syzbot+00be5da1d75f1cc95f6b@syzkaller.appspotmail.com
> > > Reproduced with a little tweaking of the reproducer, debug in progress.
> > > 
> > 
> > I think the WARN_ON at x86.c:2447 is just bogus.  You can always get it
> > to trigger if garbage is passed to KVM_SET_CLOCK.
> 
> Yep.  I worked through logic/math, mostly to gain a wee bit of knowledge
> about the clock stuff, and it's sound.  The KVM_SET_CLOCK from syzkaller
> is simply making time go backwards.

Actually, would it make sense to return -EINVAL for KVM_SET_CLOCK if the
user tries to make kvmclock_offset go backwards?

Re: WARNING in vcpu_enter_guest

[Paolo Bonzini]

On 19/03/20 18:39, Sean Christopherson wrote:
>> Yep.  I worked through logic/math, mostly to gain a wee bit of knowledge
>> about the clock stuff, and it's sound.  The KVM_SET_CLOCK from syzkaller
>> is simply making time go backwards.
> Actually, would it make sense to return -EINVAL for KVM_SET_CLOCK if the
> user tries to make kvmclock_offset go backwards?

No, it is possible to do that depending on the clock setup on the live
migration source.  You could cause the warning anyway by setting the
clock to a very high (signed) value so that kernel_ns + kvmclock_offset
overflows.

Paolo

Re: WARNING in vcpu_enter_guest

[Thomas Gleixner]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 19/03/20 18:39, Sean Christopherson wrote:
>>> Yep.  I worked through logic/math, mostly to gain a wee bit of knowledge
>>> about the clock stuff, and it's sound.  The KVM_SET_CLOCK from syzkaller
>>> is simply making time go backwards.
>> Actually, would it make sense to return -EINVAL for KVM_SET_CLOCK if the
>> user tries to make kvmclock_offset go backwards?
>
> No, it is possible to do that depending on the clock setup on the live
> migration source.  You could cause the warning anyway by setting the
> clock to a very high (signed) value so that kernel_ns + kvmclock_offset
> overflows.

If that overflow happens, then the original and the new host have an
uptime difference in the range of >200 hundreds of years. Very realistic
scenario...

Of course this can happen if you feed crap into the interface, but do
you really think that forwarding all crap to a guest is the right thing
to do?

As we all know the hypervisor orchestration stuff is perfect and would
never feed crap into the kernel which happily proliferates that crap to
the guest...

Seriously??

Thanks,

        tglx

Re: WARNING in vcpu_enter_guest

[Paolo Bonzini]

On 20/03/20 01:18, Thomas Gleixner wrote:
>> No, it is possible to do that depending on the clock setup on the live
>> migration source.  You could cause the warning anyway by setting the
>> clock to a very high (signed) value so that kernel_ns + kvmclock_offset
>> overflows.
>
> If that overflow happens, then the original and the new host have an
> uptime difference in the range of >200 hundreds of years. Very realistic
> scenario...
> 
> Of course this can happen if you feed crap into the interface, but do
> you really think that forwarding all crap to a guest is the right thing
> to do?
> 
> As we all know the hypervisor orchestration stuff is perfect and would
> never feed crap into the kernel which happily proliferates that crap to
> the guest...

But the point is, is there a sensible way to detect it?  Only allowing
>= -2^62 and < 2^62 or something like that is an ad hoc fix for a
warning that probably will never trigger outside fuzzing.  I would
expect that passing the wrong sign is a more likely mistake than being
off by 2^63.

This data is available everywhere between strace, kernel tracepoints and
QEMU tracepoints or guest checkpoint (live migration) data.  I just
don't see much advantage in keeping the warning.

Paolo

Re: WARNING in vcpu_enter_guest

[Thomas Gleixner]

Paolo Bonzini <pbonzini@redhat.com> writes:
> On 20/03/20 01:18, Thomas Gleixner wrote:
>>> No, it is possible to do that depending on the clock setup on the live
>>> migration source.  You could cause the warning anyway by setting the
>>> clock to a very high (signed) value so that kernel_ns + kvmclock_offset
>>> overflows.
>>
>> If that overflow happens, then the original and the new host have an
>> uptime difference in the range of >200 hundreds of years. Very realistic
>> scenario...
>> 
>> Of course this can happen if you feed crap into the interface, but do
>> you really think that forwarding all crap to a guest is the right thing
>> to do?
>> 
>> As we all know the hypervisor orchestration stuff is perfect and would
>> never feed crap into the kernel which happily proliferates that crap to
>> the guest...
>
> But the point is, is there a sensible way to detect it?  Only allowing
> >= -2^62 and < 2^62 or something like that is an ad hoc fix for a
> warning that probably will never trigger outside fuzzing.  I would
> expect that passing the wrong sign is a more likely mistake than being
> off by 2^63.
>
> This data is available everywhere between strace, kernel tracepoints and
> QEMU tracepoints or guest checkpoint (live migration) data.  I just
> don't see much advantage in keeping the warning.

The warning is useless. But you want a sanity check in the ioctl and
return -EMORON if it is out of bounds simply because the guest will
malfunction if your offset is bogus. Look at the timekeeping and time
namespace sanity checks.

Thanks,

        tglx

Re: WARNING in vcpu_enter_guest

[syzbot]

syzbot has bisected this bug to:

commit 9446e6fce0ab9dfd44b96f630b4e3a0a0ab879fd
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Wed Feb 12 12:27:10 2020 +0000

    KVM: x86: fix WARN_ON check of an unsigned less than zero

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=1744891de00000
start commit:   5076190d mm: slub: be more careful about the double cmpxch..
git tree:       upstream
final crash:    https://syzkaller.appspot.com/x/report.txt?x=14c4891de00000
console output: https://syzkaller.appspot.com/x/log.txt?x=10c4891de00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=9f894bd92023de02
dashboard link: https://syzkaller.appspot.com/bug?extid=00be5da1d75f1cc95f6b
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10bb4023e00000

Reported-by: syzbot+00be5da1d75f1cc95f6b@syzkaller.appspotmail.com
Fixes: 9446e6fce0ab ("KVM: x86: fix WARN_ON check of an unsigned less than zero")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection