KCSAN + KVM = host reset

KCSAN + KVM = host reset

[Qian Cai]

Running a simple thing on this AMD host would trigger a reset right away.
Unselect KCSAN kconfig makes everything work fine (the host would also
reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).

/usr/libexec/qemu-kvm -name ubuntu-18.04-server-cloudimg -cpu host -smp 2 -m 2G -hda ubuntu-18.04-server-cloudimg.qcow2 -cdrom ubuntu-18.04-server-cloudimg.iso -nic user,hostfwd=tcp::2222-:22 -serial mon:stdio -nographic

With this config on today’s linux-next,

https://raw.githubusercontent.com/cailca/linux-mm/master/kcsan.config

Cherry-picked a few commits from -rcu (in case if it ever matters)

48b1fc1 kcsan: Add option to allow watcher interruptions
2402d0e kcsan: Add option for verbose reporting
43f7646 x86/mm/pat: Mark an intentional data race

=== console output ===
Kernel 5.6.0-next-20200408+ on an x86_64

hp-dl385g10-05 login: 

<...host reset...>

=======================
HPE ProLiant System BIOS A40 v1.20 (03/09/2018)
(C) Copyright 1982-2018 Hewlett Packard Enterprise Development LP
Early system initialization, please wait... 


iLO 5 IPv4: 10.73.196.44
iLO 5 IPv6: FE80::D6C9:EFFF:FECE:717E

  2%: Early Processor Initialization
  4%: Processor Root Ports Initialization
  8%: SMBIOS Table Initialization
 12%: HPE SmartMemory Initialization
 17%: iLO Embedded Health Initialization
 21%: ACPI Table Initialization
 25%: System Security Initialization
 30%: BIOS Configuration Initialization
 39%: Early PCI Initialization - Start
 47%: Early PCI Initialization - Complete
 60%: Switching console output to Primary Video. Please wait…
========

# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              32
On-line CPU(s) list: 0-31
Thread(s) per core:  2
Core(s) per socket:  8
Socket(s):           2
NUMA node(s):        8
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               1
Model name:          AMD EPYC 7251 8-Core Processor
Stepping:            2
CPU MHz:             2830.383
CPU max MHz:         2100.0000
CPU min MHz:         1200.0000
BogoMIPS:            4191.58
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           64K
L2 cache:            512K
L3 cache:            4096K
NUMA node0 CPU(s):   0,1,16,17
NUMA node1 CPU(s):   2,3,18,19
NUMA node2 CPU(s):   4,5,20,21
NUMA node3 CPU(s):   6,7,22,23
NUMA node4 CPU(s):   8,9,24,25
NUMA node5 CPU(s):   10,11,26,27
NUMA node6 CPU(s):   12,13,28,29
NUMA node7 CPU(s):   14,15,30,31
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca

# cat /sys/kernel/debug/kcsan 
enabled: 1
used_watchpoints: 0
setup_watchpoints: 13777602
data_races: 47
assert_failures: 0
no_capacity: 598865
report_races: 0
races_unknown_origin: 226
unencodable_accesses: 0
encoding_false_positives: 0

Re: KCSAN + KVM = host reset

[Paolo Bonzini]

On 08/04/20 22:59, Qian Cai wrote:
> Running a simple thing on this AMD host would trigger a reset right away.
> Unselect KCSAN kconfig makes everything work fine (the host would also
> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).

Is this a regression or something you've just started to play with?  (If
anything, the assembly language conversion of the AMD world switch that
is in linux-next could have reduced the likelihood of such a failure,
not increased it).

Paolo

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> On 08/04/20 22:59, Qian Cai wrote:
>> Running a simple thing on this AMD host would trigger a reset right away.
>> Unselect KCSAN kconfig makes everything work fine (the host would also
>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
> 
> Is this a regression or something you've just started to play with?  (If
> anything, the assembly language conversion of the AMD world switch that
> is in linux-next could have reduced the likelihood of such a failure,
> not increased it).

I don’t remember I had tried this combination before, so don’t know if it is a
regression or not.

Re: KCSAN + KVM = host reset

[Marco Elver]

On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >
> > On 08/04/20 22:59, Qian Cai wrote:
> >> Running a simple thing on this AMD host would trigger a reset right away.
> >> Unselect KCSAN kconfig makes everything work fine (the host would also
> >> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
> >
> > Is this a regression or something you've just started to play with?  (If
> > anything, the assembly language conversion of the AMD world switch that
> > is in linux-next could have reduced the likelihood of such a failure,
> > not increased it).
>
> I don’t remember I had tried this combination before, so don’t know if it is a
> regression or not.

What happens with KASAN? My guess is that, since it also happens with
"off", something that should not be instrumented is being
instrumented.

What happens if you put a 'KCSAN_SANITIZE := n' into
arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this
exact system, I'd ask you to narrow it down by placing 'KCSAN_SANITIZE
:= n' into suspect subsystems' Makefiles. Once you get it to work with
that, we can refine the solution.

Thanks,
-- Marco

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 9, 2020, at 3:03 AM, Marco Elver <elver@google.com> wrote:
> 
> On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
>> 
>> 
>> 
>>> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>> 
>>> On 08/04/20 22:59, Qian Cai wrote:
>>>> Running a simple thing on this AMD host would trigger a reset right away.
>>>> Unselect KCSAN kconfig makes everything work fine (the host would also
>>>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
>>> 
>>> Is this a regression or something you've just started to play with?  (If
>>> anything, the assembly language conversion of the AMD world switch that
>>> is in linux-next could have reduced the likelihood of such a failure,
>>> not increased it).
>> 
>> I don’t remember I had tried this combination before, so don’t know if it is a
>> regression or not.
> 
> What happens with KASAN? My guess is that, since it also happens with
> "off", something that should not be instrumented is being
> instrumented.

No, KASAN + KVM works fine.

> 
> What happens if you put a 'KCSAN_SANITIZE := n' into
> arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this

Yes, that works, but this below alone does not work,

KCSAN_SANITIZE_kvm-amd.o := n

I have been able to reproduce this on a few AMD hosts.

> exact system, I'd ask you to narrow it down by placing 'KCSAN_SANITIZE
> := n' into suspect subsystems' Makefiles. Once you get it to work with
> that, we can refine the solution.
> 
> Thanks,
> -- Marco

Re: KCSAN + KVM = host reset

[Marco Elver]

On Thu, 9 Apr 2020 at 17:10, Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On Apr 9, 2020, at 3:03 AM, Marco Elver <elver@google.com> wrote:
> >
> > On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
> >>
> >>
> >>
> >>> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >>>
> >>> On 08/04/20 22:59, Qian Cai wrote:
> >>>> Running a simple thing on this AMD host would trigger a reset right away.
> >>>> Unselect KCSAN kconfig makes everything work fine (the host would also
> >>>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
> >>>
> >>> Is this a regression or something you've just started to play with?  (If
> >>> anything, the assembly language conversion of the AMD world switch that
> >>> is in linux-next could have reduced the likelihood of such a failure,
> >>> not increased it).
> >>
> >> I don’t remember I had tried this combination before, so don’t know if it is a
> >> regression or not.
> >
> > What happens with KASAN? My guess is that, since it also happens with
> > "off", something that should not be instrumented is being
> > instrumented.
>
> No, KASAN + KVM works fine.
>
> >
> > What happens if you put a 'KCSAN_SANITIZE := n' into
> > arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this
>
> Yes, that works, but this below alone does not work,
>
> KCSAN_SANITIZE_kvm-amd.o := n

There are some other files as well, that you could try until you hit
the right one.

But since this is in arch, 'KCSAN_SANITIZE := n' wouldn't be too bad
for now. If you can't narrow it down further, do you want to send a
patch?

Thanks,
-- Marco

> I have been able to reproduce this on a few AMD hosts.
>
> > exact system, I'd ask you to narrow it down by placing 'KCSAN_SANITIZE
> > := n' into suspect subsystems' Makefiles. Once you get it to work with
> > that, we can refine the solution.
> >
> > Thanks,
> > -- Marco
>

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 9, 2020, at 11:22 AM, Marco Elver <elver@google.com> wrote:
> 
> On Thu, 9 Apr 2020 at 17:10, Qian Cai <cai@lca.pw> wrote:
>> 
>> 
>> 
>>> On Apr 9, 2020, at 3:03 AM, Marco Elver <elver@google.com> wrote:
>>> 
>>> On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>>> 
>>>>> On 08/04/20 22:59, Qian Cai wrote:
>>>>>> Running a simple thing on this AMD host would trigger a reset right away.
>>>>>> Unselect KCSAN kconfig makes everything work fine (the host would also
>>>>>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
>>>>> 
>>>>> Is this a regression or something you've just started to play with?  (If
>>>>> anything, the assembly language conversion of the AMD world switch that
>>>>> is in linux-next could have reduced the likelihood of such a failure,
>>>>> not increased it).
>>>> 
>>>> I don’t remember I had tried this combination before, so don’t know if it is a
>>>> regression or not.
>>> 
>>> What happens with KASAN? My guess is that, since it also happens with
>>> "off", something that should not be instrumented is being
>>> instrumented.
>> 
>> No, KASAN + KVM works fine.
>> 
>>> 
>>> What happens if you put a 'KCSAN_SANITIZE := n' into
>>> arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this
>> 
>> Yes, that works, but this below alone does not work,
>> 
>> KCSAN_SANITIZE_kvm-amd.o := n
> 
> There are some other files as well, that you could try until you hit
> the right one.
> 
> But since this is in arch, 'KCSAN_SANITIZE := n' wouldn't be too bad
> for now. If you can't narrow it down further, do you want to send a
> patch?

No, that would be pretty bad because it will disable KCSAN for Intel
KVM as well which is working perfectly fine right now. It is only AMD
is broken.

> 
> Thanks,
> -- Marco
> 
>> I have been able to reproduce this on a few AMD hosts.
>> 
>>> exact system, I'd ask you to narrow it down by placing 'KCSAN_SANITIZE
>>> := n' into suspect subsystems' Makefiles. Once you get it to work with
>>> that, we can refine the solution.
>>> 
>>> Thanks,
>>> -- Marco

Re: KCSAN + KVM = host reset

[Marco Elver]

On Thu, 9 Apr 2020 at 17:30, Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On Apr 9, 2020, at 11:22 AM, Marco Elver <elver@google.com> wrote:
> >
> > On Thu, 9 Apr 2020 at 17:10, Qian Cai <cai@lca.pw> wrote:
> >>
> >>
> >>
> >>> On Apr 9, 2020, at 3:03 AM, Marco Elver <elver@google.com> wrote:
> >>>
> >>> On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
> >>>>
> >>>>
> >>>>
> >>>>> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >>>>>
> >>>>> On 08/04/20 22:59, Qian Cai wrote:
> >>>>>> Running a simple thing on this AMD host would trigger a reset right away.
> >>>>>> Unselect KCSAN kconfig makes everything work fine (the host would also
> >>>>>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
> >>>>>
> >>>>> Is this a regression or something you've just started to play with?  (If
> >>>>> anything, the assembly language conversion of the AMD world switch that
> >>>>> is in linux-next could have reduced the likelihood of such a failure,
> >>>>> not increased it).
> >>>>
> >>>> I don’t remember I had tried this combination before, so don’t know if it is a
> >>>> regression or not.
> >>>
> >>> What happens with KASAN? My guess is that, since it also happens with
> >>> "off", something that should not be instrumented is being
> >>> instrumented.
> >>
> >> No, KASAN + KVM works fine.
> >>
> >>>
> >>> What happens if you put a 'KCSAN_SANITIZE := n' into
> >>> arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this
> >>
> >> Yes, that works, but this below alone does not work,
> >>
> >> KCSAN_SANITIZE_kvm-amd.o := n
> >
> > There are some other files as well, that you could try until you hit
> > the right one.
> >
> > But since this is in arch, 'KCSAN_SANITIZE := n' wouldn't be too bad
> > for now. If you can't narrow it down further, do you want to send a
> > patch?
>
> No, that would be pretty bad because it will disable KCSAN for Intel
> KVM as well which is working perfectly fine right now. It is only AMD
> is broken.

Interesting. Unfortunately I don't have access to an AMD machine right now.

Actually I think it should be:

  KCSAN_SANITIZE_svm.o := n
  KCSAN_SANITIZE_pmu_amd.o := n

If you want to disable KCSAN for kvm-amd.

Thanks,
-- Marco

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 9, 2020, at 12:03 PM, Marco Elver <elver@google.com> wrote:
> 
> On Thu, 9 Apr 2020 at 17:30, Qian Cai <cai@lca.pw> wrote:
>> 
>> 
>> 
>>> On Apr 9, 2020, at 11:22 AM, Marco Elver <elver@google.com> wrote:
>>> 
>>> On Thu, 9 Apr 2020 at 17:10, Qian Cai <cai@lca.pw> wrote:
>>>> 
>>>> 
>>>> 
>>>>> On Apr 9, 2020, at 3:03 AM, Marco Elver <elver@google.com> wrote:
>>>>> 
>>>>> On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>>>>> 
>>>>>>> On 08/04/20 22:59, Qian Cai wrote:
>>>>>>>> Running a simple thing on this AMD host would trigger a reset right away.
>>>>>>>> Unselect KCSAN kconfig makes everything work fine (the host would also
>>>>>>>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
>>>>>>> 
>>>>>>> Is this a regression or something you've just started to play with?  (If
>>>>>>> anything, the assembly language conversion of the AMD world switch that
>>>>>>> is in linux-next could have reduced the likelihood of such a failure,
>>>>>>> not increased it).
>>>>>> 
>>>>>> I don’t remember I had tried this combination before, so don’t know if it is a
>>>>>> regression or not.
>>>>> 
>>>>> What happens with KASAN? My guess is that, since it also happens with
>>>>> "off", something that should not be instrumented is being
>>>>> instrumented.
>>>> 
>>>> No, KASAN + KVM works fine.
>>>> 
>>>>> 
>>>>> What happens if you put a 'KCSAN_SANITIZE := n' into
>>>>> arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this
>>>> 
>>>> Yes, that works, but this below alone does not work,
>>>> 
>>>> KCSAN_SANITIZE_kvm-amd.o := n
>>> 
>>> There are some other files as well, that you could try until you hit
>>> the right one.
>>> 
>>> But since this is in arch, 'KCSAN_SANITIZE := n' wouldn't be too bad
>>> for now. If you can't narrow it down further, do you want to send a
>>> patch?
>> 
>> No, that would be pretty bad because it will disable KCSAN for Intel
>> KVM as well which is working perfectly fine right now. It is only AMD
>> is broken.
> 
> Interesting. Unfortunately I don't have access to an AMD machine right now.
> 
> Actually I think it should be:
> 
>  KCSAN_SANITIZE_svm.o := n
>  KCSAN_SANITIZE_pmu_amd.o := n
> 
> If you want to disable KCSAN for kvm-amd.

KCSAN_SANITIZE_svm.o := n

That alone works fine. I am wondering which functions there could trigger
perhaps some kind of recursing with KCSAN?

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 9, 2020, at 5:28 PM, Qian Cai <cai@lca.pw> wrote:
> 
> 
> 
>> On Apr 9, 2020, at 12:03 PM, Marco Elver <elver@google.com> wrote:
>> 
>> On Thu, 9 Apr 2020 at 17:30, Qian Cai <cai@lca.pw> wrote:
>>> 
>>> 
>>> 
>>>> On Apr 9, 2020, at 11:22 AM, Marco Elver <elver@google.com> wrote:
>>>> 
>>>> On Thu, 9 Apr 2020 at 17:10, Qian Cai <cai@lca.pw> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Apr 9, 2020, at 3:03 AM, Marco Elver <elver@google.com> wrote:
>>>>>> 
>>>>>> On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>>>>>>> 
>>>>>>>> On 08/04/20 22:59, Qian Cai wrote:
>>>>>>>>> Running a simple thing on this AMD host would trigger a reset right away.
>>>>>>>>> Unselect KCSAN kconfig makes everything work fine (the host would also
>>>>>>>>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
>>>>>>>> 
>>>>>>>> Is this a regression or something you've just started to play with?  (If
>>>>>>>> anything, the assembly language conversion of the AMD world switch that
>>>>>>>> is in linux-next could have reduced the likelihood of such a failure,
>>>>>>>> not increased it).
>>>>>>> 
>>>>>>> I don’t remember I had tried this combination before, so don’t know if it is a
>>>>>>> regression or not.
>>>>>> 
>>>>>> What happens with KASAN? My guess is that, since it also happens with
>>>>>> "off", something that should not be instrumented is being
>>>>>> instrumented.
>>>>> 
>>>>> No, KASAN + KVM works fine.
>>>>> 
>>>>>> 
>>>>>> What happens if you put a 'KCSAN_SANITIZE := n' into
>>>>>> arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this
>>>>> 
>>>>> Yes, that works, but this below alone does not work,
>>>>> 
>>>>> KCSAN_SANITIZE_kvm-amd.o := n
>>>> 
>>>> There are some other files as well, that you could try until you hit
>>>> the right one.
>>>> 
>>>> But since this is in arch, 'KCSAN_SANITIZE := n' wouldn't be too bad
>>>> for now. If you can't narrow it down further, do you want to send a
>>>> patch?
>>> 
>>> No, that would be pretty bad because it will disable KCSAN for Intel
>>> KVM as well which is working perfectly fine right now. It is only AMD
>>> is broken.
>> 
>> Interesting. Unfortunately I don't have access to an AMD machine right now.
>> 
>> Actually I think it should be:
>> 
>> KCSAN_SANITIZE_svm.o := n
>> KCSAN_SANITIZE_pmu_amd.o := n
>> 
>> If you want to disable KCSAN for kvm-amd.
> 
> KCSAN_SANITIZE_svm.o := n
> 
> That alone works fine. I am wondering which functions there could trigger
> perhaps some kind of recursing with KCSAN?

Another data point is set CONFIG_KCSAN_INTERRUPT_WATCHER=n alone
also fixed the issue. I saw quite a few interrupt related function in svm.c, so
some interrupt-related recursion going on?

Re: KCSAN + KVM = host reset

[Marco Elver]

On Fri, 10 Apr 2020 at 01:00, Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On Apr 9, 2020, at 5:28 PM, Qian Cai <cai@lca.pw> wrote:
> >
> >
> >
> >> On Apr 9, 2020, at 12:03 PM, Marco Elver <elver@google.com> wrote:
> >>
> >> On Thu, 9 Apr 2020 at 17:30, Qian Cai <cai@lca.pw> wrote:
> >>>
> >>>
> >>>
> >>>> On Apr 9, 2020, at 11:22 AM, Marco Elver <elver@google.com> wrote:
> >>>>
> >>>> On Thu, 9 Apr 2020 at 17:10, Qian Cai <cai@lca.pw> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Apr 9, 2020, at 3:03 AM, Marco Elver <elver@google.com> wrote:
> >>>>>>
> >>>>>> On Wed, 8 Apr 2020 at 23:29, Qian Cai <cai@lca.pw> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Apr 8, 2020, at 5:25 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> >>>>>>>>
> >>>>>>>> On 08/04/20 22:59, Qian Cai wrote:
> >>>>>>>>> Running a simple thing on this AMD host would trigger a reset right away.
> >>>>>>>>> Unselect KCSAN kconfig makes everything work fine (the host would also
> >>>>>>>>> reset If only "echo off > /sys/kernel/debug/kcsan” before running qemu-kvm).
> >>>>>>>>
> >>>>>>>> Is this a regression or something you've just started to play with?  (If
> >>>>>>>> anything, the assembly language conversion of the AMD world switch that
> >>>>>>>> is in linux-next could have reduced the likelihood of such a failure,
> >>>>>>>> not increased it).
> >>>>>>>
> >>>>>>> I don’t remember I had tried this combination before, so don’t know if it is a
> >>>>>>> regression or not.
> >>>>>>
> >>>>>> What happens with KASAN? My guess is that, since it also happens with
> >>>>>> "off", something that should not be instrumented is being
> >>>>>> instrumented.
> >>>>>
> >>>>> No, KASAN + KVM works fine.
> >>>>>
> >>>>>>
> >>>>>> What happens if you put a 'KCSAN_SANITIZE := n' into
> >>>>>> arch/x86/kvm/Makefile? Since it's hard for me to reproduce on this
> >>>>>
> >>>>> Yes, that works, but this below alone does not work,
> >>>>>
> >>>>> KCSAN_SANITIZE_kvm-amd.o := n
> >>>>
> >>>> There are some other files as well, that you could try until you hit
> >>>> the right one.
> >>>>
> >>>> But since this is in arch, 'KCSAN_SANITIZE := n' wouldn't be too bad
> >>>> for now. If you can't narrow it down further, do you want to send a
> >>>> patch?
> >>>
> >>> No, that would be pretty bad because it will disable KCSAN for Intel
> >>> KVM as well which is working perfectly fine right now. It is only AMD
> >>> is broken.
> >>
> >> Interesting. Unfortunately I don't have access to an AMD machine right now.
> >>
> >> Actually I think it should be:
> >>
> >> KCSAN_SANITIZE_svm.o := n
> >> KCSAN_SANITIZE_pmu_amd.o := n
> >>
> >> If you want to disable KCSAN for kvm-amd.
> >
> > KCSAN_SANITIZE_svm.o := n
> >
> > That alone works fine. I am wondering which functions there could trigger
> > perhaps some kind of recursing with KCSAN?
>
> Another data point is set CONFIG_KCSAN_INTERRUPT_WATCHER=n alone
> also fixed the issue. I saw quite a few interrupt related function in svm.c, so
> some interrupt-related recursion going on?

That would contradict what you said about it working if KCSAN is
"off". What kernel are you attempting to use in the VM?

Thanks,
-- Marco

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 10, 2020, at 5:47 AM, Marco Elver <elver@google.com> wrote:
> 
> That would contradict what you said about it working if KCSAN is
> "off". What kernel are you attempting to use in the VM?

Well, I said set KCSAN debugfs to “off” did not help, i.e., it will reset the host running kvm.sh. It is the vanilla ubuntu 18.04 kernel in VM.

github.com/cailca/linux-mm/blob/master/kvm.sh

Re: KCSAN + KVM = host reset

[Marco Elver]

On Fri, 10 Apr 2020 at 13:25, Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On Apr 10, 2020, at 5:47 AM, Marco Elver <elver@google.com> wrote:
> >
> > That would contradict what you said about it working if KCSAN is
> > "off". What kernel are you attempting to use in the VM?

Ah, sorry this was a typo,
  s/working if KCSAN/not working if KCSAN/

> Well, I said set KCSAN debugfs to “off” did not help, i.e., it will reset the host running kvm.sh. It is the vanilla ubuntu 18.04 kernel in VM.
>
> github.com/cailca/linux-mm/blob/master/kvm.sh

So, if you say that CONFIG_KCSAN_INTERRUPT_WATCHER=n works, that
contradicts it not working when KCSAN is "off". Because if KCSAN is
off, it never sets up any watchpoints, and whether or not
KCSAN_INTERRUPT_WATCHER is selected or not shouldn't matter. Does that
make more sense?

But from what you say, it's not the type of kernel run in VM. I just
thought there may be some strange interaction if you also run a KCSAN
kernel inside the VM.

Since I have no way to help debug right now, if you say that
"KCSAN_SANITIZE_svm.o := n" works, I'd suggest that you just send a
patch for that. If you think that's not adequate, it may be possible
to try and find the offending function(s) in that file and add
__no_kcsan to the  function(s) that cause problems.

Thanks,
-- Marco

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 10, 2020, at 7:35 AM, Marco Elver <elver@google.com> wrote:
> 
> On Fri, 10 Apr 2020 at 13:25, Qian Cai <cai@lca.pw> wrote:
>> 
>> 
>> 
>>> On Apr 10, 2020, at 5:47 AM, Marco Elver <elver@google.com> wrote:
>>> 
>>> That would contradict what you said about it working if KCSAN is
>>> "off". What kernel are you attempting to use in the VM?
> 
> Ah, sorry this was a typo,
>  s/working if KCSAN/not working if KCSAN/
> 
>> Well, I said set KCSAN debugfs to “off” did not help, i.e., it will reset the host running kvm.sh. It is the vanilla ubuntu 18.04 kernel in VM.
>> 
>> github.com/cailca/linux-mm/blob/master/kvm.sh
> 
> So, if you say that CONFIG_KCSAN_INTERRUPT_WATCHER=n works, that
> contradicts it not working when KCSAN is "off". Because if KCSAN is
> off, it never sets up any watchpoints, and whether or not
> KCSAN_INTERRUPT_WATCHER is selected or not shouldn't matter. Does that
> make more sense?
> 
> But from what you say, it's not the type of kernel run in VM. I just
> thought there may be some strange interaction if you also run a KCSAN
> kernel inside the VM.
> 
> Since I have no way to help debug right now, if you say that
> "KCSAN_SANITIZE_svm.o := n" works, I'd suggest that you just send a
> patch for that. If you think that's not adequate, it may be possible
> to try and find the offending function(s) in that file and add
> __no_kcsan to the  function(s) that cause problems.

This works,

--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3278,7 +3278,7 @@ static void svm_cancel_injection(struct kvm_vcpu *vcpu)
 
 bool __svm_vcpu_run(unsigned long vmcb_pa, unsigned long *regs);
 
-static void svm_vcpu_run(struct kvm_vcpu *vcpu)
+static __no_kcsan void svm_vcpu_run(struct kvm_vcpu *vcpu)
 {
        struct vcpu_svm *svm = to_svm(vcpu);

Does anyone has any idea why svm_vcpu_run() would be a problem for KCSAN_INTERRUPT_WATCHER=y?

I can only see there are a bunch of assembly code in __svm_vcpu_run() that might be related?

Re: KCSAN + KVM = host reset

[Paolo Bonzini]

On 10/04/20 17:50, Qian Cai wrote:
> This works,
> 
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3278,7 +3278,7 @@ static void svm_cancel_injection(struct kvm_vcpu *vcpu)
>  
>  bool __svm_vcpu_run(unsigned long vmcb_pa, unsigned long *regs);
>  
> -static void svm_vcpu_run(struct kvm_vcpu *vcpu)
> +static __no_kcsan void svm_vcpu_run(struct kvm_vcpu *vcpu)
>  {
>         struct vcpu_svm *svm = to_svm(vcpu);
> 
> Does anyone has any idea why svm_vcpu_run() would be a problem for KCSAN_INTERRUPT_WATCHER=y?

All of svm_vcpu_run() has interrupts disabled anyway, but perhaps KCSAN
checks the interrupt flag?  That could be a problem because
svm_vcpu_run() disables the interrupts with GIF not IF (and in fact
IF=1).

You can try this patch which moves the problematic section inside
the assembly language trampoline:

 
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 27f4684a4c20..6ffa07d42e5e 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3337,8 +3337,6 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
 	 */
 	x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl);
 
-	local_irq_enable();
-
 	__svm_vcpu_run(svm->vmcb_pa, (unsigned long *)&svm->vcpu.arch.regs);
 
	/* Eliminate branch target predictions from guest mode */
@@ -3373,8 +3368,6 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
 
 	reload_tss(vcpu);
 
-	local_irq_disable();
-
 	x86_spec_ctrl_restore_host(svm->spec_ctrl, svm->virt_spec_ctrl);
 
 	vcpu->arch.cr2 = svm->vmcb->save.cr2;
diff --git a/arch/x86/kvm/svm/vmenter.S b/arch/x86/kvm/svm/vmenter.S
index fa1af90067e9..a2608ede0975 100644
--- a/arch/x86/kvm/svm/vmenter.S
+++ b/arch/x86/kvm/svm/vmenter.S
@@ -78,6 +78,7 @@ SYM_FUNC_START(__svm_vcpu_run)
 	pop %_ASM_AX
 
 	/* Enter guest mode */
+	sti
 1:	vmload %_ASM_AX
 	jmp 3f
 2:	cmpb $0, kvm_rebooting
@@ -99,6 +100,8 @@ SYM_FUNC_START(__svm_vcpu_run)
 	ud2
 	_ASM_EXTABLE(5b, 6b)
 7:
+	cli
+
 	/* "POP" @regs to RAX. */
 	pop %_ASM_AX
 

Paolo

Re: KCSAN + KVM = host reset

[Sean Christopherson]

On Fri, Apr 10, 2020 at 11:50:10AM -0400, Qian Cai wrote:
> 
> This works,
> 
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3278,7 +3278,7 @@ static void svm_cancel_injection(struct kvm_vcpu *vcpu)
>  
>  bool __svm_vcpu_run(unsigned long vmcb_pa, unsigned long *regs);
>  
> -static void svm_vcpu_run(struct kvm_vcpu *vcpu)
> +static __no_kcsan void svm_vcpu_run(struct kvm_vcpu *vcpu)
>  {
>         struct vcpu_svm *svm = to_svm(vcpu);
> 
> Does anyone has any idea why svm_vcpu_run() would be a problem for
> KCSAN_INTERRUPT_WATCHER=y?
> 
> I can only see there are a bunch of assembly code in __svm_vcpu_run() that
> might be related?

svm_vcpu_run() does all kinds of interrupt toggling, e.g. the sequence is:

  1. EFLAGS.IF == 0, from caller
  2. clgi()
  3. EFLAGS.IF <= 1
  4. __svm_vcpu_run(), i.e. enter guest
  5. EFLAGS == 0, from VM-Exit
  6. EFLAGS.IF <= 1
  7. stgi()

Re: KCSAN + KVM = host reset

[Qian Cai]

> On Apr 10, 2020, at 7:35 AM, Marco Elver <elver@google.com> wrote:
> 
> On Fri, 10 Apr 2020 at 13:25, Qian Cai <cai@lca.pw> wrote:
>> 
>> 
>> 
>>> On Apr 10, 2020, at 5:47 AM, Marco Elver <elver@google.com> wrote:
>>> 
>>> That would contradict what you said about it working if KCSAN is
>>> "off". What kernel are you attempting to use in the VM?
> 
> Ah, sorry this was a typo,
>  s/working if KCSAN/not working if KCSAN/
> 
>> Well, I said set KCSAN debugfs to “off” did not help, i.e., it will reset the host running kvm.sh. It is the vanilla ubuntu 18.04 kernel in VM.
>> 
>> github.com/cailca/linux-mm/blob/master/kvm.sh
> 
> So, if you say that CONFIG_KCSAN_INTERRUPT_WATCHER=n works, that
> contradicts it not working when KCSAN is "off". Because if KCSAN is
> off, it never sets up any watchpoints, and whether or not
> KCSAN_INTERRUPT_WATCHER is selected or not shouldn't matter. Does that
> make more sense?

Yes, you are right. CONFIG_KCSAN_INTERRUPT_WATCHER=n does not
make it work. It was a mistake when I tested it because there was a stale svm.o
leftover from the previous run, and then it will not trigger a rebuild (a bug?) when
only modify the Makefile to remove KCSAN_SANITIZE := n. Sorry for the misleading
information. I should be checking if svm.o was really recompiled in the first place.

Anyway, I’ll send a patch to add __no_kcsan for svm_vcpu_run() because I tried
to narrow down more with a kcsan_[disable|enable]_current() pair, but it does NOT
work even by enclosing the almost whole function below until Marcro has more ideas?

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 2be5bbae3a40..e58b2d5a575c 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3286,6 +3286,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
        svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP];
        svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP];
 
+       kcsan_disable_current();
        /*
         * A vmexit emulation is required before the vcpu can be executed
         * again.
@@ -3410,6 +3411,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
                svm_handle_mce(svm);
 
        mark_all_clean(svm->vmcb);
+       kcsan_enable_current();
 }
 STACK_FRAME_NON_STANDARD(svm_vcpu_run);
 

 

Re: KCSAN + KVM = host reset

[Marco Elver]

On Fri, 10 Apr 2020 at 21:57, Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On Apr 10, 2020, at 7:35 AM, Marco Elver <elver@google.com> wrote:
> >
> > On Fri, 10 Apr 2020 at 13:25, Qian Cai <cai@lca.pw> wrote:
> >>
> >>
> >>
> >>> On Apr 10, 2020, at 5:47 AM, Marco Elver <elver@google.com> wrote:
> >>>
> >>> That would contradict what you said about it working if KCSAN is
> >>> "off". What kernel are you attempting to use in the VM?
> >
> > Ah, sorry this was a typo,
> >  s/working if KCSAN/not working if KCSAN/
> >
> >> Well, I said set KCSAN debugfs to “off” did not help, i.e., it will reset the host running kvm.sh. It is the vanilla ubuntu 18.04 kernel in VM.
> >>
> >> github.com/cailca/linux-mm/blob/master/kvm.sh
> >
> > So, if you say that CONFIG_KCSAN_INTERRUPT_WATCHER=n works, that
> > contradicts it not working when KCSAN is "off". Because if KCSAN is
> > off, it never sets up any watchpoints, and whether or not
> > KCSAN_INTERRUPT_WATCHER is selected or not shouldn't matter. Does that
> > make more sense?
>
> Yes, you are right. CONFIG_KCSAN_INTERRUPT_WATCHER=n does not
> make it work. It was a mistake when I tested it because there was a stale svm.o
> leftover from the previous run, and then it will not trigger a rebuild (a bug?) when
> only modify the Makefile to remove KCSAN_SANITIZE := n. Sorry for the misleading
> information. I should be checking if svm.o was really recompiled in the first place.
>
> Anyway, I’ll send a patch to add __no_kcsan for svm_vcpu_run() because I tried
> to narrow down more with a kcsan_[disable|enable]_current() pair, but it does NOT
> work even by enclosing the almost whole function below until Marcro has more ideas?

This is expected. Instrumentation is not removed if you add
kcsan_{disable,enable}_current() (it has the same effect as a
localized "off"). Since it seems just the instrumentation and
associated calls before every memory access is enough, this won't
work. The attribute __no_kcsan removes instrumentation entirely from
the function. If the non-instrumented code should be reduced, it is
conceivable to take the problematic portion of code and factor it into
a function that has attribute '__no_kcsan_or_inline'.

Thanks,
-- Marco

> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 2be5bbae3a40..e58b2d5a575c 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3286,6 +3286,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
>         svm->vmcb->save.rsp = vcpu->arch.regs[VCPU_REGS_RSP];
>         svm->vmcb->save.rip = vcpu->arch.regs[VCPU_REGS_RIP];
>
> +       kcsan_disable_current();
>         /*
>          * A vmexit emulation is required before the vcpu can be executed
>          * again.
> @@ -3410,6 +3411,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
>                 svm_handle_mce(svm);
>
>         mark_all_clean(svm->vmcb);
> +       kcsan_enable_current();
>  }
>  STACK_FRAME_NON_STANDARD(svm_vcpu_run);
>
>
>
>
>
>