[PATCH] KVM: X86: Fix host dr6 miss restore

[PATCH] KVM: X86: Fix host dr6 miss restore

[Wanpeng Li]

From: Wanpeng Li <wanpeng.li@hotmail.com>

Reported by syzkaller:

   WARNING: CPU: 0 PID: 12927 at arch/x86/kernel/traps.c:780 do_debug+0x222/0x250
   CPU: 0 PID: 12927 Comm: syz-executor Tainted: G           OE    4.15.0-rc2+ #16
   RIP: 0010:do_debug+0x222/0x250
   Call Trace:
    <#DB>
    debug+0x3e/0x70
   RIP: 0010:copy_user_enhanced_fast_string+0x10/0x20
    </#DB>
    _copy_from_user+0x5b/0x90
    SyS_timer_create+0x33/0x80
    entry_SYSCALL_64_fastpath+0x23/0x9a

The syzkaller will mmap a buffer which is also the struct sigevent parameter of 
timer_create(), it will also call perf_event_open() to set a BP for the buffer,
so when the implementation of timer_create() in kernel tries to get the struct 
sigevent parameter by copy_from_user(), rep movsb triggers the BP. The syzkaller 
testcase also sets the debug registers for the guest, however, the kvm just 
restores host debug registers when we have active breakpoints. I can observe 
the dr6 single step bit is set and !hw_breakpoint_active() sporadically by print 
when running the testcase heavy multithreading. The do_debug() which is triggered 
by rep movsb will splash when (dr6 & DR_STEP && !user_mode(regs)). 

This patch fixes it by restoring host dr6 unconditionally before preempt/irq 
enable.

Reported-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krčmář <rkrcmar@redhat.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
---
 arch/x86/kvm/x86.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0c5d55c..a6370fd 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7065,6 +7065,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 	 */
 	if (hw_breakpoint_active())
 		hw_breakpoint_restore();
+	else
+		set_debugreg(current->thread.debugreg6, 6);
 
 	vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
 
-- 
2.7.4

Re: [PATCH] KVM: X86: Fix host dr6 miss restore

[David Hildenbrand]

On 08.12.2017 10:12, Wanpeng Li wrote:
> From: Wanpeng Li <wanpeng.li@hotmail.com>
> 
> Reported by syzkaller:
> 
>    WARNING: CPU: 0 PID: 12927 at arch/x86/kernel/traps.c:780 do_debug+0x222/0x250
>    CPU: 0 PID: 12927 Comm: syz-executor Tainted: G           OE    4.15.0-rc2+ #16
>    RIP: 0010:do_debug+0x222/0x250
>    Call Trace:
>     <#DB>
>     debug+0x3e/0x70
>    RIP: 0010:copy_user_enhanced_fast_string+0x10/0x20
>     </#DB>
>     _copy_from_user+0x5b/0x90
>     SyS_timer_create+0x33/0x80
>     entry_SYSCALL_64_fastpath+0x23/0x9a
> 
> The syzkaller will mmap a buffer which is also the struct sigevent parameter of 
> timer_create(), it will also call perf_event_open() to set a BP for the buffer,
> so when the implementation of timer_create() in kernel tries to get the struct 
> sigevent parameter by copy_from_user(), rep movsb triggers the BP. The syzkaller 
> testcase also sets the debug registers for the guest, however, the kvm just 
> restores host debug registers when we have active breakpoints. I can observe 
> the dr6 single step bit is set and !hw_breakpoint_active() sporadically by print 
> when running the testcase heavy multithreading. The do_debug() which is triggered 
> by rep movsb will splash when (dr6 & DR_STEP && !user_mode(regs)). 
> 
> This patch fixes it by restoring host dr6 unconditionally before preempt/irq 
> enable.
> 
> Reported-by: Dmitry Vyukov <dvyukov@google.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Radim Krčmář <rkrcmar@redhat.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Dmitry Vyukov <dvyukov@google.com>
> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
> ---
>  arch/x86/kvm/x86.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 0c5d55c..a6370fd 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -7065,6 +7065,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>  	 */
>  	if (hw_breakpoint_active())
>  		hw_breakpoint_restore();
> +	else
> +		set_debugreg(current->thread.debugreg6, 6);
>  
>  	vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>  
> 

If you haven't seen it, I analyzed this in
https://lkml.org/lkml/2017/11/7/638 but nobody would respond for now to
my suggestion/question.

-- 

Thanks,

David / dhildenb

Re: [PATCH] KVM: X86: Fix host dr6 miss restore

[Wanpeng Li]

2017-12-08 20:39 GMT+08:00 David Hildenbrand <david@redhat.com>:
> On 08.12.2017 10:12, Wanpeng Li wrote:
>> From: Wanpeng Li <wanpeng.li@hotmail.com>
>>
>> Reported by syzkaller:
>>
>>    WARNING: CPU: 0 PID: 12927 at arch/x86/kernel/traps.c:780 do_debug+0x222/0x250
>>    CPU: 0 PID: 12927 Comm: syz-executor Tainted: G           OE    4.15.0-rc2+ #16
>>    RIP: 0010:do_debug+0x222/0x250
>>    Call Trace:
>>     <#DB>
>>     debug+0x3e/0x70
>>    RIP: 0010:copy_user_enhanced_fast_string+0x10/0x20
>>     </#DB>
>>     _copy_from_user+0x5b/0x90
>>     SyS_timer_create+0x33/0x80
>>     entry_SYSCALL_64_fastpath+0x23/0x9a
>>
>> The syzkaller will mmap a buffer which is also the struct sigevent parameter of
>> timer_create(), it will also call perf_event_open() to set a BP for the buffer,
>> so when the implementation of timer_create() in kernel tries to get the struct
>> sigevent parameter by copy_from_user(), rep movsb triggers the BP. The syzkaller
>> testcase also sets the debug registers for the guest, however, the kvm just
>> restores host debug registers when we have active breakpoints. I can observe
>> the dr6 single step bit is set and !hw_breakpoint_active() sporadically by print
>> when running the testcase heavy multithreading. The do_debug() which is triggered
>> by rep movsb will splash when (dr6 & DR_STEP && !user_mode(regs)).
>>
>> This patch fixes it by restoring host dr6 unconditionally before preempt/irq
>> enable.
>>
>> Reported-by: Dmitry Vyukov <dvyukov@google.com>
>> Cc: Paolo Bonzini <pbonzini@redhat.com>
>> Cc: Radim Krčmář <rkrcmar@redhat.com>
>> Cc: David Hildenbrand <david@redhat.com>
>> Cc: Dmitry Vyukov <dvyukov@google.com>
>> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
>> ---
>>  arch/x86/kvm/x86.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>> index 0c5d55c..a6370fd 100644
>> --- a/arch/x86/kvm/x86.c
>> +++ b/arch/x86/kvm/x86.c
>> @@ -7065,6 +7065,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>>        */
>>       if (hw_breakpoint_active())
>>               hw_breakpoint_restore();
>> +     else
>> +             set_debugreg(current->thread.debugreg6, 6);
>>
>>       vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>>
>>
>
> If you haven't seen it, I analyzed this in
> https://lkml.org/lkml/2017/11/7/638 but nobody would respond for now to
> my suggestion/question.

I think it's fine to restore dr6 before preempt/irq enable.

Regards,
Wanpeng Li

Re: [PATCH] KVM: X86: Fix host dr6 miss restore

[David Hildenbrand]

On 10.12.2017 01:44, Wanpeng Li wrote:
> 2017-12-08 20:39 GMT+08:00 David Hildenbrand <david@redhat.com>:
>> On 08.12.2017 10:12, Wanpeng Li wrote:
>>> From: Wanpeng Li <wanpeng.li@hotmail.com>
>>>
>>> Reported by syzkaller:
>>>
>>>    WARNING: CPU: 0 PID: 12927 at arch/x86/kernel/traps.c:780 do_debug+0x222/0x250
>>>    CPU: 0 PID: 12927 Comm: syz-executor Tainted: G           OE    4.15.0-rc2+ #16
>>>    RIP: 0010:do_debug+0x222/0x250
>>>    Call Trace:
>>>     <#DB>
>>>     debug+0x3e/0x70
>>>    RIP: 0010:copy_user_enhanced_fast_string+0x10/0x20
>>>     </#DB>
>>>     _copy_from_user+0x5b/0x90
>>>     SyS_timer_create+0x33/0x80
>>>     entry_SYSCALL_64_fastpath+0x23/0x9a
>>>
>>> The syzkaller will mmap a buffer which is also the struct sigevent parameter of
>>> timer_create(), it will also call perf_event_open() to set a BP for the buffer,
>>> so when the implementation of timer_create() in kernel tries to get the struct
>>> sigevent parameter by copy_from_user(), rep movsb triggers the BP. The syzkaller
>>> testcase also sets the debug registers for the guest, however, the kvm just
>>> restores host debug registers when we have active breakpoints. I can observe
>>> the dr6 single step bit is set and !hw_breakpoint_active() sporadically by print
>>> when running the testcase heavy multithreading. The do_debug() which is triggered
>>> by rep movsb will splash when (dr6 & DR_STEP && !user_mode(regs)).
>>>
>>> This patch fixes it by restoring host dr6 unconditionally before preempt/irq
>>> enable.
>>>
>>> Reported-by: Dmitry Vyukov <dvyukov@google.com>
>>> Cc: Paolo Bonzini <pbonzini@redhat.com>
>>> Cc: Radim Krčmář <rkrcmar@redhat.com>
>>> Cc: David Hildenbrand <david@redhat.com>
>>> Cc: Dmitry Vyukov <dvyukov@google.com>
>>> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
>>> ---
>>>  arch/x86/kvm/x86.c | 2 ++
>>>  1 file changed, 2 insertions(+)
>>>
>>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>>> index 0c5d55c..a6370fd 100644
>>> --- a/arch/x86/kvm/x86.c
>>> +++ b/arch/x86/kvm/x86.c
>>> @@ -7065,6 +7065,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>>>        */
>>>       if (hw_breakpoint_active())
>>>               hw_breakpoint_restore();
>>> +     else
>>> +             set_debugreg(current->thread.debugreg6, 6);
>>>
>>>       vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>>>
>>>
>>
>> If you haven't seen it, I analyzed this in
>> https://lkml.org/lkml/2017/11/7/638 but nobody would respond for now to
>> my suggestion/question.
> 
> I think it's fine to restore dr6 before preempt/irq enable.

That make sense, as I assume this is the first time that a trap would be
delivered.

Reviewed-by: David Hildenbrand <david@redhat.com>

And certainly stable material?

> 
> Regards,
> Wanpeng Li
> 


-- 

Thanks,

David / dhildenb

Re: [PATCH] KVM: X86: Fix host dr6 miss restore

[Paolo Bonzini]

On 10/12/2017 01:44, Wanpeng Li wrote:
> 2017-12-08 20:39 GMT+08:00 David Hildenbrand <david@redhat.com>:
>> On 08.12.2017 10:12, Wanpeng Li wrote:
>>> From: Wanpeng Li <wanpeng.li@hotmail.com>
>>>
>>> Reported by syzkaller:
>>>
>>>    WARNING: CPU: 0 PID: 12927 at arch/x86/kernel/traps.c:780 do_debug+0x222/0x250
>>>    CPU: 0 PID: 12927 Comm: syz-executor Tainted: G           OE    4.15.0-rc2+ #16
>>>    RIP: 0010:do_debug+0x222/0x250
>>>    Call Trace:
>>>     <#DB>
>>>     debug+0x3e/0x70
>>>    RIP: 0010:copy_user_enhanced_fast_string+0x10/0x20
>>>     </#DB>
>>>     _copy_from_user+0x5b/0x90
>>>     SyS_timer_create+0x33/0x80
>>>     entry_SYSCALL_64_fastpath+0x23/0x9a
>>>
>>> The syzkaller will mmap a buffer which is also the struct sigevent parameter of
>>> timer_create(), it will also call perf_event_open() to set a BP for the buffer,
>>> so when the implementation of timer_create() in kernel tries to get the struct
>>> sigevent parameter by copy_from_user(), rep movsb triggers the BP. The syzkaller
>>> testcase also sets the debug registers for the guest, however, the kvm just
>>> restores host debug registers when we have active breakpoints. I can observe
>>> the dr6 single step bit is set and !hw_breakpoint_active() sporadically by print
>>> when running the testcase heavy multithreading. The do_debug() which is triggered
>>> by rep movsb will splash when (dr6 & DR_STEP && !user_mode(regs)).
>>>
>>> This patch fixes it by restoring host dr6 unconditionally before preempt/irq
>>> enable.
>>>
>>> Reported-by: Dmitry Vyukov <dvyukov@google.com>
>>> Cc: Paolo Bonzini <pbonzini@redhat.com>
>>> Cc: Radim Krčmář <rkrcmar@redhat.com>
>>> Cc: David Hildenbrand <david@redhat.com>
>>> Cc: Dmitry Vyukov <dvyukov@google.com>
>>> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
>>> ---
>>>  arch/x86/kvm/x86.c | 2 ++
>>>  1 file changed, 2 insertions(+)
>>>
>>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>>> index 0c5d55c..a6370fd 100644
>>> --- a/arch/x86/kvm/x86.c
>>> +++ b/arch/x86/kvm/x86.c
>>> @@ -7065,6 +7065,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>>>        */
>>>       if (hw_breakpoint_active())
>>>               hw_breakpoint_restore();
>>> +     else
>>> +             set_debugreg(current->thread.debugreg6, 6);
>>>
>>>       vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>>>
>>>
>>
>> If you haven't seen it, I analyzed this in
>> https://lkml.org/lkml/2017/11/7/638 but nobody would respond for now to
>> my suggestion/question.
> 
> I think it's fine to restore dr6 before preempt/irq enable.

If no breakpoint is active, you should be able to restore it only in
sched_out.

Paolo

Re: [PATCH] KVM: X86: Fix host dr6 miss restore

[Wanpeng Li]

2017-12-12 23:08 GMT+08:00 Paolo Bonzini <pbonzini@redhat.com>:
> On 10/12/2017 01:44, Wanpeng Li wrote:
>> 2017-12-08 20:39 GMT+08:00 David Hildenbrand <david@redhat.com>:
>>> On 08.12.2017 10:12, Wanpeng Li wrote:
>>>> From: Wanpeng Li <wanpeng.li@hotmail.com>
>>>>
>>>> Reported by syzkaller:
>>>>
>>>>    WARNING: CPU: 0 PID: 12927 at arch/x86/kernel/traps.c:780 do_debug+0x222/0x250
>>>>    CPU: 0 PID: 12927 Comm: syz-executor Tainted: G           OE    4.15.0-rc2+ #16
>>>>    RIP: 0010:do_debug+0x222/0x250
>>>>    Call Trace:
>>>>     <#DB>
>>>>     debug+0x3e/0x70
>>>>    RIP: 0010:copy_user_enhanced_fast_string+0x10/0x20
>>>>     </#DB>
>>>>     _copy_from_user+0x5b/0x90
>>>>     SyS_timer_create+0x33/0x80
>>>>     entry_SYSCALL_64_fastpath+0x23/0x9a
>>>>
>>>> The syzkaller will mmap a buffer which is also the struct sigevent parameter of
>>>> timer_create(), it will also call perf_event_open() to set a BP for the buffer,
>>>> so when the implementation of timer_create() in kernel tries to get the struct
>>>> sigevent parameter by copy_from_user(), rep movsb triggers the BP. The syzkaller
>>>> testcase also sets the debug registers for the guest, however, the kvm just
>>>> restores host debug registers when we have active breakpoints. I can observe
>>>> the dr6 single step bit is set and !hw_breakpoint_active() sporadically by print
>>>> when running the testcase heavy multithreading. The do_debug() which is triggered
>>>> by rep movsb will splash when (dr6 & DR_STEP && !user_mode(regs)).
>>>>
>>>> This patch fixes it by restoring host dr6 unconditionally before preempt/irq
>>>> enable.
>>>>
>>>> Reported-by: Dmitry Vyukov <dvyukov@google.com>
>>>> Cc: Paolo Bonzini <pbonzini@redhat.com>
>>>> Cc: Radim Krčmář <rkrcmar@redhat.com>
>>>> Cc: David Hildenbrand <david@redhat.com>
>>>> Cc: Dmitry Vyukov <dvyukov@google.com>
>>>> Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
>>>> ---
>>>>  arch/x86/kvm/x86.c | 2 ++
>>>>  1 file changed, 2 insertions(+)
>>>>
>>>> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>>>> index 0c5d55c..a6370fd 100644
>>>> --- a/arch/x86/kvm/x86.c
>>>> +++ b/arch/x86/kvm/x86.c
>>>> @@ -7065,6 +7065,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
>>>>        */
>>>>       if (hw_breakpoint_active())
>>>>               hw_breakpoint_restore();
>>>> +     else
>>>> +             set_debugreg(current->thread.debugreg6, 6);
>>>>
>>>>       vcpu->arch.last_guest_tsc = kvm_read_l1_tsc(vcpu, rdtsc());
>>>>
>>>>
>>>
>>> If you haven't seen it, I analyzed this in
>>> https://lkml.org/lkml/2017/11/7/638 but nobody would respond for now to
>>> my suggestion/question.
>>
>> I think it's fine to restore dr6 before preempt/irq enable.
>
> If no breakpoint is active, you should be able to restore it only in
> sched_out.

Do it in v2. :)

Regards,
Wanpeng Li