Debug-Registers in HVM domain destroyed

Debug-Registers in HVM domain destroyed

[Juergen Gross]

Hi,

we've found a problem with debug registers in HVM domains with Xen (we are
running 4.2, but the code in the hypervisor seems to be unchanged in unstable)
on INTEL processors:

Debug registers are restored on vcpu switch only if db7 has any debug events
activated. This leads to problems in the following cases:

- db0-3 are changed by the guest before events are set "active" in db7. In case
   of a vcpu switch between setting db0-3 and db7, db0-3 are lost. BTW: setting
   db7 before db0-3 is no option, as this could trigger debug interrupts due to
   stale db0-3 contents.

- single stepping is used and vcpu switch occurs between the single step trap
   and reading of db6 in the guest. db6 contents (single step indicator) are
   lost in this case.

Any thoughts?


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Debug-Registers in HVM domain destroyed

[Jan Beulich]

>>> On 14.02.14 at 10:33, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
> Debug registers are restored on vcpu switch only if db7 has any debug events
> activated. This leads to problems in the following cases:
> 
> - db0-3 are changed by the guest before events are set "active" in db7. In case
>    of a vcpu switch between setting db0-3 and db7, db0-3 are lost. BTW: setting
>    db7 before db0-3 is no option, as this could trigger debug interrupts due to
>    stale db0-3 contents.
> 
> - single stepping is used and vcpu switch occurs between the single step trap
>    and reading of db6 in the guest. db6 contents (single step indicator) are
>    lost in this case.

Not exactly, at least not looking at how things are supposed to work:
__restore_debug_registers() gets called when
- context switching in (vmx_restore_dr())
- injecting TRAP_debug
- any DRn is being accessed

So when your guest writes DR[0-3], debug registers should get
restored (from their original zero values) and the guest would be
permitted direct access to the hardware registers. Once context
switched out, vmx_save_dr() ought to be saving the values
(irrespective of DR7 contents, only depending upon
v->arch.hvm_vcpu.flag_dr_dirty). During the next context
switch in, they would get restored immediately if DR7 already has
some breakpoint enabled, or again during first DR access if not.

Hence I think that in general this ought to work. Question is
whether one of the more modern feature additions broke any of
this. Assuming that your guest isn't doing heavy accesses to the
debug registers, instrumenting the hypervisor side code to track
the saving/restoring shouldn't be causing too much log output (as
long as you suppress output from the context switch path when
no relevant state changed).

Jan

Re: Debug-Registers in HVM domain destroyed

[Juergen Gross]

On 14.02.2014 11:40, Jan Beulich wrote:
>>>> On 14.02.14 at 10:33, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>> Debug registers are restored on vcpu switch only if db7 has any debug events
>> activated. This leads to problems in the following cases:
>>
>> - db0-3 are changed by the guest before events are set "active" in db7. In case
>>     of a vcpu switch between setting db0-3 and db7, db0-3 are lost. BTW: setting
>>     db7 before db0-3 is no option, as this could trigger debug interrupts due to
>>     stale db0-3 contents.
>>
>> - single stepping is used and vcpu switch occurs between the single step trap
>>     and reading of db6 in the guest. db6 contents (single step indicator) are
>>     lost in this case.
>
> Not exactly, at least not looking at how things are supposed to work:
> __restore_debug_registers() gets called when
> - context switching in (vmx_restore_dr())
> - injecting TRAP_debug

Is this the case when the guest itself uses single stepping? Initially the
debug trap shouldn't cause a VMEXIT, I think. And I'm not sure the hypervisor
will see a guest setting TF via an IRET.

I _have_ seen a debug trap in the guest after single stepping without db6
having the single step indicator set...

> - any DRn is being accessed
>
> So when your guest writes DR[0-3], debug registers should get
> restored (from their original zero values) and the guest would be
> permitted direct access to the hardware registers. Once context
> switched out, vmx_save_dr() ought to be saving the values
> (irrespective of DR7 contents, only depending upon
> v->arch.hvm_vcpu.flag_dr_dirty). During the next context
> switch in, they would get restored immediately if DR7 already has
> some breakpoint enabled, or again during first DR access if not.

Okay, I'll check that. A little test routine in my domU should be able to
verify that debug registers won't change under it's feet in case of no
activated events in db7.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Debug-Registers in HVM domain destroyed

[Jan Beulich]

>>> On 14.02.14 at 13:18, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
> On 14.02.2014 11:40, Jan Beulich wrote:
>>>>> On 14.02.14 at 10:33, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>>> Debug registers are restored on vcpu switch only if db7 has any debug events
>>> activated. This leads to problems in the following cases:
>>>
>>> - db0-3 are changed by the guest before events are set "active" in db7. In 
> case
>>>     of a vcpu switch between setting db0-3 and db7, db0-3 are lost. BTW: 
> setting
>>>     db7 before db0-3 is no option, as this could trigger debug interrupts due 
> to
>>>     stale db0-3 contents.
>>>
>>> - single stepping is used and vcpu switch occurs between the single step trap
>>>     and reading of db6 in the guest. db6 contents (single step indicator) 
> are
>>>     lost in this case.
>>
>> Not exactly, at least not looking at how things are supposed to work:
>> __restore_debug_registers() gets called when
>> - context switching in (vmx_restore_dr())
>> - injecting TRAP_debug
> 
> Is this the case when the guest itself uses single stepping? Initially the
> debug trap shouldn't cause a VMEXIT, I think.

That looks like a bug, indeed - it's missing from the initially set
exception_bitmap. Could you check whether adding this in
construct_vmcs() addresses that part of the issue? (A proper fix
would likely include further adjustments to the setting of this flag,
e.g. clearing it alongside clearing the DR intercept.) But then
again all of this already depends on cpu_has_monitor_trap_flag -
if that's set on your system, maybe you could try suppressing its
detection (by removing CPU_BASED_MONITOR_TRAP_FLAG from
the optional feature set in vmx_init_vmcs_config())?

> And I'm not sure the 
> hypervisor will see a guest setting TF via an IRET.

It shouldn't need to know of this.

Jan

Re: Debug-Registers in HVM domain destroyed

[Juergen Gross]

[-- Attachment #1: Type: text/plain, Size: 2577 bytes --]

On 14.02.2014 14:02, Jan Beulich wrote:
>>>> On 14.02.14 at 13:18, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>> On 14.02.2014 11:40, Jan Beulich wrote:
>>>>>> On 14.02.14 at 10:33, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>>>> Debug registers are restored on vcpu switch only if db7 has any debug events
>>>> activated. This leads to problems in the following cases:
>>>>
>>>> - db0-3 are changed by the guest before events are set "active" in db7. In
>> case
>>>>      of a vcpu switch between setting db0-3 and db7, db0-3 are lost. BTW:
>> setting
>>>>      db7 before db0-3 is no option, as this could trigger debug interrupts due
>> to
>>>>      stale db0-3 contents.
>>>>
>>>> - single stepping is used and vcpu switch occurs between the single step trap
>>>>      and reading of db6 in the guest. db6 contents (single step indicator)
>> are
>>>>      lost in this case.
>>>
>>> Not exactly, at least not looking at how things are supposed to work:
>>> __restore_debug_registers() gets called when
>>> - context switching in (vmx_restore_dr())
>>> - injecting TRAP_debug

Okay, db0-3 seem to be preserved. I did a test modifying the registers without
activating any debug traps. Even under heavy vcpu scheduling load everything
was fine.

>>
>> Is this the case when the guest itself uses single stepping? Initially the
>> debug trap shouldn't cause a VMEXIT, I think.
>
> That looks like a bug, indeed - it's missing from the initially set
> exception_bitmap. Could you check whether adding this in
> construct_vmcs() addresses that part of the issue? (A proper fix
> would likely include further adjustments to the setting of this flag,
> e.g. clearing it alongside clearing the DR intercept.) But then
> again all of this already depends on cpu_has_monitor_trap_flag -
> if that's set on your system, maybe you could try suppressing its
> detection (by removing CPU_BASED_MONITOR_TRAP_FLAG from
> the optional feature set in vmx_init_vmcs_config())?

I've currently a test running with the attached patch (the bug was hit about
once every 3 hours, test is running now for about 4 hours without problem).
Test machine is running with Xen 4.2.3 hypervisor from SLES11 SP3.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

[-- Attachment #2: single-step.patch --]
[-- Type: text/x-patch, Size: 1927 bytes --]

--- xen-4.2.3-testing.orig/xen/include/asm-x86/hvm/hvm.h	2014-02-14 19:05:59.000000000 +0100
+++ xen-4.2.3-testing/xen/include/asm-x86/hvm/hvm.h	2014-02-17 07:43:05.000000000 +0100
@@ -374,7 +374,8 @@ static inline int hvm_do_pmu_interrupt(s
         (cpu_has_xsave ? X86_CR4_OSXSAVE : 0))))
 
 /* These exceptions must always be intercepted. */
-#define HVM_TRAP_MASK ((1U << TRAP_machine_check) | (1U << TRAP_invalid_op))
+#define HVM_TRAP_MASK ((1U << TRAP_machine_check) | (1U << TRAP_invalid_op) |\
+	(1 << TRAP_debug))
 
 /*
  * x86 event types. This enumeration is valid for:
--- xen-4.2.3-testing.orig/xen/arch/x86/hvm/vmx/vmcs.c	2014-02-17 07:48:43.000000000 +0100
+++ xen-4.2.3-testing/xen/arch/x86/hvm/vmx/vmcs.c	2014-02-17 10:16:25.000000000 +0100
@@ -168,7 +168,7 @@ static int vmx_init_vmcs_config(void)
            CPU_BASED_RDTSC_EXITING);
     opt = (CPU_BASED_ACTIVATE_MSR_BITMAP |
            CPU_BASED_TPR_SHADOW |
-           CPU_BASED_MONITOR_TRAP_FLAG |
+           /* CPU_BASED_MONITOR_TRAP_FLAG | */
            CPU_BASED_ACTIVATE_SECONDARY_CONTROLS);
     _vmx_cpu_based_exec_control = adjust_vmx_controls(
         "CPU-Based Exec Control", min, opt,
--- xen-4.2.3-testing.orig/xen/arch/x86/hvm/vmx/vmx.c	2014-02-18 08:04:23.000000000 +0100
+++ xen-4.2.3-testing/xen/arch/x86/hvm/vmx/vmx.c	2014-02-18 10:45:42.000000000 +0100
@@ -2646,7 +2646,11 @@ void vmx_vmexit_handler(struct cpu_user_
             HVMTRACE_1D(TRAP_DEBUG, exit_qualification);
             write_debugreg(6, exit_qualification | 0xffff0ff0);
             if ( !v->domain->debugger_attached || cpu_has_monitor_trap_flag )
-                goto exit_and_crash;
+            {
+                __restore_debug_registers(v);
+                hvm_inject_hw_exception(TRAP_debug, HVM_DELIVER_NO_ERROR_CODE);
+                break;
+            }
             domain_pause_for_debugger();
             break;
         case TRAP_int3: 

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Debug-Registers in HVM domain destroyed

[Jan Beulich]

>>> On 18.02.14 at 13:48, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
> On 14.02.2014 14:02, Jan Beulich wrote:
>>>>> On 14.02.14 at 13:18, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>>> Is this the case when the guest itself uses single stepping? Initially the
>>> debug trap shouldn't cause a VMEXIT, I think.
>>
>> That looks like a bug, indeed - it's missing from the initially set
>> exception_bitmap. Could you check whether adding this in
>> construct_vmcs() addresses that part of the issue? (A proper fix
>> would likely include further adjustments to the setting of this flag,
>> e.g. clearing it alongside clearing the DR intercept.) But then
>> again all of this already depends on cpu_has_monitor_trap_flag -
>> if that's set on your system, maybe you could try suppressing its
>> detection (by removing CPU_BASED_MONITOR_TRAP_FLAG from
>> the optional feature set in vmx_init_vmcs_config())?
> 
> I've currently a test running with the attached patch (the bug was hit about
> once every 3 hours, test is running now for about 4 hours without problem).
> Test machine is running with Xen 4.2.3 hypervisor from SLES11 SP3.

Which, if it continues running fine, would confirm the theory.
I'd like to defer to the VMX folks though for putting together a
proper fix then - I'd likely overlook some corner case.

Jan

Re: Debug-Registers in HVM domain destroyed

[Juergen Gross]

On 18.02.2014 13:56, Jan Beulich wrote:
>>>> On 18.02.14 at 13:48, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>> On 14.02.2014 14:02, Jan Beulich wrote:
>>>>>> On 14.02.14 at 13:18, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>>>> Is this the case when the guest itself uses single stepping? Initially the
>>>> debug trap shouldn't cause a VMEXIT, I think.
>>>
>>> That looks like a bug, indeed - it's missing from the initially set
>>> exception_bitmap. Could you check whether adding this in
>>> construct_vmcs() addresses that part of the issue? (A proper fix
>>> would likely include further adjustments to the setting of this flag,
>>> e.g. clearing it alongside clearing the DR intercept.) But then
>>> again all of this already depends on cpu_has_monitor_trap_flag -
>>> if that's set on your system, maybe you could try suppressing its
>>> detection (by removing CPU_BASED_MONITOR_TRAP_FLAG from
>>> the optional feature set in vmx_init_vmcs_config())?
>>
>> I've currently a test running with the attached patch (the bug was hit about
>> once every 3 hours, test is running now for about 4 hours without problem).
>> Test machine is running with Xen 4.2.3 hypervisor from SLES11 SP3.
>
> Which, if it continues running fine, would confirm the theory.
> I'd like to defer to the VMX folks though for putting together a
> proper fix then - I'd likely overlook some corner case.

Okay, theory confirmed.

Unless you want to do it, I'll start another thread with the info found so far
and include Jun Nakajima and Eddie Dong.


Juergen

-- 
Juergen Gross                 Principal Developer Operating Systems
PBG PDG ES&S SWE OS6                   Telephone: +49 (0) 89 62060 2932
Fujitsu                                   e-mail: juergen.gross@ts.fujitsu.com
Mies-van-der-Rohe-Str. 8                Internet: ts.fujitsu.com
D-80807 Muenchen                 Company details: ts.fujitsu.com/imprint.html

Re: Debug-Registers in HVM domain destroyed

[Jan Beulich]

>>> On 20.02.14 at 08:44, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
> On 18.02.2014 13:56, Jan Beulich wrote:
>>>>> On 18.02.14 at 13:48, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>>> On 14.02.2014 14:02, Jan Beulich wrote:
>>>>>>> On 14.02.14 at 13:18, Juergen Gross <juergen.gross@ts.fujitsu.com> wrote:
>>>>> Is this the case when the guest itself uses single stepping? Initially the
>>>>> debug trap shouldn't cause a VMEXIT, I think.
>>>>
>>>> That looks like a bug, indeed - it's missing from the initially set
>>>> exception_bitmap. Could you check whether adding this in
>>>> construct_vmcs() addresses that part of the issue? (A proper fix
>>>> would likely include further adjustments to the setting of this flag,
>>>> e.g. clearing it alongside clearing the DR intercept.) But then
>>>> again all of this already depends on cpu_has_monitor_trap_flag -
>>>> if that's set on your system, maybe you could try suppressing its
>>>> detection (by removing CPU_BASED_MONITOR_TRAP_FLAG from
>>>> the optional feature set in vmx_init_vmcs_config())?
>>>
>>> I've currently a test running with the attached patch (the bug was hit about
>>> once every 3 hours, test is running now for about 4 hours without problem).
>>> Test machine is running with Xen 4.2.3 hypervisor from SLES11 SP3.
>>
>> Which, if it continues running fine, would confirm the theory.
>> I'd like to defer to the VMX folks though for putting together a
>> proper fix then - I'd likely overlook some corner case.
> 
> Okay, theory confirmed.
> 
> Unless you want to do it, I'll start another thread with the info found so 
> far and include Jun Nakajima and Eddie Dong.

Please do. And perhaps also include Yang Z Zhang.

Jan