From: Maxim Levitsky <mlevitsk@redhat.com>
To: Jan Kiszka <jan.kiszka@siemens.com>,
Sean Christopherson <seanjc@google.com>
Cc: kvm@vger.kernel.org, Vitaly Kuznetsov <vkuznets@redhat.com>,
linux-kernel@vger.kernel.org,
Thomas Gleixner <tglx@linutronix.de>,
Wanpeng Li <wanpengli@tencent.com>,
Kieran Bingham <kbingham@kernel.org>,
Jessica Yu <jeyu@kernel.org>,
Andrew Morton <akpm@linux-foundation.org>,
"maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)"
<x86@kernel.org>, Joerg Roedel <joro@8bytes.org>,
Jim Mattson <jmattson@google.com>, Borislav Petkov <bp@alien8.de>,
Stefano Garzarella <sgarzare@redhat.com>,
"H. Peter Anvin" <hpa@zytor.com>,
Paolo Bonzini <pbonzini@redhat.com>,
Ingo Molnar <mingo@redhat.com>
Subject: Re: [PATCH 2/3] KVM: x86: guest debug: don't inject interrupts while single stepping
Date: Tue, 16 Mar 2021 12:59:07 +0200 [thread overview]
Message-ID: <bede3450413a7c5e7e55b19a47c8f079edaa55a2.camel@redhat.com> (raw)
In-Reply-To: <1259724f-1bdb-6229-2772-3192f6d17a4a@siemens.com>
On Tue, 2021-03-16 at 10:16 +0100, Jan Kiszka wrote:
> On 16.03.21 00:37, Sean Christopherson wrote:
> > On Tue, Mar 16, 2021, Maxim Levitsky wrote:
> > > This change greatly helps with two issues:
> > >
> > > * Resuming from a breakpoint is much more reliable.
> > >
> > > When resuming execution from a breakpoint, with interrupts enabled, more often
> > > than not, KVM would inject an interrupt and make the CPU jump immediately to
> > > the interrupt handler and eventually return to the breakpoint, to trigger it
> > > again.
> > >
> > > From the user point of view it looks like the CPU never executed a
> > > single instruction and in some cases that can even prevent forward progress,
> > > for example, when the breakpoint is placed by an automated script
> > > (e.g lx-symbols), which does something in response to the breakpoint and then
> > > continues the guest automatically.
> > > If the script execution takes enough time for another interrupt to arrive,
> > > the guest will be stuck on the same breakpoint RIP forever.
> > >
> > > * Normal single stepping is much more predictable, since it won't land the
> > > debugger into an interrupt handler, so it is much more usable.
> > >
> > > (If entry to an interrupt handler is desired, the user can still place a
> > > breakpoint at it and resume the guest, which won't activate this workaround
> > > and let the gdb still stop at the interrupt handler)
> > >
> > > Since this change is only active when guest is debugged, it won't affect
> > > KVM running normal 'production' VMs.
> > >
> > >
> > > Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
> > > Tested-by: Stefano Garzarella <sgarzare@redhat.com>
> > > ---
> > > arch/x86/kvm/x86.c | 6 ++++++
> > > 1 file changed, 6 insertions(+)
> > >
> > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > > index a9d95f90a0487..b75d990fcf12b 100644
> > > --- a/arch/x86/kvm/x86.c
> > > +++ b/arch/x86/kvm/x86.c
> > > @@ -8458,6 +8458,12 @@ static void inject_pending_event(struct kvm_vcpu *vcpu, bool *req_immediate_exit
> > > can_inject = false;
> > > }
> > >
> > > + /*
> > > + * Don't inject interrupts while single stepping to make guest debug easier
> > > + */
> > > + if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
> > > + return;
> >
> > Is this something userspace can deal with? E.g. disable IRQs and/or set NMI
> > blocking at the start of single-stepping, unwind at the end? Deviating this far
> > from architectural behavior will end in tears at some point.
> >
>
> Does this happen to address this suspicious workaround in the kernel?
>
> /*
> * The kernel doesn't use TF single-step outside of:
> *
> * - Kprobes, consumed through kprobe_debug_handler()
> * - KGDB, consumed through notify_debug()
> *
> * So if we get here with DR_STEP set, something is wonky.
> *
> * A known way to trigger this is through QEMU's GDB stub,
> * which leaks #DB into the guest and causes IST recursion.
> */
> if (WARN_ON_ONCE(dr6 & DR_STEP))
> regs->flags &= ~X86_EFLAGS_TF;
>
> (arch/x86/kernel/traps.c, exc_debug_kernel)
>
> I wonder why this got merged while no one fixed QEMU/KVM, for years? Oh,
> yeah, question to myself as well, dancing around broken guest debugging
> for a long time while trying to fix other issues...
To be honest I didn't see that warning even once, but I can imagine KVM
leaking #DB due to bugs in that code. That area historically didn't receive
much attention since it can only be triggered by
KVM_GET/SET_GUEST_DEBUG which isn't used in production.
The only issue that I on the other hand did
see which is mostly gdb fault is that it fails to remove a software breakpoint
when resuming over it, if that breakpoint's python handler messes up
with gdb's symbols, which is what lx-symbols does.
And that despite the fact that lx-symbol doesn't mess with the object
(that is the kernel) where the breakpoint is defined.
Just adding/removing one symbol file is enough to trigger this issue.
Since lx-symbols already works this around when it reloads all symbols,
I extended that workaround to happen also when loading/unloading
only a single symbol file.
Best regards,
Maxim Levitsky
>
> Jan
>
> > > +
> > > /*
> > > * Finally, inject interrupt events. If an event cannot be injected
> > > * due to architectural conditions (e.g. IF=0) a window-open exit
> > > --
> > > 2.26.2
> > >
next prev parent reply other threads:[~2021-03-16 11:00 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-15 22:10 [PATCH 0/3] KVM: my debug patch queue Maxim Levitsky
2021-03-15 22:10 ` [PATCH 1/3] scripts/gdb: rework lx-symbols gdb script Maxim Levitsky
2021-03-16 13:38 ` Jan Kiszka
2021-03-16 14:12 ` Maxim Levitsky
2021-03-15 22:10 ` [PATCH 2/3] KVM: x86: guest debug: don't inject interrupts while single stepping Maxim Levitsky
2021-03-15 23:37 ` Sean Christopherson
2021-03-16 9:16 ` Jan Kiszka
2021-03-16 10:59 ` Maxim Levitsky [this message]
2021-03-16 11:27 ` Jan Kiszka
2021-03-16 12:34 ` Maxim Levitsky
2021-03-16 13:46 ` Jan Kiszka
2021-03-16 14:34 ` Maxim Levitsky
2021-03-16 15:31 ` Jan Kiszka
[not found] ` <e2cd978e357155dbab21a523bb8981973bd10da7.camel@redhat.com>
2021-03-16 15:56 ` Jan Kiszka
2021-03-16 16:50 ` Sean Christopherson
2021-03-16 17:01 ` Jan Kiszka
2021-03-16 17:26 ` Sean Christopherson
2021-03-17 9:20 ` Jan Kiszka
2021-03-16 18:02 ` Maxim Levitsky
2021-03-18 16:02 ` Jan Kiszka
2021-03-18 12:21 ` Jan Kiszka
2021-03-16 10:55 ` Maxim Levitsky
2021-03-15 22:10 ` [PATCH 3/3] KVM: SVM: allow to intercept all exceptions for debug Maxim Levitsky
2021-03-16 8:32 ` Joerg Roedel
2021-03-16 10:51 ` Maxim Levitsky
2021-03-18 9:19 ` Joerg Roedel
2021-03-18 9:24 ` Maxim Levitsky
2021-03-18 15:47 ` Joerg Roedel
2021-03-18 16:35 ` Sean Christopherson
2021-03-18 16:41 ` Maxim Levitsky
2021-03-18 17:22 ` Sean Christopherson
2021-03-16 8:34 ` Borislav Petkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bede3450413a7c5e7e55b19a47c8f079edaa55a2.camel@redhat.com \
--to=mlevitsk@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=bp@alien8.de \
--cc=hpa@zytor.com \
--cc=jan.kiszka@siemens.com \
--cc=jeyu@kernel.org \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=kbingham@kernel.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=pbonzini@redhat.com \
--cc=seanjc@google.com \
--cc=sgarzare@redhat.com \
--cc=tglx@linutronix.de \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).