* [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
@ 2013-03-16 20:49 Jan Kiszka
2013-03-17 8:47 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Jan Kiszka @ 2013-03-16 20:49 UTC (permalink / raw)
To: Gleb Natapov, Marcelo Tosatti; +Cc: kvm
From: Jan Kiszka <jan.kiszka@siemens.com>
If the guest didn't take the last APIC timer interrupt yet and generates
another one on top, e.g. via periodic mode, we do not block the VCPU
even if the guest state is halted. The reason is that
apic_has_pending_timer continues to return a non-zero value.
Fix this busy loop by taking the IRR content for the LVT vector in
apic_has_pending_timer into account.
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
Not a critical issue, we are looping fully interruptible, but it's ugly
to do so IMHO.
arch/x86/kvm/lapic.c | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index a8e9369..658abf5 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
struct kvm_lapic *apic = vcpu->arch.apic;
if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
- apic_lvt_enabled(apic, APIC_LVTT))
+ apic_lvt_enabled(apic, APIC_LVTT) &&
+ !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
+ apic->regs + APIC_IRR))
return atomic_read(&apic->lapic_timer.pending);
return 0;
--
1.7.3.4
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-16 20:49 [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers Jan Kiszka
@ 2013-03-17 8:47 ` Gleb Natapov
2013-03-17 10:45 ` Jan Kiszka
2013-04-28 10:15 ` Jan Kiszka
0 siblings, 2 replies; 26+ messages in thread
From: Gleb Natapov @ 2013-03-17 8:47 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Marcelo Tosatti, kvm
On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
>
> If the guest didn't take the last APIC timer interrupt yet and generates
> another one on top, e.g. via periodic mode, we do not block the VCPU
> even if the guest state is halted. The reason is that
> apic_has_pending_timer continues to return a non-zero value.
>
> Fix this busy loop by taking the IRR content for the LVT vector in
> apic_has_pending_timer into account.
>
Just drop coalescing tacking for lapic interrupt. After posted interrupt
will be merged __apic_accept_irq() will not longer return coalescing
information, so the code will be dead anyway.
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
>
> Not a critical issue, we are looping fully interruptible, but it's ugly
> to do so IMHO.
>
> arch/x86/kvm/lapic.c | 4 +++-
> 1 files changed, 3 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> index a8e9369..658abf5 100644
> --- a/arch/x86/kvm/lapic.c
> +++ b/arch/x86/kvm/lapic.c
> @@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
> struct kvm_lapic *apic = vcpu->arch.apic;
>
> if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
> - apic_lvt_enabled(apic, APIC_LVTT))
> + apic_lvt_enabled(apic, APIC_LVTT) &&
> + !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
> + apic->regs + APIC_IRR))
> return atomic_read(&apic->lapic_timer.pending);
>
> return 0;
> --
> 1.7.3.4
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-17 8:47 ` Gleb Natapov
@ 2013-03-17 10:45 ` Jan Kiszka
2013-03-17 10:47 ` Gleb Natapov
2013-04-28 10:15 ` Jan Kiszka
1 sibling, 1 reply; 26+ messages in thread
From: Jan Kiszka @ 2013-03-17 10:45 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Marcelo Tosatti, kvm
[-- Attachment #1: Type: text/plain, Size: 881 bytes --]
On 2013-03-17 09:47, Gleb Natapov wrote:
> On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> If the guest didn't take the last APIC timer interrupt yet and generates
>> another one on top, e.g. via periodic mode, we do not block the VCPU
>> even if the guest state is halted. The reason is that
>> apic_has_pending_timer continues to return a non-zero value.
>>
>> Fix this busy loop by taking the IRR content for the LVT vector in
>> apic_has_pending_timer into account.
>>
> Just drop coalescing tacking for lapic interrupt. After posted interrupt
> will be merged __apic_accept_irq() will not longer return coalescing
> information, so the code will be dead anyway.
That requires the RTC decoalescing series to go first to avoid a
regression, no? Then let's postpone this topic for now.
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-17 10:45 ` Jan Kiszka
@ 2013-03-17 10:47 ` Gleb Natapov
2013-03-20 19:30 ` Marcelo Tosatti
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-17 10:47 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Marcelo Tosatti, kvm
On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> On 2013-03-17 09:47, Gleb Natapov wrote:
> > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> If the guest didn't take the last APIC timer interrupt yet and generates
> >> another one on top, e.g. via periodic mode, we do not block the VCPU
> >> even if the guest state is halted. The reason is that
> >> apic_has_pending_timer continues to return a non-zero value.
> >>
> >> Fix this busy loop by taking the IRR content for the LVT vector in
> >> apic_has_pending_timer into account.
> >>
> > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > will be merged __apic_accept_irq() will not longer return coalescing
> > information, so the code will be dead anyway.
>
> That requires the RTC decoalescing series to go first to avoid a
> regression, no? Then let's postpone this topic for now.
>
Yes, but decoalescing will work only for RTC :(
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-17 10:47 ` Gleb Natapov
@ 2013-03-20 19:30 ` Marcelo Tosatti
2013-03-20 20:03 ` Marcelo Tosatti
0 siblings, 1 reply; 26+ messages in thread
From: Marcelo Tosatti @ 2013-03-20 19:30 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Jan Kiszka, kvm
On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> > On 2013-03-17 09:47, Gleb Natapov wrote:
> > > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> > >> From: Jan Kiszka <jan.kiszka@siemens.com>
> > >>
> > >> If the guest didn't take the last APIC timer interrupt yet and generates
> > >> another one on top, e.g. via periodic mode, we do not block the VCPU
> > >> even if the guest state is halted. The reason is that
> > >> apic_has_pending_timer continues to return a non-zero value.
> > >>
> > >> Fix this busy loop by taking the IRR content for the LVT vector in
> > >> apic_has_pending_timer into account.
> > >>
> > > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > > will be merged __apic_accept_irq() will not longer return coalescing
> > > information, so the code will be dead anyway.
> >
> > That requires the RTC decoalescing series to go first to avoid a
> > regression, no? Then let's postpone this topic for now.
> >
> Yes, but decoalescing will work only for RTC :(
Are you proposing to drop LAPIC interrupt reinjection?
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-20 19:30 ` Marcelo Tosatti
@ 2013-03-20 20:03 ` Marcelo Tosatti
2013-03-20 21:32 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Marcelo Tosatti @ 2013-03-20 20:03 UTC (permalink / raw)
To: Gleb Natapov, Yang Zhang; +Cc: Jan Kiszka, kvm
On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
> On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> > On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> > > On 2013-03-17 09:47, Gleb Natapov wrote:
> > > > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> > > >> From: Jan Kiszka <jan.kiszka@siemens.com>
> > > >>
> > > >> If the guest didn't take the last APIC timer interrupt yet and generates
> > > >> another one on top, e.g. via periodic mode, we do not block the VCPU
> > > >> even if the guest state is halted. The reason is that
> > > >> apic_has_pending_timer continues to return a non-zero value.
> > > >>
> > > >> Fix this busy loop by taking the IRR content for the LVT vector in
> > > >> apic_has_pending_timer into account.
> > > >>
> > > > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > > > will be merged __apic_accept_irq() will not longer return coalescing
> > > > information, so the code will be dead anyway.
> > >
> > > That requires the RTC decoalescing series to go first to avoid a
> > > regression, no? Then let's postpone this topic for now.
> > >
> > Yes, but decoalescing will work only for RTC :(
>
> Are you proposing to drop LAPIC interrupt reinjection?
Since timer handling and injection is VCPU-local for LAPIC,
__apic_accept_irq can (and must) return coalesced information (cannot
drop LAPIC interrupt reinjection).
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-20 20:03 ` Marcelo Tosatti
@ 2013-03-20 21:32 ` Gleb Natapov
2013-03-20 23:19 ` Marcelo Tosatti
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-20 21:32 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Yang Zhang, Jan Kiszka, kvm
On Wed, Mar 20, 2013 at 05:03:19PM -0300, Marcelo Tosatti wrote:
> On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
> > On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> > > On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> > > > On 2013-03-17 09:47, Gleb Natapov wrote:
> > > > > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> > > > >> From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > >>
> > > > >> If the guest didn't take the last APIC timer interrupt yet and generates
> > > > >> another one on top, e.g. via periodic mode, we do not block the VCPU
> > > > >> even if the guest state is halted. The reason is that
> > > > >> apic_has_pending_timer continues to return a non-zero value.
> > > > >>
> > > > >> Fix this busy loop by taking the IRR content for the LVT vector in
> > > > >> apic_has_pending_timer into account.
> > > > >>
> > > > > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > > > > will be merged __apic_accept_irq() will not longer return coalescing
> > > > > information, so the code will be dead anyway.
> > > >
> > > > That requires the RTC decoalescing series to go first to avoid a
> > > > regression, no? Then let's postpone this topic for now.
> > > >
> > > Yes, but decoalescing will work only for RTC :(
> >
> > Are you proposing to drop LAPIC interrupt reinjection?
>
> Since timer handling and injection is VCPU-local for LAPIC,
> __apic_accept_irq can (and must) return coalesced information (cannot
> drop LAPIC interrupt reinjection).
>
Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt
patches do not properly check for interrupt coalescing even for
VCPU-local injection.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-20 21:32 ` Gleb Natapov
@ 2013-03-20 23:19 ` Marcelo Tosatti
2013-03-21 4:54 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Marcelo Tosatti @ 2013-03-20 23:19 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Yang Zhang, Jan Kiszka, kvm
On Wed, Mar 20, 2013 at 11:32:38PM +0200, Gleb Natapov wrote:
> On Wed, Mar 20, 2013 at 05:03:19PM -0300, Marcelo Tosatti wrote:
> > On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
> > > On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> > > > On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> > > > > On 2013-03-17 09:47, Gleb Natapov wrote:
> > > > > > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> > > > > >> From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > >>
> > > > > >> If the guest didn't take the last APIC timer interrupt yet and generates
> > > > > >> another one on top, e.g. via periodic mode, we do not block the VCPU
> > > > > >> even if the guest state is halted. The reason is that
> > > > > >> apic_has_pending_timer continues to return a non-zero value.
> > > > > >>
> > > > > >> Fix this busy loop by taking the IRR content for the LVT vector in
> > > > > >> apic_has_pending_timer into account.
> > > > > >>
> > > > > > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > > > > > will be merged __apic_accept_irq() will not longer return coalescing
> > > > > > information, so the code will be dead anyway.
> > > > >
> > > > > That requires the RTC decoalescing series to go first to avoid a
> > > > > regression, no? Then let's postpone this topic for now.
> > > > >
> > > > Yes, but decoalescing will work only for RTC :(
> > >
> > > Are you proposing to drop LAPIC interrupt reinjection?
> >
> > Since timer handling and injection is VCPU-local for LAPIC,
> > __apic_accept_irq can (and must) return coalesced information (cannot
> > drop LAPIC interrupt reinjection).
> >
> Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt
> patches do not properly check for interrupt coalescing even for
> VCPU-local injection.
>
> --
> Gleb.
Because older Linux guests depend on reinjection for proper timekeeping.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-20 23:19 ` Marcelo Tosatti
@ 2013-03-21 4:54 ` Gleb Natapov
2013-03-21 14:02 ` Marcelo Tosatti
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-21 4:54 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Yang Zhang, Jan Kiszka, kvm
On Wed, Mar 20, 2013 at 08:19:13PM -0300, Marcelo Tosatti wrote:
> On Wed, Mar 20, 2013 at 11:32:38PM +0200, Gleb Natapov wrote:
> > On Wed, Mar 20, 2013 at 05:03:19PM -0300, Marcelo Tosatti wrote:
> > > On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
> > > > On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> > > > > On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> > > > > > On 2013-03-17 09:47, Gleb Natapov wrote:
> > > > > > > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> > > > > > >> From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > > >>
> > > > > > >> If the guest didn't take the last APIC timer interrupt yet and generates
> > > > > > >> another one on top, e.g. via periodic mode, we do not block the VCPU
> > > > > > >> even if the guest state is halted. The reason is that
> > > > > > >> apic_has_pending_timer continues to return a non-zero value.
> > > > > > >>
> > > > > > >> Fix this busy loop by taking the IRR content for the LVT vector in
> > > > > > >> apic_has_pending_timer into account.
> > > > > > >>
> > > > > > > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > > > > > > will be merged __apic_accept_irq() will not longer return coalescing
> > > > > > > information, so the code will be dead anyway.
> > > > > >
> > > > > > That requires the RTC decoalescing series to go first to avoid a
> > > > > > regression, no? Then let's postpone this topic for now.
> > > > > >
> > > > > Yes, but decoalescing will work only for RTC :(
> > > >
> > > > Are you proposing to drop LAPIC interrupt reinjection?
> > >
> > > Since timer handling and injection is VCPU-local for LAPIC,
> > > __apic_accept_irq can (and must) return coalesced information (cannot
> > > drop LAPIC interrupt reinjection).
> > >
> > Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt
> > patches do not properly check for interrupt coalescing even for
> > VCPU-local injection.
> >
> > --
> > Gleb.
>
> Because older Linux guests depend on reinjection for proper timekeeping.
Which versions? Those without kvmclock? Can we make them use PIT instead?
Posted interrupts going to break them.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 4:54 ` Gleb Natapov
@ 2013-03-21 14:02 ` Marcelo Tosatti
2013-03-21 14:18 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Marcelo Tosatti @ 2013-03-21 14:02 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Yang Zhang, Jan Kiszka, kvm
On Thu, Mar 21, 2013 at 06:54:46AM +0200, Gleb Natapov wrote:
> On Wed, Mar 20, 2013 at 08:19:13PM -0300, Marcelo Tosatti wrote:
> > On Wed, Mar 20, 2013 at 11:32:38PM +0200, Gleb Natapov wrote:
> > > On Wed, Mar 20, 2013 at 05:03:19PM -0300, Marcelo Tosatti wrote:
> > > > On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
> > > > > On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> > > > > > On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> > > > > > > On 2013-03-17 09:47, Gleb Natapov wrote:
> > > > > > > > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> > > > > > > >> From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > > > >>
> > > > > > > >> If the guest didn't take the last APIC timer interrupt yet and generates
> > > > > > > >> another one on top, e.g. via periodic mode, we do not block the VCPU
> > > > > > > >> even if the guest state is halted. The reason is that
> > > > > > > >> apic_has_pending_timer continues to return a non-zero value.
> > > > > > > >>
> > > > > > > >> Fix this busy loop by taking the IRR content for the LVT vector in
> > > > > > > >> apic_has_pending_timer into account.
> > > > > > > >>
> > > > > > > > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > > > > > > > will be merged __apic_accept_irq() will not longer return coalescing
> > > > > > > > information, so the code will be dead anyway.
> > > > > > >
> > > > > > > That requires the RTC decoalescing series to go first to avoid a
> > > > > > > regression, no? Then let's postpone this topic for now.
> > > > > > >
> > > > > > Yes, but decoalescing will work only for RTC :(
> > > > >
> > > > > Are you proposing to drop LAPIC interrupt reinjection?
> > > >
> > > > Since timer handling and injection is VCPU-local for LAPIC,
> > > > __apic_accept_irq can (and must) return coalesced information (cannot
> > > > drop LAPIC interrupt reinjection).
> > > >
> > > Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt
> > > patches do not properly check for interrupt coalescing even for
> > > VCPU-local injection.
> > >
> > > --
> > > Gleb.
> >
> > Because older Linux guests depend on reinjection for proper timekeeping.
> Which versions? Those without kvmclock? Can we make them use PIT instead?
> Posted interrupts going to break them.
There is no reason to break them if its OK to receive reinjection info
from LAPIC... its a matter of returning the information from
apic_accept_irq, no big deal.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 14:02 ` Marcelo Tosatti
@ 2013-03-21 14:18 ` Gleb Natapov
2013-03-21 14:27 ` Zhang, Yang Z
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-21 14:18 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Yang Zhang, Jan Kiszka, kvm
On Thu, Mar 21, 2013 at 11:02:24AM -0300, Marcelo Tosatti wrote:
> On Thu, Mar 21, 2013 at 06:54:46AM +0200, Gleb Natapov wrote:
> > On Wed, Mar 20, 2013 at 08:19:13PM -0300, Marcelo Tosatti wrote:
> > > On Wed, Mar 20, 2013 at 11:32:38PM +0200, Gleb Natapov wrote:
> > > > On Wed, Mar 20, 2013 at 05:03:19PM -0300, Marcelo Tosatti wrote:
> > > > > On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
> > > > > > On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> > > > > > > On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> > > > > > > > On 2013-03-17 09:47, Gleb Natapov wrote:
> > > > > > > > > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> > > > > > > > >> From: Jan Kiszka <jan.kiszka@siemens.com>
> > > > > > > > >>
> > > > > > > > >> If the guest didn't take the last APIC timer interrupt yet and generates
> > > > > > > > >> another one on top, e.g. via periodic mode, we do not block the VCPU
> > > > > > > > >> even if the guest state is halted. The reason is that
> > > > > > > > >> apic_has_pending_timer continues to return a non-zero value.
> > > > > > > > >>
> > > > > > > > >> Fix this busy loop by taking the IRR content for the LVT vector in
> > > > > > > > >> apic_has_pending_timer into account.
> > > > > > > > >>
> > > > > > > > > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > > > > > > > > will be merged __apic_accept_irq() will not longer return coalescing
> > > > > > > > > information, so the code will be dead anyway.
> > > > > > > >
> > > > > > > > That requires the RTC decoalescing series to go first to avoid a
> > > > > > > > regression, no? Then let's postpone this topic for now.
> > > > > > > >
> > > > > > > Yes, but decoalescing will work only for RTC :(
> > > > > >
> > > > > > Are you proposing to drop LAPIC interrupt reinjection?
> > > > >
> > > > > Since timer handling and injection is VCPU-local for LAPIC,
> > > > > __apic_accept_irq can (and must) return coalesced information (cannot
> > > > > drop LAPIC interrupt reinjection).
> > > > >
> > > > Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt
> > > > patches do not properly check for interrupt coalescing even for
> > > > VCPU-local injection.
> > > >
> > > > --
> > > > Gleb.
> > >
> > > Because older Linux guests depend on reinjection for proper timekeeping.
> > Which versions? Those without kvmclock? Can we make them use PIT instead?
> > Posted interrupts going to break them.
>
> There is no reason to break them if its OK to receive reinjection info
> from LAPIC... its a matter of returning the information from
> apic_accept_irq, no big deal.
>
But current PI patches do break them, thats my point. So we either
need to revise them again, or drop LAPIC timer reinjection. Making
apic_accept_irq semantics "it returns coalescing info, but only sometimes"
is dubious though.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 14:18 ` Gleb Natapov
@ 2013-03-21 14:27 ` Zhang, Yang Z
2013-03-21 16:27 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Zhang, Yang Z @ 2013-03-21 14:27 UTC (permalink / raw)
To: Gleb Natapov, Marcelo Tosatti; +Cc: Jan Kiszka, kvm
Gleb Natapov wrote on 2013-03-21:
> On Thu, Mar 21, 2013 at 11:02:24AM -0300, Marcelo Tosatti wrote:
>> On Thu, Mar 21, 2013 at 06:54:46AM +0200, Gleb Natapov wrote:
>>> On Wed, Mar 20, 2013 at 08:19:13PM -0300, Marcelo Tosatti wrote:
>>>> On Wed, Mar 20, 2013 at 11:32:38PM +0200, Gleb Natapov wrote:
>>>>> On Wed, Mar 20, 2013 at 05:03:19PM -0300, Marcelo Tosatti wrote:
>>>>>> On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
>>>>>>> On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
>>>>>>>> On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
>>>>>>>>> On 2013-03-17 09:47, Gleb Natapov wrote:
>>>>>>>>>> On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
>>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>>>>>
>>>>>>>>>>> If the guest didn't take the last APIC timer interrupt yet and
>>>>>>>>>>> generates another one on top, e.g. via periodic mode, we do
>>>>>>>>>>> not block the VCPU even if the guest state is halted. The
>>>>>>>>>>> reason is that apic_has_pending_timer continues to return a
>>>>>>>>>>> non-zero value.
>>>>>>>>>>>
>>>>>>>>>>> Fix this busy loop by taking the IRR content for the LVT vector in
>>>>>>>>>>> apic_has_pending_timer into account.
>>>>>>>>>>>
>>>>>>>>>> Just drop coalescing tacking for lapic interrupt. After posted
>>>>>>>>>> interrupt will be merged __apic_accept_irq() will not longer
>>>>>>>>>> return coalescing information, so the code will be dead anyway.
>>>>>>>>>
>>>>>>>>> That requires the RTC decoalescing series to go first to avoid a
>>>>>>>>> regression, no? Then let's postpone this topic for now.
>>>>>>>>>
>>>>>>>> Yes, but decoalescing will work only for RTC :(
>>>>>>>
>>>>>>> Are you proposing to drop LAPIC interrupt reinjection?
>>>>>>
>>>>>> Since timer handling and injection is VCPU-local for LAPIC,
>>>>>> __apic_accept_irq can (and must) return coalesced information (cannot
>>>>>> drop LAPIC interrupt reinjection).
>>>>>>
>>>>> Why can't we drop LAPIC interrupt reinjection? Proposed posted
>>>>> interrupt patches do not properly check for interrupt coalescing
>>>>> even for VCPU-local injection.
>>>>>
>>>>> --
>>>>> Gleb.
>>>>
>>>> Because older Linux guests depend on reinjection for proper timekeeping.
>>> Which versions? Those without kvmclock? Can we make them use PIT
>>> instead? Posted interrupts going to break them.
>>
>> There is no reason to break them if its OK to receive reinjection info
>> from LAPIC... its a matter of returning the information from
>> apic_accept_irq, no big deal.
>>
> But current PI patches do break them, thats my point. So we either
> need to revise them again, or drop LAPIC timer reinjection. Making
> apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> is dubious though.
We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
Best regards,
Yang
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 14:27 ` Zhang, Yang Z
@ 2013-03-21 16:27 ` Gleb Natapov
2013-03-21 20:51 ` Marcelo Tosatti
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-21 16:27 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: Marcelo Tosatti, Jan Kiszka, kvm
On Thu, Mar 21, 2013 at 02:27:22PM +0000, Zhang, Yang Z wrote:
> Gleb Natapov wrote on 2013-03-21:
> > On Thu, Mar 21, 2013 at 11:02:24AM -0300, Marcelo Tosatti wrote:
> >> On Thu, Mar 21, 2013 at 06:54:46AM +0200, Gleb Natapov wrote:
> >>> On Wed, Mar 20, 2013 at 08:19:13PM -0300, Marcelo Tosatti wrote:
> >>>> On Wed, Mar 20, 2013 at 11:32:38PM +0200, Gleb Natapov wrote:
> >>>>> On Wed, Mar 20, 2013 at 05:03:19PM -0300, Marcelo Tosatti wrote:
> >>>>>> On Wed, Mar 20, 2013 at 04:30:33PM -0300, Marcelo Tosatti wrote:
> >>>>>>> On Sun, Mar 17, 2013 at 12:47:17PM +0200, Gleb Natapov wrote:
> >>>>>>>> On Sun, Mar 17, 2013 at 11:45:34AM +0100, Jan Kiszka wrote:
> >>>>>>>>> On 2013-03-17 09:47, Gleb Natapov wrote:
> >>>>>>>>>> On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> >>>>>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>>>>>>>>>>
> >>>>>>>>>>> If the guest didn't take the last APIC timer interrupt yet and
> >>>>>>>>>>> generates another one on top, e.g. via periodic mode, we do
> >>>>>>>>>>> not block the VCPU even if the guest state is halted. The
> >>>>>>>>>>> reason is that apic_has_pending_timer continues to return a
> >>>>>>>>>>> non-zero value.
> >>>>>>>>>>>
> >>>>>>>>>>> Fix this busy loop by taking the IRR content for the LVT vector in
> >>>>>>>>>>> apic_has_pending_timer into account.
> >>>>>>>>>>>
> >>>>>>>>>> Just drop coalescing tacking for lapic interrupt. After posted
> >>>>>>>>>> interrupt will be merged __apic_accept_irq() will not longer
> >>>>>>>>>> return coalescing information, so the code will be dead anyway.
> >>>>>>>>>
> >>>>>>>>> That requires the RTC decoalescing series to go first to avoid a
> >>>>>>>>> regression, no? Then let's postpone this topic for now.
> >>>>>>>>>
> >>>>>>>> Yes, but decoalescing will work only for RTC :(
> >>>>>>>
> >>>>>>> Are you proposing to drop LAPIC interrupt reinjection?
> >>>>>>
> >>>>>> Since timer handling and injection is VCPU-local for LAPIC,
> >>>>>> __apic_accept_irq can (and must) return coalesced information (cannot
> >>>>>> drop LAPIC interrupt reinjection).
> >>>>>>
> >>>>> Why can't we drop LAPIC interrupt reinjection? Proposed posted
> >>>>> interrupt patches do not properly check for interrupt coalescing
> >>>>> even for VCPU-local injection.
> >>>>>
> >>>>> --
> >>>>> Gleb.
> >>>>
> >>>> Because older Linux guests depend on reinjection for proper timekeeping.
> >>> Which versions? Those without kvmclock? Can we make them use PIT
> >>> instead? Posted interrupts going to break them.
> >>
> >> There is no reason to break them if its OK to receive reinjection info
> >> from LAPIC... its a matter of returning the information from
> >> apic_accept_irq, no big deal.
> >>
> > But current PI patches do break them, thats my point. So we either
> > need to revise them again, or drop LAPIC timer reinjection. Making
> > apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> > is dubious though.
> We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
> Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
>
We cannot touch irr while vcpu is in non-root operation, so we will have
to pass flag to apic_accept_irq() to let it know that it is called
synchronously. While all this is possible I want to know which guests
exactly will we break if we will not track interrupt coalescing for
lapic timer. If only 2.0 smp kernels will break we can probably drop it.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 16:27 ` Gleb Natapov
@ 2013-03-21 20:51 ` Marcelo Tosatti
2013-03-21 21:13 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Marcelo Tosatti @ 2013-03-21 20:51 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Zhang, Yang Z, Jan Kiszka, kvm
> > > But current PI patches do break them, thats my point. So we either
> > > need to revise them again, or drop LAPIC timer reinjection. Making
> > > apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> > > is dubious though.
> > We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
> > Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
> >
> We cannot touch irr while vcpu is in non-root operation, so we will have
> to pass flag to apic_accept_irq() to let it know that it is called
> synchronously. While all this is possible I want to know which guests
> exactly will we break if we will not track interrupt coalescing for
> lapic timer. If only 2.0 smp kernels will break we can probably drop it.
RHEL4 / RHEL5 guests.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 20:51 ` Marcelo Tosatti
@ 2013-03-21 21:13 ` Gleb Natapov
2013-03-21 23:06 ` Marcelo Tosatti
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-21 21:13 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Zhang, Yang Z, Jan Kiszka, kvm
On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
> > > > But current PI patches do break them, thats my point. So we either
> > > > need to revise them again, or drop LAPIC timer reinjection. Making
> > > > apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> > > > is dubious though.
> > > We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
> > > Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
> > >
> > We cannot touch irr while vcpu is in non-root operation, so we will have
> > to pass flag to apic_accept_irq() to let it know that it is called
> > synchronously. While all this is possible I want to know which guests
> > exactly will we break if we will not track interrupt coalescing for
> > lapic timer. If only 2.0 smp kernels will break we can probably drop it.
>
> RHEL4 / RHEL5 guests.
RHEL5 has kvmclock no? We should not break RHEL4 though.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 21:13 ` Gleb Natapov
@ 2013-03-21 23:06 ` Marcelo Tosatti
2013-03-22 1:50 ` Zhang, Yang Z
2013-03-22 6:53 ` Gleb Natapov
0 siblings, 2 replies; 26+ messages in thread
From: Marcelo Tosatti @ 2013-03-21 23:06 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Zhang, Yang Z, Jan Kiszka, kvm
On Thu, Mar 21, 2013 at 11:13:39PM +0200, Gleb Natapov wrote:
> On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
> > > > > But current PI patches do break them, thats my point. So we either
> > > > > need to revise them again, or drop LAPIC timer reinjection. Making
> > > > > apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> > > > > is dubious though.
> > > > We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
> > > > Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
> > > >
> > > We cannot touch irr while vcpu is in non-root operation, so we will have
> > > to pass flag to apic_accept_irq() to let it know that it is called
> > > synchronously. While all this is possible I want to know which guests
> > > exactly will we break if we will not track interrupt coalescing for
> > > lapic timer. If only 2.0 smp kernels will break we can probably drop it.
> >
> > RHEL4 / RHEL5 guests.
> RHEL5 has kvmclock no? We should not break RHEL4 though.
kvmclock provides no timer interrupt... either LAPIC or PIT must be used
with kvmclock.
^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 23:06 ` Marcelo Tosatti
@ 2013-03-22 1:50 ` Zhang, Yang Z
2013-03-22 6:53 ` Gleb Natapov
1 sibling, 0 replies; 26+ messages in thread
From: Zhang, Yang Z @ 2013-03-22 1:50 UTC (permalink / raw)
To: Marcelo Tosatti, Gleb Natapov; +Cc: Jan Kiszka, kvm
Marcelo Tosatti wrote on 2013-03-22:
> On Thu, Mar 21, 2013 at 11:13:39PM +0200, Gleb Natapov wrote:
>> On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
>>>>>> But current PI patches do break them, thats my point. So we either
>>>>>> need to revise them again, or drop LAPIC timer reinjection. Making
>>>>>> apic_accept_irq semantics "it returns coalescing info, but only
>>>>>> sometimes" is dubious though.
>>>>> We may rollback to the initial idea: test both irr and pir to get coalescing
> info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq()
> will return right coalescing info.
>>>>> Also, we need to add comments to tell caller, apic_accept_irq() can
>>>>> ensure the return value is correct only when caller is in target
>>>>> vcpu context.
>>>>>
>>>> We cannot touch irr while vcpu is in non-root operation, so we will have
>>>> to pass flag to apic_accept_irq() to let it know that it is called
>>>> synchronously. While all this is possible I want to know which guests
>>>> exactly will we break if we will not track interrupt coalescing for
>>>> lapic timer. If only 2.0 smp kernels will break we can probably drop it.
>>>
>>> RHEL4 / RHEL5 guests.
>> RHEL5 has kvmclock no? We should not break RHEL4 though.
>
> kvmclock provides no timer interrupt... either LAPIC or PIT must be used
> with kvmclock.
Ok, Here is the conclusion:
-- According Marcelo's comments, RHEL4/RHEL5 rely on precise LAPIC timer injection. So LAPIC timer injection logic is necessary.
--LAPIC timer injection always occurred in vcpu context, so it's safe to touch irr and pir for LAPIC timer injection.
--We cannot touch virtual apic page while vcpu is in non-root operation, so the best solution is pass a flag to apic_accept_irq and check whether it's safe to touch vIRR according this flag.
Right?
Best regards,
Yang
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-21 23:06 ` Marcelo Tosatti
2013-03-22 1:50 ` Zhang, Yang Z
@ 2013-03-22 6:53 ` Gleb Natapov
2013-03-22 10:43 ` Marcelo Tosatti
1 sibling, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-22 6:53 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Zhang, Yang Z, Jan Kiszka, kvm
On Thu, Mar 21, 2013 at 08:06:41PM -0300, Marcelo Tosatti wrote:
> On Thu, Mar 21, 2013 at 11:13:39PM +0200, Gleb Natapov wrote:
> > On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
> > > > > > But current PI patches do break them, thats my point. So we either
> > > > > > need to revise them again, or drop LAPIC timer reinjection. Making
> > > > > > apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> > > > > > is dubious though.
> > > > > We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
> > > > > Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
> > > > >
> > > > We cannot touch irr while vcpu is in non-root operation, so we will have
> > > > to pass flag to apic_accept_irq() to let it know that it is called
> > > > synchronously. While all this is possible I want to know which guests
> > > > exactly will we break if we will not track interrupt coalescing for
> > > > lapic timer. If only 2.0 smp kernels will break we can probably drop it.
> > >
> > > RHEL4 / RHEL5 guests.
> > RHEL5 has kvmclock no? We should not break RHEL4 though.
>
> kvmclock provides no timer interrupt... either LAPIC or PIT must be used
> with kvmclock.
I am confused now. If LAPIC is not used for wallclock time keeping, but
only for scheduling the reinjection is actually harmful. Reinjecting the
interrupt will cause needles task rescheduling. So the question is if
there is a Linux kernel that uses LAPIC for wallclock time keeping and
relies on accurate number of injected interrupts to not time drift.
Knowing that Linux tend to disable interrupt it is likely that it tries
to detect and compensate for missing interrupt.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-22 6:53 ` Gleb Natapov
@ 2013-03-22 10:43 ` Marcelo Tosatti
2013-03-22 11:19 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Marcelo Tosatti @ 2013-03-22 10:43 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Zhang, Yang Z, Jan Kiszka, kvm
On Fri, Mar 22, 2013 at 08:53:15AM +0200, Gleb Natapov wrote:
> On Thu, Mar 21, 2013 at 08:06:41PM -0300, Marcelo Tosatti wrote:
> > On Thu, Mar 21, 2013 at 11:13:39PM +0200, Gleb Natapov wrote:
> > > On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
> > > > > > > But current PI patches do break them, thats my point. So we either
> > > > > > > need to revise them again, or drop LAPIC timer reinjection. Making
> > > > > > > apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> > > > > > > is dubious though.
> > > > > > We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
> > > > > > Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
> > > > > >
> > > > > We cannot touch irr while vcpu is in non-root operation, so we will have
> > > > > to pass flag to apic_accept_irq() to let it know that it is called
> > > > > synchronously. While all this is possible I want to know which guests
> > > > > exactly will we break if we will not track interrupt coalescing for
> > > > > lapic timer. If only 2.0 smp kernels will break we can probably drop it.
> > > >
> > > > RHEL4 / RHEL5 guests.
> > > RHEL5 has kvmclock no? We should not break RHEL4 though.
> >
> > kvmclock provides no timer interrupt... either LAPIC or PIT must be used
> > with kvmclock.
> I am confused now. If LAPIC is not used for wallclock time keeping, but
> only for scheduling the reinjection is actually harmful. Reinjecting the
> interrupt will cause needles task rescheduling. So the question is if
> there is a Linux kernel that uses LAPIC for wallclock time keeping and
> relies on accurate number of injected interrupts to not time drift.
See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels before that
commit did not reinject. Which means that all non-RHEL Linux guests
based on that upstream code also suffer from the same problem.
Also any other algorithm which uses LAPIC timers and compare that with
other clocks (such as NMI watchdog) are potentially vulnerable.
Can drop it, and then wait until someone complains (if so).
> Knowing that Linux tend to disable interrupt it is likely that it tries
> to detect and compensate for missing interrupt.
As said above, any algorithm which compares LAPIC timer interrupt with
another clock is vulnerable.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-22 10:43 ` Marcelo Tosatti
@ 2013-03-22 11:19 ` Gleb Natapov
2013-03-24 10:45 ` Zhang, Yang Z
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-03-22 11:19 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: Zhang, Yang Z, Jan Kiszka, kvm
On Fri, Mar 22, 2013 at 07:43:03AM -0300, Marcelo Tosatti wrote:
> On Fri, Mar 22, 2013 at 08:53:15AM +0200, Gleb Natapov wrote:
> > On Thu, Mar 21, 2013 at 08:06:41PM -0300, Marcelo Tosatti wrote:
> > > On Thu, Mar 21, 2013 at 11:13:39PM +0200, Gleb Natapov wrote:
> > > > On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
> > > > > > > > But current PI patches do break them, thats my point. So we either
> > > > > > > > need to revise them again, or drop LAPIC timer reinjection. Making
> > > > > > > > apic_accept_irq semantics "it returns coalescing info, but only sometimes"
> > > > > > > > is dubious though.
> > > > > > > We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.
> > > > > > > Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.
> > > > > > >
> > > > > > We cannot touch irr while vcpu is in non-root operation, so we will have
> > > > > > to pass flag to apic_accept_irq() to let it know that it is called
> > > > > > synchronously. While all this is possible I want to know which guests
> > > > > > exactly will we break if we will not track interrupt coalescing for
> > > > > > lapic timer. If only 2.0 smp kernels will break we can probably drop it.
> > > > >
> > > > > RHEL4 / RHEL5 guests.
> > > > RHEL5 has kvmclock no? We should not break RHEL4 though.
> > >
> > > kvmclock provides no timer interrupt... either LAPIC or PIT must be used
> > > with kvmclock.
> > I am confused now. If LAPIC is not used for wallclock time keeping, but
> > only for scheduling the reinjection is actually harmful. Reinjecting the
> > interrupt will cause needles task rescheduling. So the question is if
> > there is a Linux kernel that uses LAPIC for wallclock time keeping and
> > relies on accurate number of injected interrupts to not time drift.
>
> See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels before that
> commit did not reinject. Which means that all non-RHEL Linux guests
> based on that upstream code also suffer from the same problem.
>
The commit actually fixes guest, not host. The existence of the commit
also means that LAPIC timer reinjection does not solve the problem and
all guests without this commit will suffer from the bug regardless of
what we will decide to do here. Without LAPIC timer reinfection the
effect of the bug will be much more visible and long lasting though.
> Also any other algorithm which uses LAPIC timers and compare that with
> other clocks (such as NMI watchdog) are potentially vulnerable.
They are with or without timer reinjection as commit you pointed to
shows.
>
> Can drop it, and then wait until someone complains (if so).
>
Yes, tough decision to make. All the complains will be guest bugs which
can be hit without reinjection too, but with less probability. Why we so
keen on keeping RTC reinject is that the guests that depends on it
cannot be fixed.
> > Knowing that Linux tend to disable interrupt it is likely that it tries
> > to detect and compensate for missing interrupt.
>
> As said above, any algorithm which compares LAPIC timer interrupt with
> another clock is vulnerable.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* RE: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-22 11:19 ` Gleb Natapov
@ 2013-03-24 10:45 ` Zhang, Yang Z
2013-03-24 19:03 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Zhang, Yang Z @ 2013-03-24 10:45 UTC (permalink / raw)
To: Gleb Natapov, Marcelo Tosatti; +Cc: Jan Kiszka, kvm
Gleb Natapov wrote on 2013-03-22:
> On Fri, Mar 22, 2013 at 07:43:03AM -0300, Marcelo Tosatti wrote:
>> On Fri, Mar 22, 2013 at 08:53:15AM +0200, Gleb Natapov wrote:
>>> On Thu, Mar 21, 2013 at 08:06:41PM -0300, Marcelo Tosatti wrote:
>>>> On Thu, Mar 21, 2013 at 11:13:39PM +0200, Gleb Natapov wrote:
>>>>> On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
>>>>>>>>> But current PI patches do break them, thats my point. So we
>>>>>>>>> either need to revise them again, or drop LAPIC timer
>>>>>>>>> reinjection. Making apic_accept_irq semantics "it returns
>>>>>>>>> coalescing info, but only sometimes" is dubious though.
>>>>>>>> We may rollback to the initial idea: test both irr and pir to get
> coalescing info. In this case, inject LAPIC timer always in vcpu context. So
> apic_accept_irq() will return right coalescing info.
>>>>>>>> Also, we need to add comments to tell caller, apic_accept_irq()
>>>>>>>> can ensure the return value is correct only when caller is in
>>>>>>>> target vcpu context.
>>>>>>>>
>>>>>>> We cannot touch irr while vcpu is in non-root operation, so we
>>>>>>> will have to pass flag to apic_accept_irq() to let it know that it
>>>>>>> is called synchronously. While all this is possible I want to know
>>>>>>> which guests exactly will we break if we will not track interrupt
>>>>>>> coalescing for lapic timer. If only 2.0 smp kernels will break we
>>>>>>> can probably drop it.
>>>>>>
>>>>>> RHEL4 / RHEL5 guests.
>>>>> RHEL5 has kvmclock no? We should not break RHEL4 though.
>>>>
>>>> kvmclock provides no timer interrupt... either LAPIC or PIT must be used
>>>> with kvmclock.
>>> I am confused now. If LAPIC is not used for wallclock time keeping, but
>>> only for scheduling the reinjection is actually harmful. Reinjecting the
>>> interrupt will cause needles task rescheduling. So the question is if
>>> there is a Linux kernel that uses LAPIC for wallclock time keeping and
>>> relies on accurate number of injected interrupts to not time drift.
>>
>> See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels
>> before that commit did not reinject. Which means that all non-RHEL
>> Linux guests based on that upstream code also suffer from the same
>> problem.
>>
> The commit actually fixes guest, not host. The existence of the commit
> also means that LAPIC timer reinjection does not solve the problem and
> all guests without this commit will suffer from the bug regardless of
> what we will decide to do here. Without LAPIC timer reinfection the
> effect of the bug will be much more visible and long lasting though.
>
>> Also any other algorithm which uses LAPIC timers and compare that with
>> other clocks (such as NMI watchdog) are potentially vulnerable.
> They are with or without timer reinjection as commit you pointed to
> shows.
>
>>
>> Can drop it, and then wait until someone complains (if so).
>>
> Yes, tough decision to make. All the complains will be guest bugs which
> can be hit without reinjection too, but with less probability. Why we so
> keen on keeping RTC reinject is that the guests that depends on it
> cannot be fixed.
>
>>> Knowing that Linux tend to disable interrupt it is likely that it tries
>>> to detect and compensate for missing interrupt.
>>
>> As said above, any algorithm which compares LAPIC timer interrupt with
>> another clock is vulnerable.
Any conclusion?
Best regards,
Yang
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-24 10:45 ` Zhang, Yang Z
@ 2013-03-24 19:03 ` Gleb Natapov
0 siblings, 0 replies; 26+ messages in thread
From: Gleb Natapov @ 2013-03-24 19:03 UTC (permalink / raw)
To: Zhang, Yang Z; +Cc: Marcelo Tosatti, Jan Kiszka, kvm
On Sun, Mar 24, 2013 at 10:45:53AM +0000, Zhang, Yang Z wrote:
> Gleb Natapov wrote on 2013-03-22:
> > On Fri, Mar 22, 2013 at 07:43:03AM -0300, Marcelo Tosatti wrote:
> >> On Fri, Mar 22, 2013 at 08:53:15AM +0200, Gleb Natapov wrote:
> >>> On Thu, Mar 21, 2013 at 08:06:41PM -0300, Marcelo Tosatti wrote:
> >>>> On Thu, Mar 21, 2013 at 11:13:39PM +0200, Gleb Natapov wrote:
> >>>>> On Thu, Mar 21, 2013 at 05:51:50PM -0300, Marcelo Tosatti wrote:
> >>>>>>>>> But current PI patches do break them, thats my point. So we
> >>>>>>>>> either need to revise them again, or drop LAPIC timer
> >>>>>>>>> reinjection. Making apic_accept_irq semantics "it returns
> >>>>>>>>> coalescing info, but only sometimes" is dubious though.
> >>>>>>>> We may rollback to the initial idea: test both irr and pir to get
> > coalescing info. In this case, inject LAPIC timer always in vcpu context. So
> > apic_accept_irq() will return right coalescing info.
> >>>>>>>> Also, we need to add comments to tell caller, apic_accept_irq()
> >>>>>>>> can ensure the return value is correct only when caller is in
> >>>>>>>> target vcpu context.
> >>>>>>>>
> >>>>>>> We cannot touch irr while vcpu is in non-root operation, so we
> >>>>>>> will have to pass flag to apic_accept_irq() to let it know that it
> >>>>>>> is called synchronously. While all this is possible I want to know
> >>>>>>> which guests exactly will we break if we will not track interrupt
> >>>>>>> coalescing for lapic timer. If only 2.0 smp kernels will break we
> >>>>>>> can probably drop it.
> >>>>>>
> >>>>>> RHEL4 / RHEL5 guests.
> >>>>> RHEL5 has kvmclock no? We should not break RHEL4 though.
> >>>>
> >>>> kvmclock provides no timer interrupt... either LAPIC or PIT must be used
> >>>> with kvmclock.
> >>> I am confused now. If LAPIC is not used for wallclock time keeping, but
> >>> only for scheduling the reinjection is actually harmful. Reinjecting the
> >>> interrupt will cause needles task rescheduling. So the question is if
> >>> there is a Linux kernel that uses LAPIC for wallclock time keeping and
> >>> relies on accurate number of injected interrupts to not time drift.
> >>
> >> See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels
> >> before that commit did not reinject. Which means that all non-RHEL
> >> Linux guests based on that upstream code also suffer from the same
> >> problem.
> >>
> > The commit actually fixes guest, not host. The existence of the commit
> > also means that LAPIC timer reinjection does not solve the problem and
> > all guests without this commit will suffer from the bug regardless of
> > what we will decide to do here. Without LAPIC timer reinfection the
> > effect of the bug will be much more visible and long lasting though.
> >
> >> Also any other algorithm which uses LAPIC timers and compare that with
> >> other clocks (such as NMI watchdog) are potentially vulnerable.
> > They are with or without timer reinjection as commit you pointed to
> > shows.
> >
> >>
> >> Can drop it, and then wait until someone complains (if so).
> >>
> > Yes, tough decision to make. All the complains will be guest bugs which
> > can be hit without reinjection too, but with less probability. Why we so
> > keen on keeping RTC reinject is that the guests that depends on it
> > cannot be fixed.
> >
> >>> Knowing that Linux tend to disable interrupt it is likely that it tries
> >>> to detect and compensate for missing interrupt.
> >>
> >> As said above, any algorithm which compares LAPIC timer interrupt with
> >> another clock is vulnerable.
> Any conclusion?
>
Lets not check for coalescing in PI patches for now.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-03-17 8:47 ` Gleb Natapov
2013-03-17 10:45 ` Jan Kiszka
@ 2013-04-28 10:15 ` Jan Kiszka
2013-04-28 10:19 ` Gleb Natapov
1 sibling, 1 reply; 26+ messages in thread
From: Jan Kiszka @ 2013-04-28 10:15 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Marcelo Tosatti, kvm
[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]
On 2013-03-17 09:47, Gleb Natapov wrote:
> On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> If the guest didn't take the last APIC timer interrupt yet and generates
>> another one on top, e.g. via periodic mode, we do not block the VCPU
>> even if the guest state is halted. The reason is that
>> apic_has_pending_timer continues to return a non-zero value.
>>
>> Fix this busy loop by taking the IRR content for the LVT vector in
>> apic_has_pending_timer into account.
>>
> Just drop coalescing tacking for lapic interrupt. After posted interrupt
> will be merged __apic_accept_irq() will not longer return coalescing
> information, so the code will be dead anyway.
If I understood the follow-up discussion correctly, we aren't dropping
de-coalescing support yet. So how to proceed with this fix here?
Jan
>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> Not a critical issue, we are looping fully interruptible, but it's ugly
>> to do so IMHO.
>>
>> arch/x86/kvm/lapic.c | 4 +++-
>> 1 files changed, 3 insertions(+), 1 deletions(-)
>>
>> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
>> index a8e9369..658abf5 100644
>> --- a/arch/x86/kvm/lapic.c
>> +++ b/arch/x86/kvm/lapic.c
>> @@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
>> struct kvm_lapic *apic = vcpu->arch.apic;
>>
>> if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
>> - apic_lvt_enabled(apic, APIC_LVTT))
>> + apic_lvt_enabled(apic, APIC_LVTT) &&
>> + !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
>> + apic->regs + APIC_IRR))
>> return atomic_read(&apic->lapic_timer.pending);
>>
>> return 0;
>> --
>> 1.7.3.4
>
> --
> Gleb.
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-04-28 10:15 ` Jan Kiszka
@ 2013-04-28 10:19 ` Gleb Natapov
2013-04-28 10:20 ` Jan Kiszka
0 siblings, 1 reply; 26+ messages in thread
From: Gleb Natapov @ 2013-04-28 10:19 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Marcelo Tosatti, kvm
On Sun, Apr 28, 2013 at 12:15:05PM +0200, Jan Kiszka wrote:
> On 2013-03-17 09:47, Gleb Natapov wrote:
> > On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> >> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>
> >> If the guest didn't take the last APIC timer interrupt yet and generates
> >> another one on top, e.g. via periodic mode, we do not block the VCPU
> >> even if the guest state is halted. The reason is that
> >> apic_has_pending_timer continues to return a non-zero value.
> >>
> >> Fix this busy loop by taking the IRR content for the LVT vector in
> >> apic_has_pending_timer into account.
> >>
> > Just drop coalescing tacking for lapic interrupt. After posted interrupt
> > will be merged __apic_accept_irq() will not longer return coalescing
> > information, so the code will be dead anyway.
>
> If I understood the follow-up discussion correctly, we aren't dropping
> de-coalescing support yet. So how to proceed with this fix here?
>
We do. It does not work if you run on CPU with apicv support already.
> Jan
>
> >
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---
> >>
> >> Not a critical issue, we are looping fully interruptible, but it's ugly
> >> to do so IMHO.
> >>
> >> arch/x86/kvm/lapic.c | 4 +++-
> >> 1 files changed, 3 insertions(+), 1 deletions(-)
> >>
> >> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> >> index a8e9369..658abf5 100644
> >> --- a/arch/x86/kvm/lapic.c
> >> +++ b/arch/x86/kvm/lapic.c
> >> @@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
> >> struct kvm_lapic *apic = vcpu->arch.apic;
> >>
> >> if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
> >> - apic_lvt_enabled(apic, APIC_LVTT))
> >> + apic_lvt_enabled(apic, APIC_LVTT) &&
> >> + !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
> >> + apic->regs + APIC_IRR))
> >> return atomic_read(&apic->lapic_timer.pending);
> >>
> >> return 0;
> >> --
> >> 1.7.3.4
> >
> > --
> > Gleb.
>
>
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-04-28 10:19 ` Gleb Natapov
@ 2013-04-28 10:20 ` Jan Kiszka
2013-04-28 10:23 ` Gleb Natapov
0 siblings, 1 reply; 26+ messages in thread
From: Jan Kiszka @ 2013-04-28 10:20 UTC (permalink / raw)
To: Gleb Natapov; +Cc: Marcelo Tosatti, kvm
[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]
On 2013-04-28 12:19, Gleb Natapov wrote:
> On Sun, Apr 28, 2013 at 12:15:05PM +0200, Jan Kiszka wrote:
>> On 2013-03-17 09:47, Gleb Natapov wrote:
>>> On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>
>>>> If the guest didn't take the last APIC timer interrupt yet and generates
>>>> another one on top, e.g. via periodic mode, we do not block the VCPU
>>>> even if the guest state is halted. The reason is that
>>>> apic_has_pending_timer continues to return a non-zero value.
>>>>
>>>> Fix this busy loop by taking the IRR content for the LVT vector in
>>>> apic_has_pending_timer into account.
>>>>
>>> Just drop coalescing tacking for lapic interrupt. After posted interrupt
>>> will be merged __apic_accept_irq() will not longer return coalescing
>>> information, so the code will be dead anyway.
>>
>> If I understood the follow-up discussion correctly, we aren't dropping
>> de-coalescing support yet. So how to proceed with this fix here?
>>
> We do. It does not work if you run on CPU with apicv support already.
But isn't the code still there and working when apicv is absent?
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers
2013-04-28 10:20 ` Jan Kiszka
@ 2013-04-28 10:23 ` Gleb Natapov
0 siblings, 0 replies; 26+ messages in thread
From: Gleb Natapov @ 2013-04-28 10:23 UTC (permalink / raw)
To: Jan Kiszka; +Cc: Marcelo Tosatti, kvm
On Sun, Apr 28, 2013 at 12:20:12PM +0200, Jan Kiszka wrote:
> On 2013-04-28 12:19, Gleb Natapov wrote:
> > On Sun, Apr 28, 2013 at 12:15:05PM +0200, Jan Kiszka wrote:
> >> On 2013-03-17 09:47, Gleb Natapov wrote:
> >>> On Sat, Mar 16, 2013 at 09:49:07PM +0100, Jan Kiszka wrote:
> >>>> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>>>
> >>>> If the guest didn't take the last APIC timer interrupt yet and generates
> >>>> another one on top, e.g. via periodic mode, we do not block the VCPU
> >>>> even if the guest state is halted. The reason is that
> >>>> apic_has_pending_timer continues to return a non-zero value.
> >>>>
> >>>> Fix this busy loop by taking the IRR content for the LVT vector in
> >>>> apic_has_pending_timer into account.
> >>>>
> >>> Just drop coalescing tacking for lapic interrupt. After posted interrupt
> >>> will be merged __apic_accept_irq() will not longer return coalescing
> >>> information, so the code will be dead anyway.
> >>
> >> If I understood the follow-up discussion correctly, we aren't dropping
> >> de-coalescing support yet. So how to proceed with this fix here?
> >>
> > We do. It does not work if you run on CPU with apicv support already.
>
> But isn't the code still there and working when apicv is absent?
>
Remove it as a fix for busy loop. It is not a good idea to behave differently on different
types of hardware.
--
Gleb.
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2013-04-28 10:23 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-03-16 20:49 [PATCH] KVM: x86: Avoid busy loops over uninjectable pending APIC timers Jan Kiszka
2013-03-17 8:47 ` Gleb Natapov
2013-03-17 10:45 ` Jan Kiszka
2013-03-17 10:47 ` Gleb Natapov
2013-03-20 19:30 ` Marcelo Tosatti
2013-03-20 20:03 ` Marcelo Tosatti
2013-03-20 21:32 ` Gleb Natapov
2013-03-20 23:19 ` Marcelo Tosatti
2013-03-21 4:54 ` Gleb Natapov
2013-03-21 14:02 ` Marcelo Tosatti
2013-03-21 14:18 ` Gleb Natapov
2013-03-21 14:27 ` Zhang, Yang Z
2013-03-21 16:27 ` Gleb Natapov
2013-03-21 20:51 ` Marcelo Tosatti
2013-03-21 21:13 ` Gleb Natapov
2013-03-21 23:06 ` Marcelo Tosatti
2013-03-22 1:50 ` Zhang, Yang Z
2013-03-22 6:53 ` Gleb Natapov
2013-03-22 10:43 ` Marcelo Tosatti
2013-03-22 11:19 ` Gleb Natapov
2013-03-24 10:45 ` Zhang, Yang Z
2013-03-24 19:03 ` Gleb Natapov
2013-04-28 10:15 ` Jan Kiszka
2013-04-28 10:19 ` Gleb Natapov
2013-04-28 10:20 ` Jan Kiszka
2013-04-28 10:23 ` Gleb Natapov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.