KVM Archive on lore.kernel.org
 help / color / Atom feed
From: "Suthikulpanit, Suravee" <Suravee.Suthikulpanit@amd.com>
To: Alexander Graf <graf@amazon.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>
Cc: "pbonzini@redhat.com" <pbonzini@redhat.com>,
	"rkrcmar@redhat.com" <rkrcmar@redhat.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"jschoenh@amazon.de" <jschoenh@amazon.de>,
	"karahmed@amazon.de" <karahmed@amazon.de>,
	"rimasluk@amazon.com" <rimasluk@amazon.com>,
	"Grimm, Jon" <Jon.Grimm@amd.com>
Subject: Re: [PATCH v2 12/15] kvm: i8254: Check LAPIC EOI pending when injecting irq on SVM AVIC
Date: Fri, 13 Sep 2019 14:50:06 +0000
Message-ID: <d0babea8-1f40-3f12-9b96-b5515dc6636f@amd.com> (raw)
In-Reply-To: <a72ac8cc-d0df-9a3f-dff2-0e85e944fb74@amazon.com>

Alex,

On 8/27/19 4:10 AM, Alexander Graf wrote:
> 
> On 26.08.19 22:46, Suthikulpanit, Suravee wrote:
>> Alex,
>>
>> On 8/19/2019 5:42 AM, Alexander Graf wrote:
>>>
>>>
>>> On 15.08.19 18:25, Suthikulpanit, Suravee wrote:
>>>> ACK notifiers don't work with AMD SVM w/ AVIC when the PIT interrupt
>>>> is delivered as edge-triggered fixed interrupt since AMD processors
>>>> cannot exit on EOI for these interrupts.
>>>>
>>>> Add code to check LAPIC pending EOI before injecting any pending PIT
>>>> interrupt on AMD SVM when AVIC is activated.
>>>>
>>>> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
>>>> ---
>>>>    arch/x86/kvm/i8254.c | 31 +++++++++++++++++++++++++------
>>>>    1 file changed, 25 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kvm/i8254.c b/arch/x86/kvm/i8254.c
>>>> index 4a6dc54..31c4a9b 100644
>>>> --- a/arch/x86/kvm/i8254.c
>>>> +++ b/arch/x86/kvm/i8254.c
>>>> @@ -34,10 +34,12 @@
>>>>    #include <linux/kvm_host.h>
>>>>    #include <linux/slab.h>
>>>> +#include <asm/virtext.h>
>>>>    #include "ioapic.h"
>>>>    #include "irq.h"
>>>>    #include "i8254.h"
>>>> +#include "lapic.h"
>>>>    #include "x86.h"
>>>>    #ifndef CONFIG_X86_64
>>>> @@ -236,6 +238,12 @@ static void destroy_pit_timer(struct kvm_pit *pit)
>>>>        kthread_flush_work(&pit->expired);
>>>>    }
>>>> +static inline void kvm_pit_reset_reinject(struct kvm_pit *pit)
>>>> +{
>>>> +    atomic_set(&pit->pit_state.pending, 0);
>>>> +    atomic_set(&pit->pit_state.irq_ack, 1);
>>>> +}
>>>> +
>>>>    static void pit_do_work(struct kthread_work *work)
>>>>    {
>>>>        struct kvm_pit *pit = container_of(work, struct kvm_pit, expired); 
>>>>
>>>> @@ -244,6 +252,23 @@ static void pit_do_work(struct kthread_work *work)
>>>>        int i;
>>>>        struct kvm_kpit_state *ps = &pit->pit_state;
>>>> +    /*
>>>> +     * Since, AMD SVM AVIC accelerates write access to APIC EOI
>>>> +     * register for edge-trigger interrupts. PIT will not be able
>>>> +     * to receive the IRQ ACK notifier and will always be zero.
>>>> +     * Therefore, we check if any LAPIC EOI pending for vector 0
>>>> +     * and reset irq_ack if no pending.
>>>> +     */
>>>> +    if (cpu_has_svm(NULL) && kvm->arch.apicv_state == APICV_ACTIVATED) { 
>>>>
>>>> +        int eoi = 0;
>>>> +
>>>> +        kvm_for_each_vcpu(i, vcpu, kvm)
>>>> +            if (kvm_apic_pending_eoi(vcpu, 0))
>>>> +                eoi++;
>>>> +        if (!eoi)
>>>> +            kvm_pit_reset_reinject(pit);
>>>
>>> In which case would eoi be != 0 when APIC-V is active?
>>
>> That would be the case when guest has not processed and/or still 
>> processing the interrupt.
>> Once the guest writes to APIC EOI register for edge-triggered 
>> interrupt for vector 0,
>> and the AVIC hardware accelerated the access by clearing the highest 
>> priority ISR bit,
>> then the eoi should be zero.
> 
> Thinking about this a bit more, you're basically saying the irq ack 
> notifier never triggers because we don't see the EOI register write, but 
> we can determine the state asynchronously.

Yes, we should be able to determine this in lazy fashion only when we 
need to inject new interrupts.

> The irqfd code also uses the ack notifier for level irq reinjection. 
> Will that break as well?

IIUC, in case of irqfd, the notifier is only used for the case of 
resampling irqfds, which are special variety of irqfds used to emulate 
level triggered interrupts (see include/linux/kvm_irqfd.h). The AVIC 
workaround is only needed for handling EOI for edge-trigger interrupts.

> Wouldn't it make more sense to try to either maintain the ack notifier 
> API or remove it completely if we can't find a way to make it work with 
> APIC-V?

My understanding is that the ack notifier is needed for KVM to support 
KVM_REINJECT_CONTROL ioctl (mentioned here 
(https://lkml.org/lkml/2019/2/6/133).

> So what if we detect that an IRQ vector we're injecting for has an irq 
> notifier? If it does, we set up / start:
> 
>    * an hrtimer that polls for EOI on that vector
>    * a flag so that every vcpu on exit checks for EOI on that vector
>    * a direct call from pit_do_work to check on it as well
> 
> Each of them would go through a single code path that then calls the 
> ack_notifier.
> 
> That way we should be able to just maintain the old API and not get into 
> unpleasant surprises that only manifest on a tiny faction of systems, 
> right?

Let me send out v3 that will consolidate this into a single code path.

Suravee

  reply index

Thread overview: 39+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-15 16:25 [PATCH v2 00/15] kvm: x86: Support AMD SVM AVIC w/ in-kernel irqchip mode Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 01/15] kvm: x86: Modify kvm_x86_ops.get_enable_apicv() to use struct kvm parameter Suthikulpanit, Suravee
2019-08-27  7:15   ` Vitaly Kuznetsov
2019-08-15 16:25 ` [PATCH v2 02/15] kvm: x86: Introduce KVM APICv state Suthikulpanit, Suravee
2019-08-19  9:49   ` Alexander Graf
2019-08-26 19:06     ` Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 03/15] kvm: Add arch-specific per-VM debugfs support Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 04/15] kvm: x86: Add per-VM APICv state debugfs Suthikulpanit, Suravee
2019-08-19  9:57   ` Alexander Graf
2019-08-26 19:41     ` Suthikulpanit, Suravee
2019-08-27  8:20       ` Alexander Graf
2019-08-28 18:39         ` Suthikulpanit, Suravee
2019-08-28 19:36           ` Graf (AWS), Alexander
2019-08-15 16:25 ` [PATCH v2 05/15] kvm: lapic: Introduce APICv update helper function Suthikulpanit, Suravee
2019-08-27  7:22   ` Vitaly Kuznetsov
2019-08-15 16:25 ` [PATCH v2 06/15] kvm: x86: Add support for activate/de-activate APICv at runtime Suthikulpanit, Suravee
2019-08-27  7:29   ` Vitaly Kuznetsov
2019-08-27  8:05     ` Alexander Graf
2019-08-15 16:25 ` [PATCH v2 07/15] kvm: x86: svm: Add support to activate/deactivate posted interrupts Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 08/15] svm: Add support for setup/destroy virutal APIC backing page for AVIC Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 09/15] svm: Add support for activate/deactivate AVIC at runtime Suthikulpanit, Suravee
2019-08-19 10:28   ` Alexander Graf
2019-08-15 16:25 ` [PATCH v2 10/15] kvm: x86: hyperv: Use APICv deactivate request interface Suthikulpanit, Suravee
2019-08-19 10:31   ` Alexander Graf
2019-08-15 16:25 ` [PATCH v2 11/15] svm: Temporary deactivate AVIC during ExtINT handling Suthikulpanit, Suravee
2019-08-19 10:35   ` Alexander Graf
2019-08-28 15:17     ` Suthikulpanit, Suravee
2019-08-28 19:37       ` Graf (AWS), Alexander
2019-09-10 15:46         ` Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 12/15] kvm: i8254: Check LAPIC EOI pending when injecting irq on SVM AVIC Suthikulpanit, Suravee
2019-08-19 10:42   ` Alexander Graf
2019-08-26 20:46     ` Suthikulpanit, Suravee
2019-08-27  9:10       ` Alexander Graf
2019-09-13 14:50         ` Suthikulpanit, Suravee [this message]
2019-08-15 16:25 ` [PATCH v2 13/15] kvm: lapic: Clean up APIC predefined macros Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 14/15] kvm: ioapic: Delay update IOAPIC EOI for RTC Suthikulpanit, Suravee
2019-08-19 11:00   ` Alexander Graf
2019-09-04  2:06     ` Suthikulpanit, Suravee
2019-08-15 16:25 ` [PATCH v2 15/15] svm: Allow AVIC with in-kernel irqchip mode Suthikulpanit, Suravee

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d0babea8-1f40-3f12-9b96-b5515dc6636f@amd.com \
    --to=suravee.suthikulpanit@amd.com \
    --cc=Jon.Grimm@amd.com \
    --cc=graf@amazon.com \
    --cc=joro@8bytes.org \
    --cc=jschoenh@amazon.de \
    --cc=karahmed@amazon.de \
    --cc=kvm@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=pbonzini@redhat.com \
    --cc=rimasluk@amazon.com \
    --cc=rkrcmar@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

KVM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/kvm/0 kvm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 kvm kvm/ https://lore.kernel.org/kvm \
		kvm@vger.kernel.org
	public-inbox-index kvm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.kvm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git