* [patch 1/3] KVM: x86: add start_assignment hook to kvm_x86_ops
2021-05-25 13:41 [patch 0/3] VMX: configure posted interrupt descriptor when assigning device (v5) Marcelo Tosatti
@ 2021-05-25 13:41 ` Marcelo Tosatti
2021-05-25 19:52 ` Peter Xu
2021-05-25 13:41 ` [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK Marcelo Tosatti
` (2 subsequent siblings)
3 siblings, 1 reply; 15+ messages in thread
From: Marcelo Tosatti @ 2021-05-25 13:41 UTC (permalink / raw)
To: kvm
Cc: Paolo Bonzini, Alex Williamson, Sean Christopherson, Peter Xu,
Marcelo Tosatti
Add a start_assignment hook to kvm_x86_ops, which is called when
kvm_arch_start_assignment is done.
The hook is required to update the wakeup vector of a sleeping vCPU
when a device is assigned to the guest.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Index: kvm/arch/x86/include/asm/kvm_host.h
===================================================================
--- kvm.orig/arch/x86/include/asm/kvm_host.h
+++ kvm/arch/x86/include/asm/kvm_host.h
@@ -1322,6 +1322,7 @@ struct kvm_x86_ops {
int (*update_pi_irte)(struct kvm *kvm, unsigned int host_irq,
uint32_t guest_irq, bool set);
+ void (*start_assignment)(struct kvm *kvm);
void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);
bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu);
Index: kvm/arch/x86/kvm/x86.c
===================================================================
--- kvm.orig/arch/x86/kvm/x86.c
+++ kvm/arch/x86/kvm/x86.c
@@ -11295,7 +11295,8 @@ bool kvm_arch_can_dequeue_async_page_pre
void kvm_arch_start_assignment(struct kvm *kvm)
{
- atomic_inc(&kvm->arch.assigned_device_count);
+ if (atomic_inc_return(&kvm->arch.assigned_device_count) == 1)
+ static_call_cond(kvm_x86_start_assignment)(kvm);
}
EXPORT_SYMBOL_GPL(kvm_arch_start_assignment);
Index: kvm/arch/x86/include/asm/kvm-x86-ops.h
===================================================================
--- kvm.orig/arch/x86/include/asm/kvm-x86-ops.h
+++ kvm/arch/x86/include/asm/kvm-x86-ops.h
@@ -99,6 +99,7 @@ KVM_X86_OP_NULL(post_block)
KVM_X86_OP_NULL(vcpu_blocking)
KVM_X86_OP_NULL(vcpu_unblocking)
KVM_X86_OP_NULL(update_pi_irte)
+KVM_X86_OP_NULL(start_assignment)
KVM_X86_OP_NULL(apicv_post_state_restore)
KVM_X86_OP_NULL(dy_apicv_has_pending_interrupt)
KVM_X86_OP_NULL(set_hv_timer)
^ permalink raw reply [flat|nested] 15+ messages in thread
* [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
2021-05-25 13:41 [patch 0/3] VMX: configure posted interrupt descriptor when assigning device (v5) Marcelo Tosatti
2021-05-25 13:41 ` [patch 1/3] KVM: x86: add start_assignment hook to kvm_x86_ops Marcelo Tosatti
@ 2021-05-25 13:41 ` Marcelo Tosatti
2021-05-25 19:14 ` Peter Xu
2021-05-27 11:57 ` Paolo Bonzini
2021-05-25 13:41 ` [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device Marcelo Tosatti
2021-05-27 11:59 ` [patch 0/3] VMX: configure posted interrupt descriptor when assigning device (v5) Paolo Bonzini
3 siblings, 2 replies; 15+ messages in thread
From: Marcelo Tosatti @ 2021-05-25 13:41 UTC (permalink / raw)
To: kvm
Cc: Paolo Bonzini, Alex Williamson, Sean Christopherson, Peter Xu,
Marcelo Tosatti
KVM_REQ_UNBLOCK will be used to exit a vcpu from
its inner vcpu halt emulation loop.
Rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK, switch
PowerPC to arch specific request bit.
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Index: kvm/include/linux/kvm_host.h
===================================================================
--- kvm.orig/include/linux/kvm_host.h
+++ kvm/include/linux/kvm_host.h
@@ -146,7 +146,7 @@ static inline bool is_error_page(struct
*/
#define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
#define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
-#define KVM_REQ_PENDING_TIMER 2
+#define KVM_REQ_UNBLOCK 2
#define KVM_REQ_UNHALT 3
#define KVM_REQUEST_ARCH_BASE 8
Index: kvm/virt/kvm/kvm_main.c
===================================================================
--- kvm.orig/virt/kvm/kvm_main.c
+++ kvm/virt/kvm/kvm_main.c
@@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k
goto out;
if (signal_pending(current))
goto out;
+ if (kvm_check_request(KVM_REQ_UNBLOCK, vcpu))
+ goto out;
ret = 0;
out:
Index: kvm/Documentation/virt/kvm/vcpu-requests.rst
===================================================================
--- kvm.orig/Documentation/virt/kvm/vcpu-requests.rst
+++ kvm/Documentation/virt/kvm/vcpu-requests.rst
@@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD
necessary to inform each VCPU to completely refresh the tables. This
request is used for that.
-KVM_REQ_PENDING_TIMER
+KVM_REQ_UNBLOCK
This request may be made from a timer handler run on the host on behalf
- of a VCPU. It informs the VCPU thread to inject a timer interrupt.
+ of a VCPU, or when device assignment is performed. It informs the VCPU to
+ exit the vcpu halt inner loop.
KVM_REQ_UNHALT
Index: kvm/arch/powerpc/include/asm/kvm_host.h
===================================================================
--- kvm.orig/arch/powerpc/include/asm/kvm_host.h
+++ kvm/arch/powerpc/include/asm/kvm_host.h
@@ -51,6 +51,7 @@
/* PPC-specific vcpu->requests bit members */
#define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)
#define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1)
+#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)
#include <linux/mmu_notifier.h>
Index: kvm/arch/x86/kvm/lapic.c
===================================================================
--- kvm.orig/arch/x86/kvm/lapic.c
+++ kvm/arch/x86/kvm/lapic.c
@@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv
}
atomic_inc(&apic->lapic_timer.pending);
- kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
+ kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
if (from_timer_fn)
kvm_vcpu_kick(vcpu);
}
Index: kvm/arch/x86/kvm/x86.c
===================================================================
--- kvm.orig/arch/x86/kvm/x86.c
+++ kvm/arch/x86/kvm/x86.c
@@ -9300,7 +9300,7 @@ static int vcpu_run(struct kvm_vcpu *vcp
if (r <= 0)
break;
- kvm_clear_request(KVM_REQ_PENDING_TIMER, vcpu);
+ kvm_clear_request(KVM_REQ_UNBLOCK, vcpu);
if (kvm_cpu_has_pending_timer(vcpu))
kvm_inject_pending_timer_irqs(vcpu);
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
2021-05-25 13:41 ` [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK Marcelo Tosatti
@ 2021-05-25 19:14 ` Peter Xu
2021-05-25 19:26 ` Marcelo Tosatti
2021-05-27 11:57 ` Paolo Bonzini
1 sibling, 1 reply; 15+ messages in thread
From: Peter Xu @ 2021-05-25 19:14 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: kvm, Paolo Bonzini, Alex Williamson, Sean Christopherson
On Tue, May 25, 2021 at 10:41:17AM -0300, Marcelo Tosatti wrote:
> KVM_REQ_UNBLOCK will be used to exit a vcpu from
> its inner vcpu halt emulation loop.
>
> Rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK, switch
> PowerPC to arch specific request bit.
>
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>
> Index: kvm/include/linux/kvm_host.h
> ===================================================================
> --- kvm.orig/include/linux/kvm_host.h
> +++ kvm/include/linux/kvm_host.h
> @@ -146,7 +146,7 @@ static inline bool is_error_page(struct
> */
> #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
> #define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
> -#define KVM_REQ_PENDING_TIMER 2
> +#define KVM_REQ_UNBLOCK 2
> #define KVM_REQ_UNHALT 3
> #define KVM_REQUEST_ARCH_BASE 8
>
> Index: kvm/virt/kvm/kvm_main.c
> ===================================================================
> --- kvm.orig/virt/kvm/kvm_main.c
> +++ kvm/virt/kvm/kvm_main.c
> @@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k
> goto out;
> if (signal_pending(current))
> goto out;
> + if (kvm_check_request(KVM_REQ_UNBLOCK, vcpu))
> + goto out;
>
> ret = 0;
> out:
> Index: kvm/Documentation/virt/kvm/vcpu-requests.rst
> ===================================================================
> --- kvm.orig/Documentation/virt/kvm/vcpu-requests.rst
> +++ kvm/Documentation/virt/kvm/vcpu-requests.rst
> @@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD
> necessary to inform each VCPU to completely refresh the tables. This
> request is used for that.
>
> -KVM_REQ_PENDING_TIMER
> +KVM_REQ_UNBLOCK
>
> This request may be made from a timer handler run on the host on behalf
> - of a VCPU. It informs the VCPU thread to inject a timer interrupt.
> + of a VCPU, or when device assignment is performed. It informs the VCPU to
> + exit the vcpu halt inner loop.
>
> KVM_REQ_UNHALT
>
> Index: kvm/arch/powerpc/include/asm/kvm_host.h
> ===================================================================
> --- kvm.orig/arch/powerpc/include/asm/kvm_host.h
> +++ kvm/arch/powerpc/include/asm/kvm_host.h
> @@ -51,6 +51,7 @@
> /* PPC-specific vcpu->requests bit members */
> #define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)
> #define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1)
> +#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)
>
> #include <linux/mmu_notifier.h>
>
> Index: kvm/arch/x86/kvm/lapic.c
> ===================================================================
> --- kvm.orig/arch/x86/kvm/lapic.c
> +++ kvm/arch/x86/kvm/lapic.c
> @@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv
> }
>
> atomic_inc(&apic->lapic_timer.pending);
> - kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
> + kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
> if (from_timer_fn)
> kvm_vcpu_kick(vcpu);
> }
Pure question on the existing code: why do we need kvm_make_request() for
timer? As I see kvm_vcpu_check_block() already checks explicitly for timers:
static int kvm_vcpu_check_block(struct kvm_vcpu *vcpu)
{
...
if (kvm_cpu_has_pending_timer(vcpu))
goto out;
...
}
for x86:
int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
{
if (lapic_in_kernel(vcpu))
return apic_has_pending_timer(vcpu);
return 0;
}
So wondering why we can drop the two references to KVM_REQ_PENDING_TIMER in x86
directly..
> Index: kvm/arch/x86/kvm/x86.c
> ===================================================================
> --- kvm.orig/arch/x86/kvm/x86.c
> +++ kvm/arch/x86/kvm/x86.c
> @@ -9300,7 +9300,7 @@ static int vcpu_run(struct kvm_vcpu *vcp
> if (r <= 0)
> break;
>
> - kvm_clear_request(KVM_REQ_PENDING_TIMER, vcpu);
> + kvm_clear_request(KVM_REQ_UNBLOCK, vcpu);
> if (kvm_cpu_has_pending_timer(vcpu))
> kvm_inject_pending_timer_irqs(vcpu);
>
>
>
--
Peter Xu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
2021-05-25 19:14 ` Peter Xu
@ 2021-05-25 19:26 ` Marcelo Tosatti
2021-05-25 19:52 ` Peter Xu
0 siblings, 1 reply; 15+ messages in thread
From: Marcelo Tosatti @ 2021-05-25 19:26 UTC (permalink / raw)
To: Peter Xu; +Cc: kvm, Paolo Bonzini, Alex Williamson, Sean Christopherson
On Tue, May 25, 2021 at 03:14:33PM -0400, Peter Xu wrote:
> On Tue, May 25, 2021 at 10:41:17AM -0300, Marcelo Tosatti wrote:
> > KVM_REQ_UNBLOCK will be used to exit a vcpu from
> > its inner vcpu halt emulation loop.
> >
> > Rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK, switch
> > PowerPC to arch specific request bit.
> >
> > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> >
> > Index: kvm/include/linux/kvm_host.h
> > ===================================================================
> > --- kvm.orig/include/linux/kvm_host.h
> > +++ kvm/include/linux/kvm_host.h
> > @@ -146,7 +146,7 @@ static inline bool is_error_page(struct
> > */
> > #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
> > #define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
> > -#define KVM_REQ_PENDING_TIMER 2
> > +#define KVM_REQ_UNBLOCK 2
> > #define KVM_REQ_UNHALT 3
> > #define KVM_REQUEST_ARCH_BASE 8
> >
> > Index: kvm/virt/kvm/kvm_main.c
> > ===================================================================
> > --- kvm.orig/virt/kvm/kvm_main.c
> > +++ kvm/virt/kvm/kvm_main.c
> > @@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k
> > goto out;
> > if (signal_pending(current))
> > goto out;
> > + if (kvm_check_request(KVM_REQ_UNBLOCK, vcpu))
> > + goto out;
> >
> > ret = 0;
> > out:
> > Index: kvm/Documentation/virt/kvm/vcpu-requests.rst
> > ===================================================================
> > --- kvm.orig/Documentation/virt/kvm/vcpu-requests.rst
> > +++ kvm/Documentation/virt/kvm/vcpu-requests.rst
> > @@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD
> > necessary to inform each VCPU to completely refresh the tables. This
> > request is used for that.
> >
> > -KVM_REQ_PENDING_TIMER
> > +KVM_REQ_UNBLOCK
> >
> > This request may be made from a timer handler run on the host on behalf
> > - of a VCPU. It informs the VCPU thread to inject a timer interrupt.
> > + of a VCPU, or when device assignment is performed. It informs the VCPU to
> > + exit the vcpu halt inner loop.
> >
> > KVM_REQ_UNHALT
> >
> > Index: kvm/arch/powerpc/include/asm/kvm_host.h
> > ===================================================================
> > --- kvm.orig/arch/powerpc/include/asm/kvm_host.h
> > +++ kvm/arch/powerpc/include/asm/kvm_host.h
> > @@ -51,6 +51,7 @@
> > /* PPC-specific vcpu->requests bit members */
> > #define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)
> > #define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1)
> > +#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)
> >
> > #include <linux/mmu_notifier.h>
> >
> > Index: kvm/arch/x86/kvm/lapic.c
> > ===================================================================
> > --- kvm.orig/arch/x86/kvm/lapic.c
> > +++ kvm/arch/x86/kvm/lapic.c
> > @@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv
> > }
> >
> > atomic_inc(&apic->lapic_timer.pending);
> > - kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
> > + kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
> > if (from_timer_fn)
> > kvm_vcpu_kick(vcpu);
> > }
>
> Pure question on the existing code: why do we need kvm_make_request() for
> timer? As I see kvm_vcpu_check_block() already checks explicitly for timers:
>
> static int kvm_vcpu_check_block(struct kvm_vcpu *vcpu)
> {
> ...
> if (kvm_cpu_has_pending_timer(vcpu))
> goto out;
> ...
> }
>
> for x86:
>
> int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
> {
> if (lapic_in_kernel(vcpu))
> return apic_has_pending_timer(vcpu);
>
> return 0;
> }
>
> So wondering why we can drop the two references to KVM_REQ_PENDING_TIMER in x86
> directly..
See commit 06e05645661211b9eaadaf6344c335d2e80f0ba2
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
2021-05-25 19:26 ` Marcelo Tosatti
@ 2021-05-25 19:52 ` Peter Xu
2021-05-27 11:57 ` Paolo Bonzini
0 siblings, 1 reply; 15+ messages in thread
From: Peter Xu @ 2021-05-25 19:52 UTC (permalink / raw)
To: Marcelo Tosatti; +Cc: kvm, Paolo Bonzini, Alex Williamson, Sean Christopherson
On Tue, May 25, 2021 at 04:26:37PM -0300, Marcelo Tosatti wrote:
> On Tue, May 25, 2021 at 03:14:33PM -0400, Peter Xu wrote:
> > On Tue, May 25, 2021 at 10:41:17AM -0300, Marcelo Tosatti wrote:
> > > KVM_REQ_UNBLOCK will be used to exit a vcpu from
> > > its inner vcpu halt emulation loop.
> > >
> > > Rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK, switch
> > > PowerPC to arch specific request bit.
> > >
> > > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
> > >
> > > Index: kvm/include/linux/kvm_host.h
> > > ===================================================================
> > > --- kvm.orig/include/linux/kvm_host.h
> > > +++ kvm/include/linux/kvm_host.h
> > > @@ -146,7 +146,7 @@ static inline bool is_error_page(struct
> > > */
> > > #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
> > > #define KVM_REQ_MMU_RELOAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
> > > -#define KVM_REQ_PENDING_TIMER 2
> > > +#define KVM_REQ_UNBLOCK 2
> > > #define KVM_REQ_UNHALT 3
> > > #define KVM_REQUEST_ARCH_BASE 8
> > >
> > > Index: kvm/virt/kvm/kvm_main.c
> > > ===================================================================
> > > --- kvm.orig/virt/kvm/kvm_main.c
> > > +++ kvm/virt/kvm/kvm_main.c
> > > @@ -2794,6 +2794,8 @@ static int kvm_vcpu_check_block(struct k
> > > goto out;
> > > if (signal_pending(current))
> > > goto out;
> > > + if (kvm_check_request(KVM_REQ_UNBLOCK, vcpu))
> > > + goto out;
> > >
> > > ret = 0;
> > > out:
> > > Index: kvm/Documentation/virt/kvm/vcpu-requests.rst
> > > ===================================================================
> > > --- kvm.orig/Documentation/virt/kvm/vcpu-requests.rst
> > > +++ kvm/Documentation/virt/kvm/vcpu-requests.rst
> > > @@ -118,10 +118,11 @@ KVM_REQ_MMU_RELOAD
> > > necessary to inform each VCPU to completely refresh the tables. This
> > > request is used for that.
> > >
> > > -KVM_REQ_PENDING_TIMER
> > > +KVM_REQ_UNBLOCK
> > >
> > > This request may be made from a timer handler run on the host on behalf
> > > - of a VCPU. It informs the VCPU thread to inject a timer interrupt.
> > > + of a VCPU, or when device assignment is performed. It informs the VCPU to
> > > + exit the vcpu halt inner loop.
> > >
> > > KVM_REQ_UNHALT
> > >
> > > Index: kvm/arch/powerpc/include/asm/kvm_host.h
> > > ===================================================================
> > > --- kvm.orig/arch/powerpc/include/asm/kvm_host.h
> > > +++ kvm/arch/powerpc/include/asm/kvm_host.h
> > > @@ -51,6 +51,7 @@
> > > /* PPC-specific vcpu->requests bit members */
> > > #define KVM_REQ_WATCHDOG KVM_ARCH_REQ(0)
> > > #define KVM_REQ_EPR_EXIT KVM_ARCH_REQ(1)
> > > +#define KVM_REQ_PENDING_TIMER KVM_ARCH_REQ(2)
> > >
> > > #include <linux/mmu_notifier.h>
> > >
> > > Index: kvm/arch/x86/kvm/lapic.c
> > > ===================================================================
> > > --- kvm.orig/arch/x86/kvm/lapic.c
> > > +++ kvm/arch/x86/kvm/lapic.c
> > > @@ -1657,7 +1657,7 @@ static void apic_timer_expired(struct kv
> > > }
> > >
> > > atomic_inc(&apic->lapic_timer.pending);
> > > - kvm_make_request(KVM_REQ_PENDING_TIMER, vcpu);
> > > + kvm_make_request(KVM_REQ_UNBLOCK, vcpu);
> > > if (from_timer_fn)
> > > kvm_vcpu_kick(vcpu);
> > > }
> >
> > Pure question on the existing code: why do we need kvm_make_request() for
> > timer? As I see kvm_vcpu_check_block() already checks explicitly for timers:
> >
> > static int kvm_vcpu_check_block(struct kvm_vcpu *vcpu)
> > {
> > ...
> > if (kvm_cpu_has_pending_timer(vcpu))
> > goto out;
> > ...
> > }
> >
> > for x86:
> >
> > int kvm_cpu_has_pending_timer(struct kvm_vcpu *vcpu)
> > {
> > if (lapic_in_kernel(vcpu))
> > return apic_has_pending_timer(vcpu);
> >
> > return 0;
> > }
> >
> > So wondering why we can drop the two references to KVM_REQ_PENDING_TIMER in x86
> > directly..
>
> See commit 06e05645661211b9eaadaf6344c335d2e80f0ba2
I see, thanks Marcelo.
Then we might have checked twice on timer pending in kvm_vcpu_check_block() for
x86, as we also checks KVM_REQ_UNBLOCK now. Didn't think further on how to make
it better, e.g. simply dropping kvm_cpu_has_pending_timer() won't work since
it seems to still be useful for non-x86..
Then this patch looks good to me:
Reviewed-by: Peter Xu <peterx@redhat.com>
--
Peter Xu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
2021-05-25 19:52 ` Peter Xu
@ 2021-05-27 11:57 ` Paolo Bonzini
0 siblings, 0 replies; 15+ messages in thread
From: Paolo Bonzini @ 2021-05-27 11:57 UTC (permalink / raw)
To: Peter Xu, Marcelo Tosatti; +Cc: kvm, Alex Williamson, Sean Christopherson
On 25/05/21 21:52, Peter Xu wrote:
> Didn't think further on how to make it better, e.g. simply dropping
> kvm_cpu_has_pending_timer() won't work since it seems to still be
> useful for non-x86..
It should be possible to use KVM_REQ_UNBLOCK in other architectures, and
drop kvm_cpu_has_pending_timer().
Paolo
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK
2021-05-25 13:41 ` [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK Marcelo Tosatti
2021-05-25 19:14 ` Peter Xu
@ 2021-05-27 11:57 ` Paolo Bonzini
1 sibling, 0 replies; 15+ messages in thread
From: Paolo Bonzini @ 2021-05-27 11:57 UTC (permalink / raw)
To: Marcelo Tosatti, kvm; +Cc: Alex Williamson, Sean Christopherson, Peter Xu
On 25/05/21 15:41, Marcelo Tosatti wrote:
> -KVM_REQ_PENDING_TIMER
> +KVM_REQ_UNBLOCK
>
> This request may be made from a timer handler run on the host on behalf
> - of a VCPU. It informs the VCPU thread to inject a timer interrupt.
> + of a VCPU, or when device assignment is performed. It informs the VCPU to
> + exit the vcpu halt inner loop.
>
A small change:
This request informs the vCPU to exit kvm_vcpu_block. It is used for
example from timer handlers that run on the host on behalf of a vCPU,
or in order to update the interrupt routing and ensure that assigned
devices will wake up the vCPU.
Paolo
^ permalink raw reply [flat|nested] 15+ messages in thread
* [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
2021-05-25 13:41 [patch 0/3] VMX: configure posted interrupt descriptor when assigning device (v5) Marcelo Tosatti
2021-05-25 13:41 ` [patch 1/3] KVM: x86: add start_assignment hook to kvm_x86_ops Marcelo Tosatti
2021-05-25 13:41 ` [patch 2/3] KVM: rename KVM_REQ_PENDING_TIMER to KVM_REQ_UNBLOCK Marcelo Tosatti
@ 2021-05-25 13:41 ` Marcelo Tosatti
2021-05-25 19:55 ` Peter Xu
2021-05-25 21:09 ` kernel test robot
2021-05-27 11:59 ` [patch 0/3] VMX: configure posted interrupt descriptor when assigning device (v5) Paolo Bonzini
3 siblings, 2 replies; 15+ messages in thread
From: Marcelo Tosatti @ 2021-05-25 13:41 UTC (permalink / raw)
To: kvm
Cc: Paolo Bonzini, Alex Williamson, Sean Christopherson, Peter Xu,
Pei Zhang, Marcelo Tosatti
For VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector"
to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the
vector programmed for the device on the CPU, test-and-set the
ON bit on the posted interrupt descriptor, and if the ON bit is clear
generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up
the target vcpu.
Problem is that pi_post_block only programs the notification vector
if kvm_arch_has_assigned_device() is true. Its possible for the
following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false,
notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,
kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until
RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick
vcpus out of the halt loop, so the notification vector is
properly reprogrammed to the wakeup vector.
Reported-by: Pei Zhang <pezhang@redhat.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
Index: kvm/arch/x86/kvm/vmx/posted_intr.c
===================================================================
--- kvm.orig/arch/x86/kvm/vmx/posted_intr.c
+++ kvm/arch/x86/kvm/vmx/posted_intr.c
@@ -236,6 +236,13 @@ bool pi_has_pending_interrupt(struct kvm
(pi_test_sn(pi_desc) && !pi_is_pir_empty(pi_desc));
}
+void vmx_pi_start_assignment(struct kvm *kvm)
+{
+ if (!irq_remapping_cap(IRQ_POSTING_CAP))
+ return;
+
+ kvm_make_all_cpus_request(kvm, KVM_REQ_UNBLOCK);
+}
/*
* pi_update_irte - set IRTE for Posted-Interrupts
Index: kvm/arch/x86/kvm/vmx/posted_intr.h
===================================================================
--- kvm.orig/arch/x86/kvm/vmx/posted_intr.h
+++ kvm/arch/x86/kvm/vmx/posted_intr.h
@@ -95,5 +95,6 @@ void __init pi_init_cpu(int cpu);
bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu);
int pi_update_irte(struct kvm *kvm, unsigned int host_irq, uint32_t guest_irq,
bool set);
+void vmx_pi_start_assignment(struct kvm *kvm);
#endif /* __KVM_X86_VMX_POSTED_INTR_H */
Index: kvm/arch/x86/kvm/vmx/vmx.c
===================================================================
--- kvm.orig/arch/x86/kvm/vmx/vmx.c
+++ kvm/arch/x86/kvm/vmx/vmx.c
@@ -7732,6 +7732,7 @@ static struct kvm_x86_ops vmx_x86_ops __
.nested_ops = &vmx_nested_ops,
.update_pi_irte = pi_update_irte,
+ .start_assignment = vmx_pi_start_assignment,
#ifdef CONFIG_X86_64
.set_hv_timer = vmx_set_hv_timer,
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
2021-05-25 13:41 ` [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device Marcelo Tosatti
@ 2021-05-25 19:55 ` Peter Xu
2021-05-26 17:20 ` [patch 3/3 V2] " Marcelo Tosatti
2021-05-25 21:09 ` kernel test robot
1 sibling, 1 reply; 15+ messages in thread
From: Peter Xu @ 2021-05-25 19:55 UTC (permalink / raw)
To: Marcelo Tosatti
Cc: kvm, Paolo Bonzini, Alex Williamson, Sean Christopherson, Pei Zhang
On Tue, May 25, 2021 at 10:41:18AM -0300, Marcelo Tosatti wrote:
> For VMX, when a vcpu enters HLT emulation, pi_post_block will:
>
> 1) Add vcpu to per-cpu list of blocked vcpus.
>
> 2) Program the posted-interrupt descriptor "notification vector"
> to POSTED_INTR_WAKEUP_VECTOR
>
> With interrupt remapping, an interrupt will set the PIR bit for the
> vector programmed for the device on the CPU, test-and-set the
> ON bit on the posted interrupt descriptor, and if the ON bit is clear
> generate an interrupt for the notification vector.
>
> This way, the target CPU wakes upon a device interrupt and wakes up
> the target vcpu.
>
> Problem is that pi_post_block only programs the notification vector
> if kvm_arch_has_assigned_device() is true. Its possible for the
> following to happen:
>
> 1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false,
> notification vector is not programmed
> 2) device is assigned to VM
> 3) device interrupts vcpu V, sets ON bit
> (notification vector not programmed, so pcpu P remains in idle)
> 4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,
> kvm_vcpu_kick is skipped
> 5) vcpu 0 busy spins on vcpu V's response for several seconds, until
> RCU watchdog NMIs all vCPUs.
>
> To fix this, use the start_assignment kvm_x86_ops callback to kick
> vcpus out of the halt loop, so the notification vector is
> properly reprogrammed to the wakeup vector.
>
> Reported-by: Pei Zhang <pezhang@redhat.com>
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>
>
> Index: kvm/arch/x86/kvm/vmx/posted_intr.c
> ===================================================================
> --- kvm.orig/arch/x86/kvm/vmx/posted_intr.c
> +++ kvm/arch/x86/kvm/vmx/posted_intr.c
> @@ -236,6 +236,13 @@ bool pi_has_pending_interrupt(struct kvm
> (pi_test_sn(pi_desc) && !pi_is_pir_empty(pi_desc));
> }
>
> +void vmx_pi_start_assignment(struct kvm *kvm)
> +{
> + if (!irq_remapping_cap(IRQ_POSTING_CAP))
> + return;
> +
> + kvm_make_all_cpus_request(kvm, KVM_REQ_UNBLOCK);
Shall we add a simple comment block explaining why we need this?
> +}
The patch itself looks right to me.
Reviewed-by: Peter Xu <peterx@redhat.com>
Thanks,
--
Peter Xu
^ permalink raw reply [flat|nested] 15+ messages in thread
* [patch 3/3 V2] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
2021-05-25 19:55 ` Peter Xu
@ 2021-05-26 17:20 ` Marcelo Tosatti
0 siblings, 0 replies; 15+ messages in thread
From: Marcelo Tosatti @ 2021-05-26 17:20 UTC (permalink / raw)
To: Peter Xu
Cc: kvm, Paolo Bonzini, Alex Williamson, Sean Christopherson, Pei Zhang
For VMX, when a vcpu enters HLT emulation, pi_post_block will:
1) Add vcpu to per-cpu list of blocked vcpus.
2) Program the posted-interrupt descriptor "notification vector"
to POSTED_INTR_WAKEUP_VECTOR
With interrupt remapping, an interrupt will set the PIR bit for the
vector programmed for the device on the CPU, test-and-set the
ON bit on the posted interrupt descriptor, and if the ON bit is clear
generate an interrupt for the notification vector.
This way, the target CPU wakes upon a device interrupt and wakes up
the target vcpu.
Problem is that pi_post_block only programs the notification vector
if kvm_arch_has_assigned_device() is true. Its possible for the
following to happen:
1) vcpu V HLTs on pcpu P, kvm_arch_has_assigned_device is false,
notification vector is not programmed
2) device is assigned to VM
3) device interrupts vcpu V, sets ON bit
(notification vector not programmed, so pcpu P remains in idle)
4) vcpu 0 IPIs vcpu V (in guest), but since pi descriptor ON bit is set,
kvm_vcpu_kick is skipped
5) vcpu 0 busy spins on vcpu V's response for several seconds, until
RCU watchdog NMIs all vCPUs.
To fix this, use the start_assignment kvm_x86_ops callback to kick
vcpus out of the halt loop, so the notification vector is
properly reprogrammed to the wakeup vector.
Reported-by: Pei Zhang <pezhang@redhat.com>
Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
For build error:
Reported-by: kernel test robot <lkp@intel.com>
---
v2: Add brief comment to vmx_pi_start_assignment (Peter Xu).
Export kvm_make_all_cpus_request (kernel test robot).
Index: linux-2.6/arch/x86/kvm/vmx/posted_intr.c
===================================================================
--- linux-2.6.orig/arch/x86/kvm/vmx/posted_intr.c
+++ linux-2.6/arch/x86/kvm/vmx/posted_intr.c
@@ -238,6 +238,20 @@ bool pi_has_pending_interrupt(struct kvm
/*
+ * Bail out of the block loop if the VM has an assigned
+ * device, but the blocking vCPU didn't reconfigure the
+ * PI.NV to the wakeup vector, i.e. the assigned device
+ * came along after the initial check in pi_pre_block().
+ */
+void vmx_pi_start_assignment(struct kvm *kvm)
+{
+ if (!irq_remapping_cap(IRQ_POSTING_CAP))
+ return;
+
+ kvm_make_all_cpus_request(kvm, KVM_REQ_UNBLOCK);
+}
+
+/*
* pi_update_irte - set IRTE for Posted-Interrupts
*
* @kvm: kvm
Index: linux-2.6/arch/x86/kvm/vmx/posted_intr.h
===================================================================
--- linux-2.6.orig/arch/x86/kvm/vmx/posted_intr.h
+++ linux-2.6/arch/x86/kvm/vmx/posted_intr.h
@@ -95,5 +95,6 @@ void __init pi_init_cpu(int cpu);
bool pi_has_pending_interrupt(struct kvm_vcpu *vcpu);
int pi_update_irte(struct kvm *kvm, unsigned int host_irq, uint32_t guest_irq,
bool set);
+void vmx_pi_start_assignment(struct kvm *kvm);
#endif /* __KVM_X86_VMX_POSTED_INTR_H */
Index: linux-2.6/arch/x86/kvm/vmx/vmx.c
===================================================================
--- linux-2.6.orig/arch/x86/kvm/vmx/vmx.c
+++ linux-2.6/arch/x86/kvm/vmx/vmx.c
@@ -7732,6 +7732,7 @@ static struct kvm_x86_ops vmx_x86_ops __
.nested_ops = &vmx_nested_ops,
.update_pi_irte = pi_update_irte,
+ .start_assignment = vmx_pi_start_assignment,
#ifdef CONFIG_X86_64
.set_hv_timer = vmx_set_hv_timer,
Index: linux-2.6/virt/kvm/kvm_main.c
===================================================================
--- linux-2.6.orig/virt/kvm/kvm_main.c
+++ linux-2.6/virt/kvm/kvm_main.c
@@ -307,6 +307,7 @@ bool kvm_make_all_cpus_request(struct kv
{
return kvm_make_all_cpus_request_except(kvm, req, NULL);
}
+EXPORT_SYMBOL_GPL(kvm_make_all_cpus_request);
#ifndef CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL
void kvm_flush_remote_tlbs(struct kvm *kvm)
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device
2021-05-25 13:41 ` [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device Marcelo Tosatti
@ 2021-05-25 21:09 ` kernel test robot
2021-05-25 21:09 ` kernel test robot
1 sibling, 0 replies; 15+ messages in thread
From: kernel test robot @ 2021-05-25 21:09 UTC (permalink / raw)
To: Marcelo Tosatti, kvm
Cc: kbuild-all, Paolo Bonzini, Alex Williamson, Sean Christopherson,
Peter Xu, Pei Zhang, Marcelo Tosatti
[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]
Hi Marcelo,
Thank you for the patch! Yet something to improve:
[auto build test ERROR on kvm/queue]
[also build test ERROR on vhost/linux-next v5.13-rc3 next-20210525]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Marcelo-Tosatti/VMX-configure-posted-interrupt-descriptor-when-assigning-device-v5/20210525-215604
base: https://git.kernel.org/pub/scm/virt/kvm/kvm.git queue
config: x86_64-rhel-8.3 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
# https://github.com/0day-ci/linux/commit/e515e68f330d3787af0952dcfb3e0fbf2d9b9f06
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Marcelo-Tosatti/VMX-configure-posted-interrupt-descriptor-when-assigning-device-v5/20210525-215604
git checkout e515e68f330d3787af0952dcfb3e0fbf2d9b9f06
# save the attached .config to linux build tree
make W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>, old ones prefixed by <<):
>> ERROR: modpost: "kvm_make_all_cpus_request" [arch/x86/kvm/kvm-intel.ko] undefined!
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 41426 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch 0/3] VMX: configure posted interrupt descriptor when assigning device (v5)
2021-05-25 13:41 [patch 0/3] VMX: configure posted interrupt descriptor when assigning device (v5) Marcelo Tosatti
` (2 preceding siblings ...)
2021-05-25 13:41 ` [patch 3/3] KVM: VMX: update vcpu posted-interrupt descriptor when assigning device Marcelo Tosatti
@ 2021-05-27 11:59 ` Paolo Bonzini
3 siblings, 0 replies; 15+ messages in thread
From: Paolo Bonzini @ 2021-05-27 11:59 UTC (permalink / raw)
To: Marcelo Tosatti, kvm; +Cc: Alex Williamson, Sean Christopherson, Peter Xu
On 25/05/21 15:41, Marcelo Tosatti wrote:
> Configuration of the posted interrupt descriptor is incorrect when devices
> are hotplugged to the guest (and vcpus are halted).
>
> See patch 3 for details.
>
> ---
>
> v5: use KVM_REQ_UNBLOCK instead of kvm_check_block callback (Paolo / Peter Xu)
>
> v4: remove NULL assignments from kvm_x86_ops (Peter Xu)
> check for return value of ->start_assignment directly (Peter Xu)
>
> v3: improved comments (Sean)
> use kvm_vcpu_wake_up (Sean)
> drop device_count from start_assignment function (Peter Xu)
>
> v2: rather than using a potentially racy IPI (vs vcpu->cpu switches),
> kick the vcpus when assigning a device and let the blocked per-CPU
> list manipulation happen locally at ->pre_block and ->post_block
> (Sean Christopherson).
>
>
>
Queued, thanks.
Paolo
^ permalink raw reply [flat|nested] 15+ messages in thread