[PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
evtchn delivery set up on a per-vCPU basis via
HVMOP_set_evtchn_upcall_vector.

is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
physical IRQ to event channel mappings.

Signed-off-by: Jane Malalane <jane.malalane@citrix.com>
---
CC: Jan Beulich <jbeulich@suse.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: "Roger Pau Monné" <roger.pau@citrix.com>
CC: Wei Liu <wl@xen.org>
---
 xen/arch/x86/include/asm/domain.h   | 8 +++++++-
 xen/arch/x86/traps.c                | 3 +++
 xen/include/public/arch-x86/cpuid.h | 2 ++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
index 35898d725f..f044e0a492 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -14,8 +14,14 @@
 
 #define has_32bit_shinfo(d)    ((d)->arch.has_32bit_shinfo)
 
+/*
+ * Set to true if either the global vector-type callback or per-vCPU
+ * LAPIC vectors are used. Assume all vCPUs will use
+ * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
+ */
 #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
-        (d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
+        ((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
+         (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
 #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
 #define is_domain_direct_mapped(d) ((void)(d), 0)
 
diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index 25bffe47d7..2c51faab2c 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -1152,6 +1152,9 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,
         res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
         res->c = d->domain_id;
 
+        /* Per-vCPU event channel upcalls are implemented. */
+        res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
+
         break;
 
     case 5: /* PV-specific parameters */
diff --git a/xen/include/public/arch-x86/cpuid.h b/xen/include/public/arch-x86/cpuid.h
index f2b2b3632c..1760e2c405 100644
--- a/xen/include/public/arch-x86/cpuid.h
+++ b/xen/include/public/arch-x86/cpuid.h
@@ -109,6 +109,8 @@
  * field from 8 to 15 bits, allowing to target APIC IDs up 32768.
  */
 #define XEN_HVM_CPUID_EXT_DEST_ID      (1u << 5)
+/* Per-vCPU event channel upcalls. */
+#define XEN_HVM_CPUID_UPCALL_VECTOR    (1u << 6)
 
 /*
  * Leaf 6 (0x40000x05)
-- 
2.11.0

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Roger Pau Monné]

On Wed, May 11, 2022 at 04:14:23PM +0100, Jane Malalane wrote:
> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
> evtchn delivery set up on a per-vCPU basis via
> HVMOP_set_evtchn_upcall_vector.
> 
> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
> physical IRQ to event channel mappings.

I would add something like:

The naming of the CPUID bit is a bit generic about upcall support
being available.  That's done so that the define name doesn't get
overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
such.

Guests that don't care about physical interrupts routed over event
channels can just test for the availability of the hypercall directly
(HVMOP_set_evtchn_upcall_vector) without checking the CPUID bit.

> 
> Signed-off-by: Jane Malalane <jane.malalane@citrix.com>
> ---
> CC: Jan Beulich <jbeulich@suse.com>
> CC: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: "Roger Pau Monné" <roger.pau@citrix.com>
> CC: Wei Liu <wl@xen.org>
> ---
>  xen/arch/x86/include/asm/domain.h   | 8 +++++++-
>  xen/arch/x86/traps.c                | 3 +++
>  xen/include/public/arch-x86/cpuid.h | 2 ++
>  3 files changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
> index 35898d725f..f044e0a492 100644
> --- a/xen/arch/x86/include/asm/domain.h
> +++ b/xen/arch/x86/include/asm/domain.h
> @@ -14,8 +14,14 @@
>  
>  #define has_32bit_shinfo(d)    ((d)->arch.has_32bit_shinfo)
>  
> +/*
> + * Set to true if either the global vector-type callback or per-vCPU
> + * LAPIC vectors are used. Assume all vCPUs will use

I think you should remove LAPIC here.  There's no such thing as 'LAPIC
vectors', it's just that the old mechanism was bypassing the lapic
EOI.

> + * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
> + */
>  #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
> -        (d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
> +        ((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
> +         (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
>  #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
>  #define is_domain_direct_mapped(d) ((void)(d), 0)
>  
> diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
> index 25bffe47d7..2c51faab2c 100644
> --- a/xen/arch/x86/traps.c
> +++ b/xen/arch/x86/traps.c
> @@ -1152,6 +1152,9 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,
>          res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
>          res->c = d->domain_id;
>  
> +        /* Per-vCPU event channel upcalls are implemented. */

... are implemented and work correctly with PIRQs routed over event
channels.

> +        res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
> +
>          break;
>  
>      case 5: /* PV-specific parameters */
> diff --git a/xen/include/public/arch-x86/cpuid.h b/xen/include/public/arch-x86/cpuid.h
> index f2b2b3632c..1760e2c405 100644
> --- a/xen/include/public/arch-x86/cpuid.h
> +++ b/xen/include/public/arch-x86/cpuid.h
> @@ -109,6 +109,8 @@
>   * field from 8 to 15 bits, allowing to target APIC IDs up 32768.
>   */
>  #define XEN_HVM_CPUID_EXT_DEST_ID      (1u << 5)
> +/* Per-vCPU event channel upcalls. */

I would maybe expand the message to:

"Per-vCPU event channel upcalls work correctly with physical IRQs bound
to event channels."

Thanks, Roger.

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

On 13/05/2022 16:39, Roger Pau Monné wrote:
> On Wed, May 11, 2022 at 04:14:23PM +0100, Jane Malalane wrote:
>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>> evtchn delivery set up on a per-vCPU basis via
>> HVMOP_set_evtchn_upcall_vector.
>>
>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>> physical IRQ to event channel mappings.
> 
> I would add something like:
> 
> The naming of the CPUID bit is a bit generic about upcall support
> being available.  That's done so that the define name doesn't get
> overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
> such.
> 
> Guests that don't care about physical interrupts routed over event
> channels can just test for the availability of the hypercall directly
> (HVMOP_set_evtchn_upcall_vector) without checking the CPUID bit.
> 
>>
>> Signed-off-by: Jane Malalane <jane.malalane@citrix.com>
>> ---
>> CC: Jan Beulich <jbeulich@suse.com>
>> CC: Andrew Cooper <andrew.cooper3@citrix.com>
>> CC: "Roger Pau Monné" <roger.pau@citrix.com>
>> CC: Wei Liu <wl@xen.org>
>> ---
>>   xen/arch/x86/include/asm/domain.h   | 8 +++++++-
>>   xen/arch/x86/traps.c                | 3 +++
>>   xen/include/public/arch-x86/cpuid.h | 2 ++
>>   3 files changed, 12 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/domain.h
>> index 35898d725f..f044e0a492 100644
>> --- a/xen/arch/x86/include/asm/domain.h
>> +++ b/xen/arch/x86/include/asm/domain.h
>> @@ -14,8 +14,14 @@
>>   
>>   #define has_32bit_shinfo(d)    ((d)->arch.has_32bit_shinfo)
>>   
>> +/*
>> + * Set to true if either the global vector-type callback or per-vCPU
>> + * LAPIC vectors are used. Assume all vCPUs will use
> 
> I think you should remove LAPIC here.  There's no such thing as 'LAPIC
> vectors', it's just that the old mechanism was bypassing the lapic
> EOI.
> 
>> + * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
>> + */
>>   #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
>> -        (d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
>> +        ((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
>> +         (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
>>   #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
>>   #define is_domain_direct_mapped(d) ((void)(d), 0)
>>   
>> diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
>> index 25bffe47d7..2c51faab2c 100644
>> --- a/xen/arch/x86/traps.c
>> +++ b/xen/arch/x86/traps.c
>> @@ -1152,6 +1152,9 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, uint32_t leaf,
>>           res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
>>           res->c = d->domain_id;
>>   
>> +        /* Per-vCPU event channel upcalls are implemented. */
> 
> ... are implemented and work correctly with PIRQs routed over event
> channels.
> 
>> +        res->a |= XEN_HVM_CPUID_UPCALL_VECTOR;
>> +
>>           break;
>>   
>>       case 5: /* PV-specific parameters */
>> diff --git a/xen/include/public/arch-x86/cpuid.h b/xen/include/public/arch-x86/cpuid.h
>> index f2b2b3632c..1760e2c405 100644
>> --- a/xen/include/public/arch-x86/cpuid.h
>> +++ b/xen/include/public/arch-x86/cpuid.h
>> @@ -109,6 +109,8 @@
>>    * field from 8 to 15 bits, allowing to target APIC IDs up 32768.
>>    */
>>   #define XEN_HVM_CPUID_EXT_DEST_ID      (1u << 5)
>> +/* Per-vCPU event channel upcalls. */
> 
> I would maybe expand the message to:
> 
> "Per-vCPU event channel upcalls work correctly with physical IRQs bound
> to event channels."

Thanks. Yes, if others agree that the CPUID bit can still take this name 
if better explained, despite it being, as you say, quite generic, I will 
add these comments in a v2.

Thanks,

Jane.

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jan Beulich]

On 13.05.2022 17:39, Roger Pau Monné wrote:
> On Wed, May 11, 2022 at 04:14:23PM +0100, Jane Malalane wrote:
>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>> evtchn delivery set up on a per-vCPU basis via
>> HVMOP_set_evtchn_upcall_vector.
>>
>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>> physical IRQ to event channel mappings.
> 
> I would add something like:
> 
> The naming of the CPUID bit is a bit generic about upcall support
> being available.  That's done so that the define name doesn't get
> overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
> such.

On top of this at least half a sentence wants saying on why a new
CPUID bit is introduced in the first place. This doesn't derive in
any way from title or description. It would be only then when it
is additionally explained why the name was chosen like this.

Jan

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

On 18/05/2022 10:09, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
> 
> On 13.05.2022 17:39, Roger Pau Monné wrote:
>> On Wed, May 11, 2022 at 04:14:23PM +0100, Jane Malalane wrote:
>>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>>> evtchn delivery set up on a per-vCPU basis via
>>> HVMOP_set_evtchn_upcall_vector.
>>>
>>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>>> physical IRQ to event channel mappings.
>>
>> I would add something like:
>>
>> The naming of the CPUID bit is a bit generic about upcall support
>> being available.  That's done so that the define name doesn't get
>> overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
>> such.
> 
> On top of this at least half a sentence wants saying on why a new
> CPUID bit is introduced in the first place. This doesn't derive in
> any way from title or description. It would be only then when it
> is additionally explained why the name was chosen like this.Indeed it is incomplete, thanks for pointing that out.

I could add:
"A CPUID bit is added so that guests know whether the check
in is_hvm_pv_evtchn_domain() will fail when using
HVMOP_set_evtchn_upcall_vector. This matters for guests that route
PIRQs over event channels since is_hvm_pv_evtchn_domain() is a
condition in physdev_map_pirq()."

Would this be enough clarification?

Thank you,

Jane.

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jan Beulich]

On 18.05.2022 12:38, Jane Malalane wrote:
> On 18/05/2022 10:09, Jan Beulich wrote:
>> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
>>
>> On 13.05.2022 17:39, Roger Pau Monné wrote:
>>> On Wed, May 11, 2022 at 04:14:23PM +0100, Jane Malalane wrote:
>>>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>>>> evtchn delivery set up on a per-vCPU basis via
>>>> HVMOP_set_evtchn_upcall_vector.
>>>>
>>>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>>>> physical IRQ to event channel mappings.
>>>
>>> I would add something like:
>>>
>>> The naming of the CPUID bit is a bit generic about upcall support
>>> being available.  That's done so that the define name doesn't get
>>> overly long like XEN_HVM_CPUID_UPCALL_VECTOR_SUPPORTS_PIRQ or some
>>> such.
>>
>> On top of this at least half a sentence wants saying on why a new
>> CPUID bit is introduced in the first place. This doesn't derive in
>> any way from title or description. It would be only then when it
>> is additionally explained why the name was chosen like this.Indeed it is incomplete, thanks for pointing that out.
> 
> I could add:
> "A CPUID bit is added so that guests know whether the check
> in is_hvm_pv_evtchn_domain() will fail when using
> HVMOP_set_evtchn_upcall_vector. This matters for guests that route
> PIRQs over event channels since is_hvm_pv_evtchn_domain() is a
> condition in physdev_map_pirq()."
> 
> Would this be enough clarification?

Yes, thanks.

Jan

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jan Beulich]

On 11.05.2022 17:14, Jane Malalane wrote:
> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
> evtchn delivery set up on a per-vCPU basis via
> HVMOP_set_evtchn_upcall_vector.

I'm confused: You say "per-vCPU" here, but ...

> --- a/xen/arch/x86/include/asm/domain.h
> +++ b/xen/arch/x86/include/asm/domain.h
> @@ -14,8 +14,14 @@
>  
>  #define has_32bit_shinfo(d)    ((d)->arch.has_32bit_shinfo)
>  
> +/*
> + * Set to true if either the global vector-type callback or per-vCPU
> + * LAPIC vectors are used. Assume all vCPUs will use
> + * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
> + */
>  #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
> -        (d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
> +        ((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
> +         (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))

... you use (d)->vcpu[0] here (and, yes, you say so in the comment)
and ...

>  #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))

... you don't alter this at all.

Also (re-ordering context) this ...

> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
> physical IRQ to event channel mappings.

... isn't really true - it's is_hvm_pv_evtchn_domain() which controls
this (which in turn is why above you need to make the assumption I've
put under question). With that assumption I think is_hvm_pv_evtchn_vcpu()
would better go away.

Jan

Re: [PATCH] x86/hvm: Widen condition for is_hvm_pv_evtchn_vcpu()

[Jane Malalane]

On 18/05/2022 10:07, Jan Beulich wrote:
> [CAUTION - EXTERNAL EMAIL] DO NOT reply, click links, or open attachments unless you have verified the sender and know the content is safe.
> 
> On 11.05.2022 17:14, Jane Malalane wrote:
>> Have is_hvm_pv_evtchn_vcpu() return true for vector callbacks for
>> evtchn delivery set up on a per-vCPU basis via
>> HVMOP_set_evtchn_upcall_vector.
> 
> I'm confused: You say "per-vCPU" here, but ...
> 
>> --- a/xen/arch/x86/include/asm/domain.h
>> +++ b/xen/arch/x86/include/asm/domain.h
>> @@ -14,8 +14,14 @@
>>   
>>   #define has_32bit_shinfo(d)    ((d)->arch.has_32bit_shinfo)
>>   
>> +/*
>> + * Set to true if either the global vector-type callback or per-vCPU
>> + * LAPIC vectors are used. Assume all vCPUs will use
>> + * HVMOP_set_evtchn_upcall_vector as long as the initial vCPU does.
>> + */
>>   #define is_hvm_pv_evtchn_domain(d) (is_hvm_domain(d) && \
>> -        (d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector)
>> +        ((d)->arch.hvm.irq->callback_via_type == HVMIRQ_callback_vector || \
>> +         (d)->vcpu[0]->arch.hvm.evtchn_upcall_vector))
> 
> ... you use (d)->vcpu[0] here (and, yes, you say so in the comment)
> and ...
> 
>>   #define is_hvm_pv_evtchn_vcpu(v) (is_hvm_pv_evtchn_domain(v->domain))
> 
> ... you don't alter this at all.
> 
> Also (re-ordering context) this ...
> 
>> is_hvm_pv_evtchn_vcpu() returning true is a condition for setting up
>> physical IRQ to event channel mappings.
> 
> ... isn't really true - it's is_hvm_pv_evtchn_domain() which controls
> this (which in turn is why above you need to make the assumption I've
> put under question). With that assumption I think is_hvm_pv_evtchn_vcpu()
> would better go away.
Here only is_hvm_pv_evtchn_domain() should have been mentioned. The 
"per-VCPU" was in regard to the vector callback for evthcn delivery 
setup not being global but now done on a per-vCPU basis, in any case, I 
will amend the description and title. Thanks for the feedback.

Jane.