Re: [PATCH for-4.14 6/8] x86/vpt: fix injection to remote vCPU

From: Jan Beulich <jbeulich@suse.com>
To: Roger Pau Monne <roger.pau@citrix.com>
Cc: xen-devel@lists.xenproject.org,
	Andrew Cooper <andrew.cooper3@citrix.com>, Wei Liu <wl@xen.org>,
	paul@xen.org
Subject: Re: [PATCH for-4.14 6/8] x86/vpt: fix injection to remote vCPU
Date: Thu, 18 Jun 2020 17:12:17 +0200	[thread overview]
Message-ID: <57b6f9fd-4cbc-abc9-09e3-6493eba6c377@suse.com> (raw)
In-Reply-To: <20200612155640.4101-7-roger.pau@citrix.com>

On 12.06.2020 17:56, Roger Pau Monne wrote:
> vpt timers are usually added to the per-vCPU list of the vCPU where
> they get setup, but depending on the timer source type that vCPU might
> be different than the one where the interrupt vector gets injected.
> 
> For example the PIT timer use a PIC or IO-APIC pin in order to select
> the destination vCPU and vector, which might not match the vCPU they
> are configured from.
> 
> If such a situation happens pt_intr_post won't be called, and thus the
> vpt will be left in a limbo where the next interrupt won't be
> scheduled. Fix this by generalizing the special handling done to
> IO-APIC level interrupts to be applied always when the destination
> vCPU of the injected vector is different from the vCPU where the vpt
> belongs to (ie: usually the one it's been configured from).
> 
> A further improvement as noted in a comment added to the code might be
> to move the vpt so it's handled by the same vCPU where the vector gets
> injected.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
>  xen/arch/x86/hvm/vpt.c | 80 +++++++++++++++++++++---------------------
>  1 file changed, 40 insertions(+), 40 deletions(-)
> 
> diff --git a/xen/arch/x86/hvm/vpt.c b/xen/arch/x86/hvm/vpt.c
> index 6a975fc668..52ad5b90a7 100644
> --- a/xen/arch/x86/hvm/vpt.c
> +++ b/xen/arch/x86/hvm/vpt.c
> @@ -358,59 +358,59 @@ int pt_update_irq(struct vcpu *v)
>           * interrupt delivery case. Otherwise return -1 to do nothing.
>           */
>          vlapic_set_irq(vcpu_vlapic(v), irq, 0);
> -        pt_vector = irq;
> -        break;
> +        return irq;
>  
>      case PTSRC_isa:
>          hvm_isa_irq_deassert(v->domain, irq);
>          if ( platform_legacy_irq(irq) && vlapic_accept_pic_intr(v) &&
>               v->domain->arch.hvm.vpic[irq >> 3].int_output )
> -            hvm_isa_irq_assert(v->domain, irq, NULL);
> +            pt_vector = hvm_isa_irq_assert(v->domain, irq, NULL);
>          else
> -        {
>              pt_vector = hvm_isa_irq_assert(v->domain, irq, vioapic_get_vector);
> -            /*
> -             * hvm_isa_irq_assert may not set the corresponding bit in vIRR
> -             * when mask field of IOAPIC RTE is set. Check it again.
> -             */

For one, the transformation done here looks to call for folding
both calls to hvm_isa_irq_assert() into one. I'm not, however,
convinced recording the function's return value is useful in the
case where it wasn't recorded before. The change is benign right
now because hvm_isa_irq_assert() will return -1 when its last
argument is NULL, but the question is whether the code here should
start depending on such behavior.

And then, according to this comment (which doesn't get retained in
any form or shape) ...

> -            if ( pt_vector < 0 || !vlapic_test_irq(vcpu_vlapic(v), pt_vector) )
> -                pt_vector = -1;
> -        }
> +
> +        if ( pt_vector < 0 )
> +            return pt_vector;
> +
>          break;
>  
>      case PTSRC_ioapic:
>          pt_vector = hvm_ioapic_assert(v->domain, irq, level);
> -        if ( pt_vector < 0 || !vlapic_test_irq(vcpu_vlapic(v), pt_vector) )
> -        {
> -            pt_vector = -1;
> -            if ( level )
> +        if ( pt_vector < 0 )
> +            return pt_vector;
> +
> +        break;
> +    }
> +
> +    ASSERT(pt_vector >= 0);
> +    if ( !vlapic_test_irq(vcpu_vlapic(v), pt_vector) )
> +    {
> +        time_cb *cb = NULL;
> +        void *cb_priv;
> +
> +        /*
> +         * Vector has been injected to a different vCPU, call pt_irq_fired and
> +         * execute the callback, since the destination vCPU(s) won't call
> +         * pt_intr_post for it.

... this isn't the only reason to come here. Beyond what the comment
says there is the hvm_domain_use_pirq() check in assert_gsi() which
would similarly result in the IRR bit not observed set here. At the
very least these cases want mentioning; I have to admit that I'm not
entirely clear yet whether your handling is correct for both, or
whether the information needs to be propagated into here.

Also instead of ASSERT(pt_vector >= 0) would you pull the respective
if() out of the switch(), to also cover the case of a fall through
without hitting any of the explicitly handled cases, resulting in
pt_vector left at its initial value of -1?

> +         * TODO: move this vpt to one of the vCPUs where the vector gets
> +         * injected.
> +         */
> +        spin_lock(&v->arch.hvm.tm_lock);
> +        /* Make sure the timer is still on the list. */
> +        list_for_each_entry ( pt, &v->arch.hvm.tm_list, list )
> +            if ( pt == earliest_pt )
>              {
> -                /*
> -                 * Level interrupts are always asserted because the pin assert
> -                 * count is incremented regardless of whether the pin is masked
> -                 * or the vector latched in IRR, so also execute the callback
> -                 * associated with the timer.
> -                 */
> -                time_cb *cb = NULL;
> -                void *cb_priv;
> -
> -                spin_lock(&v->arch.hvm.tm_lock);
> -                /* Make sure the timer is still on the list. */
> -                list_for_each_entry ( pt, &v->arch.hvm.tm_list, list )
> -                    if ( pt == earliest_pt )
> -                    {
> -                        pt_irq_fired(v, pt);
> -                        cb = pt->cb;
> -                        cb_priv = pt->priv;
> -                        break;
> -                    }
> -                spin_unlock(&v->arch.hvm.tm_lock);
> -
> -                if ( cb != NULL )
> -                    cb(v, cb_priv);
> +                pt_irq_fired(v, pt);
> +                cb = pt->cb;
> +                cb_priv = pt->priv;
> +                break;
>              }
> -        }
> -        break;
> +        spin_unlock(&v->arch.hvm.tm_lock);
> +
> +        if ( cb != NULL )
> +            cb(v, cb_priv);
> +
> +        pt_vector = -1;
>      }
>  
>      return pt_vector;

To further reduce indentation (and seeing the significant code
churn that happens here anyway), could you consider inverting the
surrounding if() to

    if ( vlapic_test_irq(vcpu_vlapic(v), pt_vector) )
        return pt_vector;    

?

Jan