Re: [PATCH v5 04/14] vpci: cancel pending map/unmap on vpci removal

["Roger Pau Monné" <roger.pau@citrix.com>]

On Thu, Nov 25, 2021 at 01:02:41PM +0200, Oleksandr Andrushchenko wrote:
> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> 
> When a vPCI is removed for a PCI device it is possible that we have
> scheduled a delayed work for map/unmap operations for that device.
> For example, the following scenario can illustrate the problem:
> 
> pci_physdev_op
>    pci_add_device
>        init_bars -> modify_bars -> defer_map -> raise_softirq(SCHEDULE_SOFTIRQ)
>    iommu_add_device <- FAILS
>    vpci_remove_device -> xfree(pdev->vpci)
> 
> leave_hypervisor_to_guest
>    vpci_process_pending: v->vpci.mem != NULL; v->vpci.pdev->vpci == NULL
> 
> For the hardware domain we continue execution as the worse that
> could happen is that MMIO mappings are left in place when the
> device has been deassigned.
> 
> For unprivileged domains that get a failure in the middle of a vPCI
> {un}map operation we need to destroy them, as we don't know in which
> state the p2m is. This can only happen in vpci_process_pending for
> DomUs as they won't be allowed to call pci_add_device.
> 
> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> 
> ---
> Cc: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Since v4:
>  - crash guest domain if map/unmap operation didn't succeed
>  - re-work vpci cancel work to cancel work on all vCPUs
>  - use new locking scheme with pdev->vpci_lock
> New in v4
> 
> Fixes: 86dbcf6e30cb ("vpci: cancel pending map/unmap on vpci removal")
> 
> ---
> 
> Signed-off-by: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com>
> ---
>  xen/drivers/vpci/header.c | 49 ++++++++++++++++++++++++++++++---------
>  xen/drivers/vpci/vpci.c   |  2 ++
>  xen/include/xen/pci.h     |  5 ++++
>  xen/include/xen/vpci.h    |  6 +++++
>  4 files changed, 51 insertions(+), 11 deletions(-)
> 
> diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c
> index bd23c0274d48..ba333fb2f9b0 100644
> --- a/xen/drivers/vpci/header.c
> +++ b/xen/drivers/vpci/header.c
> @@ -131,7 +131,13 @@ static void modify_decoding(const struct pci_dev *pdev, uint16_t cmd,
>  
>  bool vpci_process_pending(struct vcpu *v)
>  {
> -    if ( v->vpci.mem )
> +    struct pci_dev *pdev = v->vpci.pdev;
> +
> +    if ( !pdev )
> +        return false;
> +
> +    spin_lock(&pdev->vpci_lock);
> +    if ( !pdev->vpci_cancel_pending && v->vpci.mem )

Could you just check for pdev->vpci != NULL instead of having to add a
new vpci_cancel_pending field?

I also have a suggestion below which could make the code here simpler.

>      {
>          struct map_data data = {
>              .d = v->domain,
> @@ -140,32 +146,53 @@ bool vpci_process_pending(struct vcpu *v)
>          int rc = rangeset_consume_ranges(v->vpci.mem, map_range, &data);
>  
>          if ( rc == -ERESTART )
> +        {
> +            spin_unlock(&pdev->vpci_lock);
>              return true;
> +        }
>  
> -        spin_lock(&v->vpci.pdev->vpci_lock);
> -        if ( v->vpci.pdev->vpci )
> +        if ( pdev->vpci )
>              /* Disable memory decoding unconditionally on failure. */
> -            modify_decoding(v->vpci.pdev,
> +            modify_decoding(pdev,
>                              rc ? v->vpci.cmd & ~PCI_COMMAND_MEMORY : v->vpci.cmd,
>                              !rc && v->vpci.rom_only);
> -        spin_unlock(&v->vpci.pdev->vpci_lock);
>  
> -        rangeset_destroy(v->vpci.mem);
> -        v->vpci.mem = NULL;
>          if ( rc )
> +        {
>              /*
>               * FIXME: in case of failure remove the device from the domain.
>               * Note that there might still be leftover mappings. While this is
> -             * safe for Dom0, for DomUs the domain will likely need to be
> -             * killed in order to avoid leaking stale p2m mappings on
> -             * failure.
> +             * safe for Dom0, for DomUs the domain needs to be killed in order
> +             * to avoid leaking stale p2m mappings on failure.
>               */
> -            vpci_remove_device(v->vpci.pdev);
> +            if ( is_hardware_domain(v->domain) )
> +                vpci_remove_device_locked(pdev);
> +            else
> +                domain_crash(v->domain);
> +        }
>      }
> +    spin_unlock(&pdev->vpci_lock);
>  
>      return false;
>  }
>  
> +void vpci_cancel_pending_locked(struct pci_dev *pdev)
> +{
> +    struct vcpu *v;
> +
> +    ASSERT(spin_is_locked(&pdev->vpci_lock));
> +
> +    /* Cancel any pending work now on all vCPUs. */
> +    for_each_vcpu( pdev->domain, v )
> +    {
> +        if ( v->vpci.mem && (v->vpci.pdev == pdev) )

I'm unsure this is correct. You are protecting the access to
v->vpci.pdev with an expectation that v->vpci.pdev->vpci_lock is being
held.

I wonder if it would be better to just pause all the domain vCPUs and
then perform the cleaning of any pending operations. That would assure
that there are no changes to v->vpci. vpci_cancel_pending_locked
shouldn't be a frequent operation, so the overhead of pausing all
domain vCPUs here is likely fine.

Thanks, Roger.