From: "Tian, Kevin" <kevin.tian@intel.com>
To: "Xu, Quan" <quan.xu@intel.com>,
"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Cc: "dario.faggioli@citrix.com" <dario.faggioli@citrix.com>,
"Wu, Feng" <feng.wu@intel.com>,
"jbeulich@suse.com" <jbeulich@suse.com>
Subject: Re: [PATCH v7 2/2] VT-d: Fix vt-d Device-TLB flush timeout issue
Date: Thu, 17 Mar 2016 08:17:11 +0000 [thread overview]
Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D15F7EC904@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: <1458198767-61293-3-git-send-email-quan.xu@intel.com>
> From: Xu, Quan
> Sent: Thursday, March 17, 2016 3:13 PM
> diff --git a/xen/drivers/passthrough/vtd/qinval.c b/xen/drivers/passthrough/vtd/qinval.c
> index 37a15fb..2a5c638 100644
> --- a/xen/drivers/passthrough/vtd/qinval.c
> +++ b/xen/drivers/passthrough/vtd/qinval.c
> @@ -233,6 +233,57 @@ int qinval_device_iotlb(struct iommu *iommu,
> return 0;
> }
>
> +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> + u16 seg, u8 bus, u8 devfn)
> +{
> + struct domain *d = NULL;
> + struct pci_dev *pdev;
> +
> + if ( test_bit(did, iommu->domid_bitmap) )
> + d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> +
> + if ( d == NULL )
> + return;
> +
> + pcidevs_lock();
> + for_each_pdev(d, pdev)
we need a 'safe' version here since you're deleting nodes
when walking list. for_each_pdev today is based on
list_for_each_entry. Or if it's sure that only one pdev
can match, we can break out of the loop to do removal.
> + {
> + if ( ( pdev->seg == seg ) &&
> + ( pdev->bus == bus ) &&
> + ( pdev->devfn == devfn ) )
> + {
> + ASSERT ( pdev->domain );
> + list_del(&pdev->domain_list);
> + pdev->domain = NULL;
> + pci_hide_existing_device(pdev);
> + break;
> + }
> + }
> +
> + pcidevs_unlock();
> +
> + if ( !is_hardware_domain(d) )
> + domain_crash(d);
> +
> + rcu_unlock_domain(d);
> +}
> +
> +int dev_invalidate_iotlb_sync(struct iommu *iommu, u16 did,
> + u16 seg, u8 bus, u8 devfn)
> +{
> + struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
> + int rc = 0;
> +
> + if ( qi_ctrl->qinval_maddr )
> + {
> + rc = queue_invalidate_wait(iommu, 0, 1, 1);
> + if ( rc == -ETIMEDOUT )
> + dev_invalidate_iotlb_timeout(iommu, did, seg, bus, devfn);
> + }
> +
> + return rc;
> +}
> +
Is this function a temporary one which will be removed later once we
can handle timeout for all types of flushes (at that time suppose this
logic will be reflected in invalidate_sync directly)?
> static void queue_invalidate_iec(struct iommu *iommu, u8 granu, u8 im, u16 iidx)
> {
> unsigned long flags;
> @@ -342,8 +393,6 @@ static int flush_iotlb_qi(
>
> if ( qi_ctrl->qinval_maddr != 0 )
> {
> - int rc;
> -
> /* use queued invalidation */
> if (cap_write_drain(iommu->cap))
> dw = 1;
> @@ -353,11 +402,17 @@ static int flush_iotlb_qi(
> queue_invalidate_iotlb(iommu,
> type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr,
> dw, did, size_order, 0, addr);
> +
> + /*
> + * Before Device-TLB invalidation we need to synchronize
> + * invalidation completions with hardware.
> + */
> + ret = invalidate_sync(iommu);
> + if ( ret )
> + return ret;
> +
> if ( flush_dev_iotlb )
> ret = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
> - rc = invalidate_sync(iommu);
> - if ( !ret )
> - ret = rc;
Current change looks not consistent. For IOMMU iotlb flush, we have
invalidate_sync out of invalidate operation, however below...
> }
> return ret;
> }
> diff --git a/xen/drivers/passthrough/vtd/x86/ats.c
> b/xen/drivers/passthrough/vtd/x86/ats.c
> index 334b9c1..c87ffe3 100644
> --- a/xen/drivers/passthrough/vtd/x86/ats.c
> +++ b/xen/drivers/passthrough/vtd/x86/ats.c
> @@ -162,6 +162,18 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
> return -EOPNOTSUPP;
> }
>
> + /*
> + * Synchronize with hardware for Device-TLB invalidate
> + * descriptor.
> + */
> + rc = dev_invalidate_iotlb_sync(iommu, did, pdev->seg,
> + pdev->bus, pdev->devfn);
> + if ( rc )
> + printk(XENLOG_ERR
> + "Flush error %d on device %04x:%02x:%02x.%u.\n",
> + ret, pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn),
> + PCI_FUNC(pdev->devfn));
> +
> if ( !ret )
for device iotlb flush, you moved the invalidate_sync inside the
invalidate operation.
If this is only temporary as I guessed earlier, is it clearer to change
like below:
> @@ -353,11 +402,17 @@ static int flush_iotlb_qi(
queue_invalidate_iotlb(iommu,
type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr,
dw, did, size_order, 0, addr);
/*
* Before Device-TLB invalidation we need to synchronize
* invalidation completions with hardware.
* TODO: timeout error handling to be added later
*/
ret = invalidate_sync(iommu);
if ( ret )
return ret;
if ( flush_dev_iotlb )
ret = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
rc = invalidate_sync(iommu);
if ( rc == -ETIMEDOUT )
dev_invalidate_iotlb_timeout(iommu, did, seg, bus, devfn);
if ( !ret )
ret = rc;
This way later when we have invalidate_sync handling timeout error
for all types of flushes, above two lines of timeout handling can be
removed.
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
next prev parent reply other threads:[~2016-03-17 8:17 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-17 7:12 [PATCH v7 0/2] VT-d Device-TLB flush issue Quan Xu
2016-03-17 7:12 ` [PATCH v7 1/2] VT-d: Reduce spin timeout to 1ms, which can be boot-time changed Quan Xu
2016-03-17 7:45 ` Tian, Kevin
2016-03-17 8:11 ` Xu, Quan
2016-03-17 8:13 ` Jan Beulich
2016-03-17 8:17 ` Xu, Quan
2016-03-17 7:12 ` [PATCH v7 2/2] VT-d: Fix vt-d Device-TLB flush timeout issue Quan Xu
2016-03-17 8:17 ` Tian, Kevin [this message]
2016-03-17 9:43 ` Jan Beulich
2016-03-17 11:13 ` Tian, Kevin
2016-03-17 11:30 ` Xu, Quan
2016-03-17 11:32 ` Tian, Kevin
2016-03-17 11:33 ` Jan Beulich
2016-03-18 12:21 ` Xu, Quan
2016-03-21 3:27 ` Tian, Kevin
2016-03-23 2:12 ` Xu, Quan
2016-03-23 3:29 ` Xu, Quan
2016-03-23 5:36 ` Tian, Kevin
2016-03-23 5:39 ` Xu, Quan
2016-03-18 11:18 ` Jan Beulich
2016-03-18 11:31 ` Xu, Quan
2016-03-18 13:35 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=AADFC41AFE54684AB9EE6CBC0274A5D15F7EC904@SHSMSX101.ccr.corp.intel.com \
--to=kevin.tian@intel.com \
--cc=dario.faggioli@citrix.com \
--cc=feng.wu@intel.com \
--cc=jbeulich@suse.com \
--cc=quan.xu@intel.com \
--cc=xen-devel@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).