xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v9 0/3] VT-d Device-TLB flush issue
@ 2016-04-01 14:47 Quan Xu
  2016-04-01 14:47 ` [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation Quan Xu
                   ` (2 more replies)
  0 siblings, 3 replies; 18+ messages in thread
From: Quan Xu @ 2016-04-01 14:47 UTC (permalink / raw)
  To: xen-devel; +Cc: kevin.tian, feng.wu, jbeulich, dario.faggioli, Quan Xu

This patches fix current timeout concern and also allow limited ATS support:

1. Add a command line parameter for Queued Invalidation

The command line parameter 'vtd_qi_timeout' specifies the
timeout of the VT-d Queued Invalidation in milliseconds. By
default, the timeout is 1ms, which can be boot-time changed.

Add a __must_check annotation. The followup patch titled
'VT-d IOTLB/Context/IEC flush issue' addresses the __mustcheck.
That is the other callers of this routine (two or three
levels up) ignore the return code. This patch does not
address this but the other does.

2. wrap a _sync version for all VT-d flush interfaces

The dev_invalidate_iotlb() scans ats_devices list to flush ATS devices,
and the invalidate_sync() is put after dev_invalidate_iotlb() to
synchronize with hardware for flush status. If we assign multiple
ATS devices to a domain, the flush status is about all these multiple
ATS devices. Once flush timeout expires, we couldn't find out which
one is the buggy ATS device.

Then, The invalidate_sync() variant (We need to pass down the device's
SBDF to hide the ATS device) is put within dev_invalidate_iotlb() to
synchronize for the flush status one by one. If flush timeout expires,
we could find out the buggy ATS device and hide it. However, for other
VT-d flush interfaces, the invalidate_sync() is still put after at present.
This is inconsistent.

So we wrap a _sync version for all VT-d flush interfaces. It simplifies
caller logic and makes code more readable as well.

3. Fix vt-d Device-TLB flush timeout issue

If Device-TLB flush timed out, we would hide the target ATS
device and crash the domain owning this ATS device. If impacted
domain is hardware domain, just throw out a warning (done in
queue_invalidate_wait).

By hiding the device, we make sure it can't be assigned to
any domain any longer (see device_assigned).

**NOTE**
   This patch set should base on one prereq patch sets:
    a). Check VT-d Device-TLB flush error.

----
Not covered in this series:

    a) Eliminate the panic() in IOMMU_WAIT_OP, used only in VT-d register read/write.
       Further discussion is required on whether and how to improve it.
    b) Handle IOTLB/Context/IEC flush timeout.


--Changes in v9:
#patch 1
   * Enhance the commit message and docs/misc/xen-command-line.markdown.
#patch 2
   * Enhance the commit message.
   * Add function declaration at the head of file, and then we don't need pure code movement.
#patch 3
   * Enhance the commit message.
   * 'ASSERT ( pdev->domain )' to 'ASSERT(pdev->domain)'


Quan Xu (3):
  VT-d: add a command line parameter for Queued Invalidation
  VT-d: wrap a _sync version for all VT-d flush interfaces
  VT-d: Fix vt-d Device-TLB flush timeout issue

 docs/misc/xen-command-line.markdown   |  18 +++++
 xen/drivers/passthrough/pci.c         |   6 +-
 xen/drivers/passthrough/vtd/extern.h  |   3 +
 xen/drivers/passthrough/vtd/qinval.c  | 131 ++++++++++++++++++++++++++++------
 xen/drivers/passthrough/vtd/x86/ats.c |  15 ++--
 xen/include/xen/pci.h                 |   1 +
 6 files changed, 141 insertions(+), 33 deletions(-)

-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation
  2016-04-01 14:47 [PATCH v9 0/3] VT-d Device-TLB flush issue Quan Xu
@ 2016-04-01 14:47 ` Quan Xu
  2016-04-05  9:09   ` Jan Beulich
  2016-04-01 14:47 ` [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces Quan Xu
  2016-04-01 14:47 ` [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue Quan Xu
  2 siblings, 1 reply; 18+ messages in thread
From: Quan Xu @ 2016-04-01 14:47 UTC (permalink / raw)
  To: xen-devel; +Cc: kevin.tian, feng.wu, jbeulich, dario.faggioli, Quan Xu

The command line parameter 'vtd_qi_timeout' specifies the
timeout of the VT-d Queued Invalidation in milliseconds. By
default, the timeout is 1ms, which can be boot-time changed.

Add a __must_check annotation. The followup patch titled
'VT-d IOTLB/Context/IEC flush issue' addresses the __mustcheck.
That is the other callers of this routine (two or three
levels up) ignore the return code. This patch does not
address this but the other does.

Signed-off-by: Quan Xu <quan.xu@intel.com>
---
 docs/misc/xen-command-line.markdown  | 18 ++++++++++++++++++
 xen/drivers/passthrough/vtd/qinval.c | 17 ++++++++++++-----
 2 files changed, 30 insertions(+), 5 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index ca77e3b..5a7ed5d 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1532,6 +1532,24 @@ Note that if **watchdog** option is also specified vpmu will be turned off.
 As the virtualisation is not 100% safe, don't use the vpmu flag on
 production systems (see http://xenbits.xen.org/xsa/advisory-163.html)!
 
+### vtd\_qi\_timeout (VT-d)
+> `= <integer>`
+
+> Default: `1`
+
+Specify the timeout of the VT-d Queued Invalidation in milliseconds.
+By default, the spin timeout is 1ms, which can be boot-time changed.
+
+In current code, VT-d Queued Invalidation includes Device-TLB, IOTLB,
+Context and IEC flush. If Device-TLB flush timed out, we would hide
+the target ATS device and crash the domain owning this ATS device.
+If impacted domain is hardware domain, just throw out a warning (done
+in queue\_invalidate\_wait). IOTLB, Context and IEC flush timeout are
+still in TODO-list.
+
+When you see error 'Queue invalidate wait descriptor timed out', try
+increasing the vtd\_qi\_timeout to 10ms or more.
+
 ### watchdog
 > `= force | <boolean>`
 
diff --git a/xen/drivers/passthrough/vtd/qinval.c b/xen/drivers/passthrough/vtd/qinval.c
index b81b0bd..52ba2c2 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -28,6 +28,11 @@
 #include "vtd.h"
 #include "extern.h"
 
+static unsigned int __read_mostly vtd_qi_timeout = 1;
+integer_param("vtd_qi_timeout", vtd_qi_timeout);
+
+#define IOMMU_QI_TIMEOUT (vtd_qi_timeout * MILLISECS(1))
+
 static void print_qi_regs(struct iommu *iommu)
 {
     u64 val;
@@ -130,10 +135,10 @@ static void queue_invalidate_iotlb(struct iommu *iommu,
     spin_unlock_irqrestore(&iommu->register_lock, flags);
 }
 
-static int queue_invalidate_wait(struct iommu *iommu,
+static int __must_check queue_invalidate_wait(struct iommu *iommu,
     u8 iflag, u8 sw, u8 fn)
 {
-    s_time_t start_time;
+    s_time_t timeout;
     volatile u32 poll_slot = QINVAL_STAT_INIT;
     unsigned int index;
     unsigned long flags;
@@ -164,13 +169,15 @@ static int queue_invalidate_wait(struct iommu *iommu,
     if ( sw )
     {
         /* In case all wait descriptor writes to same addr with same data */
-        start_time = NOW();
+        timeout = NOW() + IOMMU_QI_TIMEOUT;
         while ( poll_slot != QINVAL_STAT_DONE )
         {
-            if ( NOW() > (start_time + DMAR_OPERATION_TIMEOUT) )
+            if ( NOW() > timeout )
             {
                 print_qi_regs(iommu);
-                panic("queue invalidate wait descriptor was not executed");
+                printk(XENLOG_WARNING VTDPREFIX
+                       "Queue invalidate wait descriptor timed out.\n");
+                return -ETIMEDOUT;
             }
             cpu_relax();
         }
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-01 14:47 [PATCH v9 0/3] VT-d Device-TLB flush issue Quan Xu
  2016-04-01 14:47 ` [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation Quan Xu
@ 2016-04-01 14:47 ` Quan Xu
  2016-04-05  9:35   ` Jan Beulich
  2016-04-01 14:47 ` [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue Quan Xu
  2 siblings, 1 reply; 18+ messages in thread
From: Quan Xu @ 2016-04-01 14:47 UTC (permalink / raw)
  To: xen-devel; +Cc: kevin.tian, feng.wu, jbeulich, dario.faggioli, Quan Xu

The dev_invalidate_iotlb() scans ats_devices list to flush ATS devices,
and the invalidate_sync() is put after dev_invalidate_iotlb() to
synchronize with hardware for flush status. If we assign multiple
ATS devices to a domain, the flush status is about all these multiple
ATS devices. Once flush timeout expires, we couldn't find out which
one is the buggy ATS device.

Then, The invalidate_sync() variant (We need to pass down the device's
SBDF to hide the ATS device) is put within dev_invalidate_iotlb() to
synchronize for the flush status one by one. If flush timeout expires,
we could find out the buggy ATS device and hide it. However, for other
VT-d flush interfaces, the invalidate_sync() is still put after at present.
This is inconsistent.

So we wrap a _sync version for all VT-d flush interfaces. It simplifies
caller logic and makes code more readable as well.

Signed-off-by: Quan Xu <quan.xu@intel.com>
---
 xen/drivers/passthrough/vtd/extern.h  |  2 ++
 xen/drivers/passthrough/vtd/qinval.c  | 60 +++++++++++++++++++++++++----------
 xen/drivers/passthrough/vtd/x86/ats.c | 12 +++----
 3 files changed, 50 insertions(+), 24 deletions(-)

diff --git a/xen/drivers/passthrough/vtd/extern.h b/xen/drivers/passthrough/vtd/extern.h
index d4d37c3..6d3187d 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -61,6 +61,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
 
 int qinval_device_iotlb(struct iommu *iommu,
                         u32 max_invs_pend, u16 sid, u16 size, u64 addr);
+int qinval_device_iotlb_sync(struct iommu *iommu, u32 max_invs_pend,
+                             u16 sid, u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c b/xen/drivers/passthrough/vtd/qinval.c
index 52ba2c2..d12661b 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -33,6 +33,10 @@ integer_param("vtd_qi_timeout", vtd_qi_timeout);
 
 #define IOMMU_QI_TIMEOUT (vtd_qi_timeout * MILLISECS(1))
 
+static int __must_check queue_invalidate_wait(struct iommu *iommu,
+    u8 iflag, u8 sw, u8 fn);
+static int invalidate_sync(struct iommu *iommu);
+
 static void print_qi_regs(struct iommu *iommu)
 {
     u64 val;
@@ -102,6 +106,15 @@ static void queue_invalidate_context(struct iommu *iommu,
     unmap_vtd_domain_page(qinval_entries);
 }
 
+static int queue_invalidate_context_sync(struct iommu *iommu,
+    u16 did, u16 source_id, u8 function_mask, u8 granu)
+{
+    queue_invalidate_context(iommu, did, source_id,
+                             function_mask, granu);
+
+    return invalidate_sync(iommu);
+}
+
 static void queue_invalidate_iotlb(struct iommu *iommu,
     u8 granu, u8 dr, u8 dw, u16 did, u8 am, u8 ih, u64 addr)
 {
@@ -135,6 +148,14 @@ static void queue_invalidate_iotlb(struct iommu *iommu,
     spin_unlock_irqrestore(&iommu->register_lock, flags);
 }
 
+static int queue_invalidate_iotlb_sync(struct iommu *iommu,
+    u8 granu, u8 dr, u8 dw, u16 did, u8 am, u8 ih, u64 addr)
+{
+    queue_invalidate_iotlb(iommu, granu, dr, dw, did, am, ih, addr);
+
+    return invalidate_sync(iommu);
+}
+
 static int __must_check queue_invalidate_wait(struct iommu *iommu,
     u8 iflag, u8 sw, u8 fn)
 {
@@ -229,6 +250,14 @@ int qinval_device_iotlb(struct iommu *iommu,
     return 0;
 }
 
+int qinval_device_iotlb_sync(struct iommu *iommu,
+    u32 max_invs_pend, u16 sid, u16 size, u64 addr)
+{
+    qinval_device_iotlb(iommu, max_invs_pend, sid, size, addr);
+
+    return invalidate_sync(iommu);
+}
+
 static void queue_invalidate_iec(struct iommu *iommu, u8 granu, u8 im, u16 iidx)
 {
     unsigned long flags;
@@ -256,7 +285,7 @@ static void queue_invalidate_iec(struct iommu *iommu, u8 granu, u8 im, u16 iidx)
     spin_unlock_irqrestore(&iommu->register_lock, flags);
 }
 
-static int __iommu_flush_iec(struct iommu *iommu, u8 granu, u8 im, u16 iidx)
+static int queue_invalidate_iec_sync(struct iommu *iommu, u8 granu, u8 im, u16 iidx)
 {
     int ret;
 
@@ -273,12 +302,12 @@ static int __iommu_flush_iec(struct iommu *iommu, u8 granu, u8 im, u16 iidx)
 
 int iommu_flush_iec_global(struct iommu *iommu)
 {
-    return __iommu_flush_iec(iommu, IEC_GLOBAL_INVL, 0, 0);
+    return queue_invalidate_iec_sync(iommu, IEC_GLOBAL_INVL, 0, 0);
 }
 
 int iommu_flush_iec_index(struct iommu *iommu, u8 im, u16 iidx)
 {
-   return __iommu_flush_iec(iommu, IEC_INDEX_INVL, im, iidx);
+   return queue_invalidate_iec_sync(iommu, IEC_INDEX_INVL, im, iidx);
 }
 
 static int flush_context_qi(
@@ -304,11 +333,9 @@ static int flush_context_qi(
     }
 
     if ( qi_ctrl->qinval_maddr != 0 )
-    {
-        queue_invalidate_context(iommu, did, sid, fm,
-                                 type >> DMA_CCMD_INVL_GRANU_OFFSET);
-        ret = invalidate_sync(iommu);
-    }
+        ret = queue_invalidate_context_sync(iommu, did, sid, fm,
+                  type >> DMA_CCMD_INVL_GRANU_OFFSET);
+
     return ret;
 }
 
@@ -338,23 +365,24 @@ static int flush_iotlb_qi(
 
     if ( qi_ctrl->qinval_maddr != 0 )
     {
-        int rc;
-
         /* use queued invalidation */
         if (cap_write_drain(iommu->cap))
             dw = 1;
         if (cap_read_drain(iommu->cap))
             dr = 1;
         /* Need to conside the ih bit later */
-        queue_invalidate_iotlb(iommu,
-                               type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr,
-                               dw, did, size_order, 0, addr);
+        ret = queue_invalidate_iotlb_sync(iommu,
+                  type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr, dw, did,
+                  size_order, 0, addr);
+
+        /* TODO: Timeout error handling to be added later */
+        if ( ret )
+            return ret;
+
         if ( flush_dev_iotlb )
             ret = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
-        rc = invalidate_sync(iommu);
-        if ( !ret )
-            ret = rc;
     }
+
     return ret;
 }
 
diff --git a/xen/drivers/passthrough/vtd/x86/ats.c b/xen/drivers/passthrough/vtd/x86/ats.c
index 334b9c1..7b1c07b 100644
--- a/xen/drivers/passthrough/vtd/x86/ats.c
+++ b/xen/drivers/passthrough/vtd/x86/ats.c
@@ -118,7 +118,6 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
     {
         u16 sid = PCI_BDF2(pdev->bus, pdev->devfn);
         bool_t sbit;
-        int rc = 0;
 
         /* Only invalidate devices that belong to this IOMMU */
         if ( pdev->iommu != iommu )
@@ -134,8 +133,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
             /* invalidate all translations: sbit=1,bit_63=0,bit[62:12]=1 */
             sbit = 1;
             addr = (~0UL << PAGE_SHIFT_4K) & 0x7FFFFFFFFFFFFFFF;
-            rc = qinval_device_iotlb(iommu, pdev->ats_queue_depth,
-                                     sid, sbit, addr);
+            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
+                                           sid, sbit, addr);
             break;
         case DMA_TLB_PSI_FLUSH:
             if ( !device_in_domain(iommu, pdev, did) )
@@ -154,16 +153,13 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
                 addr |= (((u64)1 << (size_order - 1)) - 1) << PAGE_SHIFT_4K;
             }
 
-            rc = qinval_device_iotlb(iommu, pdev->ats_queue_depth,
-                                     sid, sbit, addr);
+            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
+                                           sid, sbit, addr);
             break;
         default:
             dprintk(XENLOG_WARNING VTDPREFIX, "invalid vt-d flush type\n");
             return -EOPNOTSUPP;
         }
-
-        if ( !ret )
-            ret = rc;
     }
 
     return ret;
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue
  2016-04-01 14:47 [PATCH v9 0/3] VT-d Device-TLB flush issue Quan Xu
  2016-04-01 14:47 ` [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation Quan Xu
  2016-04-01 14:47 ` [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces Quan Xu
@ 2016-04-01 14:47 ` Quan Xu
  2016-04-05  9:47   ` Jan Beulich
  2 siblings, 1 reply; 18+ messages in thread
From: Quan Xu @ 2016-04-01 14:47 UTC (permalink / raw)
  To: xen-devel; +Cc: kevin.tian, feng.wu, jbeulich, dario.faggioli, Quan Xu

If Device-TLB flush timed out, we would hide the target ATS
device and crash the domain owning this ATS device. If impacted
domain is hardware domain, just throw out a warning (done in
queue_invalidate_wait).

By hiding the device, we make sure it can't be assigned to
any domain any longer (see device_assigned).

Signed-off-by: Quan Xu <quan.xu@intel.com>
---
 xen/drivers/passthrough/pci.c         |  6 ++--
 xen/drivers/passthrough/vtd/extern.h  |  3 +-
 xen/drivers/passthrough/vtd/qinval.c  | 58 +++++++++++++++++++++++++++++++++--
 xen/drivers/passthrough/vtd/x86/ats.c | 11 ++++---
 xen/include/xen/pci.h                 |  1 +
 5 files changed, 68 insertions(+), 11 deletions(-)

diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index 9f1716a..9a214c6 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -420,7 +420,7 @@ static void free_pdev(struct pci_seg *pseg, struct pci_dev *pdev)
     xfree(pdev);
 }
 
-static void _pci_hide_device(struct pci_dev *pdev)
+void pci_hide_existing_device(struct pci_dev *pdev)
 {
     if ( pdev->domain )
         return;
@@ -437,7 +437,7 @@ int __init pci_hide_device(int bus, int devfn)
     pdev = alloc_pdev(get_pseg(0), bus, devfn);
     if ( pdev )
     {
-        _pci_hide_device(pdev);
+        pci_hide_existing_device(pdev);
         rc = 0;
     }
     pcidevs_unlock();
@@ -467,7 +467,7 @@ int __init pci_ro_device(int seg, int bus, int devfn)
     }
 
     __set_bit(PCI_BDF2(bus, devfn), pseg->ro_map);
-    _pci_hide_device(pdev);
+    pci_hide_existing_device(pdev);
 
     return 0;
 }
diff --git a/xen/drivers/passthrough/vtd/extern.h b/xen/drivers/passthrough/vtd/extern.h
index 6d3187d..94e2c11 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -62,7 +62,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
 int qinval_device_iotlb(struct iommu *iommu,
                         u32 max_invs_pend, u16 sid, u16 size, u64 addr);
 int qinval_device_iotlb_sync(struct iommu *iommu, u32 max_invs_pend,
-                             u16 sid, u16 size, u64 addr);
+                             u16 did, u16 seg, u8 bus, u8 devfn,
+                             u16 size, u64 addr);
 
 unsigned int get_cache_line_size(void);
 void cacheline_flush(char *);
diff --git a/xen/drivers/passthrough/vtd/qinval.c b/xen/drivers/passthrough/vtd/qinval.c
index d12661b..2c54dc0 100644
--- a/xen/drivers/passthrough/vtd/qinval.c
+++ b/xen/drivers/passthrough/vtd/qinval.c
@@ -217,6 +217,58 @@ static int invalidate_sync(struct iommu *iommu)
     return 0;
 }
 
+static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
+                                         u16 seg, u8 bus, u8 devfn)
+{
+    struct domain *d = NULL;
+    struct pci_dev *pdev;
+
+    if ( test_bit(did, iommu->domid_bitmap) )
+        d = rcu_lock_domain_by_id(iommu->domid_map[did]);
+
+    if ( d == NULL )
+        return;
+
+    pcidevs_lock();
+
+    for_each_pdev(d, pdev)
+    {
+        if ( ( pdev->seg == seg ) &&
+             ( pdev->bus == bus ) &&
+             ( pdev->devfn == devfn ) )
+        {
+            ASSERT(pdev->domain);
+            list_del(&pdev->domain_list);
+            pdev->domain = NULL;
+            pci_hide_existing_device(pdev);
+            break;
+        }
+    }
+
+    pcidevs_unlock();
+
+    if ( !is_hardware_domain(d) )
+        domain_crash(d);
+
+    rcu_unlock_domain(d);
+}
+
+int dev_invalidate_sync(struct iommu *iommu, u16 did,
+                        u16 seg, u8 bus, u8 devfn)
+{
+    struct qi_ctrl *qi_ctrl = iommu_qi_ctrl(iommu);
+    int rc = 0;
+
+    if ( qi_ctrl->qinval_maddr )
+    {
+        rc = queue_invalidate_wait(iommu, 0, 1, 1);
+        if ( rc == -ETIMEDOUT )
+            dev_invalidate_iotlb_timeout(iommu, did, seg, bus, devfn);
+    }
+
+    return rc;
+}
+
 int qinval_device_iotlb(struct iommu *iommu,
     u32 max_invs_pend, u16 sid, u16 size, u64 addr)
 {
@@ -251,11 +303,13 @@ int qinval_device_iotlb(struct iommu *iommu,
 }
 
 int qinval_device_iotlb_sync(struct iommu *iommu,
-    u32 max_invs_pend, u16 sid, u16 size, u64 addr)
+    u32 max_invs_pend, u16 did, u16 seg, u8 bus, u8 devfn, u16 size, u64 addr)
 {
+    u16 sid = PCI_BDF2(bus, devfn);
+
     qinval_device_iotlb(iommu, max_invs_pend, sid, size, addr);
 
-    return invalidate_sync(iommu);
+    return dev_invalidate_sync(iommu, did, seg, bus, devfn);
 }
 
 static void queue_invalidate_iec(struct iommu *iommu, u8 granu, u8 im, u16 iidx)
diff --git a/xen/drivers/passthrough/vtd/x86/ats.c b/xen/drivers/passthrough/vtd/x86/ats.c
index 7b1c07b..5d1ebea 100644
--- a/xen/drivers/passthrough/vtd/x86/ats.c
+++ b/xen/drivers/passthrough/vtd/x86/ats.c
@@ -116,7 +116,6 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
 
     list_for_each_entry( pdev, &ats_devices, list )
     {
-        u16 sid = PCI_BDF2(pdev->bus, pdev->devfn);
         bool_t sbit;
 
         /* Only invalidate devices that belong to this IOMMU */
@@ -133,8 +132,9 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
             /* invalidate all translations: sbit=1,bit_63=0,bit[62:12]=1 */
             sbit = 1;
             addr = (~0UL << PAGE_SHIFT_4K) & 0x7FFFFFFFFFFFFFFF;
-            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
-                                           sid, sbit, addr);
+            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth, did,
+                                           pdev->seg, pdev->bus, pdev->devfn,
+                                           sbit, addr);
             break;
         case DMA_TLB_PSI_FLUSH:
             if ( !device_in_domain(iommu, pdev, did) )
@@ -153,8 +153,9 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
                 addr |= (((u64)1 << (size_order - 1)) - 1) << PAGE_SHIFT_4K;
             }
 
-            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
-                                           sid, sbit, addr);
+            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth, did,
+                                           pdev->seg, pdev->bus, pdev->devfn,
+                                           sbit, addr);
             break;
         default:
             dprintk(XENLOG_WARNING VTDPREFIX, "invalid vt-d flush type\n");
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index 6ed29dd..bb9f791 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -116,6 +116,7 @@ const unsigned long *pci_get_ro_map(u16 seg);
 int pci_add_device(u16 seg, u8 bus, u8 devfn,
                    const struct pci_dev_info *, nodeid_t node);
 int pci_remove_device(u16 seg, u8 bus, u8 devfn);
+void pci_hide_existing_device(struct pci_dev *pdev);
 int pci_ro_device(int seg, int bus, int devfn);
 int pci_hide_device(int bus, int devfn);
 struct pci_dev *pci_get_pdev(int seg, int bus, int devfn);
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation
  2016-04-01 14:47 ` [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation Quan Xu
@ 2016-04-05  9:09   ` Jan Beulich
  2016-04-07  1:49     ` Xu, Quan
  0 siblings, 1 reply; 18+ messages in thread
From: Jan Beulich @ 2016-04-05  9:09 UTC (permalink / raw)
  To: Quan Xu; +Cc: dario.faggioli, feng.wu, kevin.tian, xen-devel

>>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:

The subject should mention "timeout", perhaps either in addition to or
in place of "command line".

> --- a/docs/misc/xen-command-line.markdown
> +++ b/docs/misc/xen-command-line.markdown
> @@ -1532,6 +1532,24 @@ Note that if **watchdog** option is also specified vpmu will be turned off.
>  As the virtualisation is not 100% safe, don't use the vpmu flag on
>  production systems (see http://xenbits.xen.org/xsa/advisory-163.html)! 
>  
> +### vtd\_qi\_timeout (VT-d)
> +> `= <integer>`
> +
> +> Default: `1`
> +
> +Specify the timeout of the VT-d Queued Invalidation in milliseconds.
> +By default, the spin timeout is 1ms, which can be boot-time changed.

Especially the part after the comma makes little sense considering
which file we're in.

> +In current code, VT-d Queued Invalidation includes Device-TLB, IOTLB,
> +Context and IEC flush. If Device-TLB flush timed out, we would hide
> +the target ATS device and crash the domain owning this ATS device.
> +If impacted domain is hardware domain, just throw out a warning (done
> +in queue\_invalidate\_wait). IOTLB, Context and IEC flush timeout are
> +still in TODO-list.

Much of this doesn't seem to belong here either.

> +When you see error 'Queue invalidate wait descriptor timed out', try
> +increasing the vtd\_qi\_timeout to 10ms or more.

Why 10ms? (If there's no specific reason, I think you'd better
drop any explicit number.) Also there's no reason the spell out the
command line option again here - the context makes clear which
value needs increasing.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-01 14:47 ` [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces Quan Xu
@ 2016-04-05  9:35   ` Jan Beulich
  2016-04-07  7:44     ` Xu, Quan
  0 siblings, 1 reply; 18+ messages in thread
From: Jan Beulich @ 2016-04-05  9:35 UTC (permalink / raw)
  To: Quan Xu; +Cc: dario.faggioli, feng.wu, kevin.tian, xen-devel

>>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> The dev_invalidate_iotlb() scans ats_devices list to flush ATS devices,
> and the invalidate_sync() is put after dev_invalidate_iotlb() to
> synchronize with hardware for flush status. If we assign multiple
> ATS devices to a domain, the flush status is about all these multiple
> ATS devices. Once flush timeout expires, we couldn't find out which
> one is the buggy ATS device.

Is that true? Or is that just a limitation of our implementation?

> Then, The invalidate_sync() variant (We need to pass down the device's
> SBDF to hide the ATS device) is put within dev_invalidate_iotlb() to
> synchronize for the flush status one by one.

I don't think this is stating current state of things. So ...

> If flush timeout expires,
> we could find out the buggy ATS device and hide it. However, for other
> VT-d flush interfaces, the invalidate_sync() is still put after at present.
> This is inconsistent.

... taken together, what is inconsistent here needs to be
described better, as well as what it is you do to eliminate the
inconsistency. Note that you should not refer back (or imply
knowledge of) the previous discussion on the earlier version.
In any of that discussion is useful here, you need to summarize
it instead.

> --- a/xen/drivers/passthrough/vtd/extern.h
> +++ b/xen/drivers/passthrough/vtd/extern.h
> @@ -61,6 +61,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
>  
>  int qinval_device_iotlb(struct iommu *iommu,
>                          u32 max_invs_pend, u16 sid, u16 size, u64 addr);
> +int qinval_device_iotlb_sync(struct iommu *iommu, u32 max_invs_pend,
> +                             u16 sid, u16 size, u64 addr);

So are then both functions needed to be externally accessible?
That would seem contrary to the last paragraph of the patch
description.

> --- a/xen/drivers/passthrough/vtd/qinval.c
> +++ b/xen/drivers/passthrough/vtd/qinval.c
> @@ -33,6 +33,10 @@ integer_param("vtd_qi_timeout", vtd_qi_timeout);
>  
>  #define IOMMU_QI_TIMEOUT (vtd_qi_timeout * MILLISECS(1))
>  
> +static int __must_check queue_invalidate_wait(struct iommu *iommu,
> +    u8 iflag, u8 sw, u8 fn);

Indentation.

> @@ -102,6 +106,15 @@ static void queue_invalidate_context(struct iommu *iommu,
>      unmap_vtd_domain_page(qinval_entries);
>  }
>  
> +static int queue_invalidate_context_sync(struct iommu *iommu,

__must_check?

> +    u16 did, u16 source_id, u8 function_mask, u8 granu)

Indentation again.

> +{
> +    queue_invalidate_context(iommu, did, source_id,
> +                             function_mask, granu);
> +
> +    return invalidate_sync(iommu);
> +}

Further down you replace the only call to
queue_invalidate_context() - why keep both functions instead of
just making the existing one do the sync? (That would the likely
also apply to qinval_device_iotlb() and others below.)

> @@ -338,23 +365,24 @@ static int flush_iotlb_qi(
>  
>      if ( qi_ctrl->qinval_maddr != 0 )
>      {
> -        int rc;
> -
>          /* use queued invalidation */
>          if (cap_write_drain(iommu->cap))
>              dw = 1;
>          if (cap_read_drain(iommu->cap))
>              dr = 1;
>          /* Need to conside the ih bit later */
> -        queue_invalidate_iotlb(iommu,
> -                               type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr,
> -                               dw, did, size_order, 0, addr);
> +        ret = queue_invalidate_iotlb_sync(iommu,
> +                  type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr, dw, did,
> +                  size_order, 0, addr);
> +
> +        /* TODO: Timeout error handling to be added later */

As of today queue_invalidate_wait() panics, so this comment is
not very helpful as there is not timeout that could possibly be
detected here.

> +        if ( ret )
> +            return ret;
> +
>          if ( flush_dev_iotlb )
>              ret = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
> -        rc = invalidate_sync(iommu);
> -        if ( !ret )
> -            ret = rc;
>      }

I think leaving the existing logic as is would be better - best effort
invalidation even when an error has occurred.

> --- a/xen/drivers/passthrough/vtd/x86/ats.c
> +++ b/xen/drivers/passthrough/vtd/x86/ats.c
> @@ -118,7 +118,6 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
>      {
>          u16 sid = PCI_BDF2(pdev->bus, pdev->devfn);
>          bool_t sbit;
> -        int rc = 0;
>  
>          /* Only invalidate devices that belong to this IOMMU */
>          if ( pdev->iommu != iommu )
> @@ -134,8 +133,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
>              /* invalidate all translations: sbit=1,bit_63=0,bit[62:12]=1 */
>              sbit = 1;
>              addr = (~0UL << PAGE_SHIFT_4K) & 0x7FFFFFFFFFFFFFFF;
> -            rc = qinval_device_iotlb(iommu, pdev->ats_queue_depth,
> -                                     sid, sbit, addr);
> +            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> +                                           sid, sbit, addr);
>              break;
>          case DMA_TLB_PSI_FLUSH:
>              if ( !device_in_domain(iommu, pdev, did) )
> @@ -154,16 +153,13 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
>                  addr |= (((u64)1 << (size_order - 1)) - 1) << PAGE_SHIFT_4K;
>              }
>  
> -            rc = qinval_device_iotlb(iommu, pdev->ats_queue_depth,
> -                                     sid, sbit, addr);
> +            ret = qinval_device_iotlb_sync(iommu, pdev->ats_queue_depth,
> +                                           sid, sbit, addr);
>              break;
>          default:
>              dprintk(XENLOG_WARNING VTDPREFIX, "invalid vt-d flush type\n");
>              return -EOPNOTSUPP;
>          }
> -
> -        if ( !ret )
> -            ret = rc;
>      }

The removal of "rc" (which btw is unrelated to the purpose of your
patch) means that if an earlier iteration encountering an error is
followed by later successful ones, no error would get reported to
the caller. Hence this part of the change needs to be undone.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue
  2016-04-01 14:47 ` [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue Quan Xu
@ 2016-04-05  9:47   ` Jan Beulich
  2016-04-10 14:28     ` Xu, Quan
  0 siblings, 1 reply; 18+ messages in thread
From: Jan Beulich @ 2016-04-05  9:47 UTC (permalink / raw)
  To: Quan Xu; +Cc: dario.faggioli, feng.wu, kevin.tian, xen-devel

>>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> If Device-TLB flush timed out, we would hide the target ATS

Please re-consider the use of the word "would" in all your patch
descriptions.

> --- a/xen/drivers/passthrough/vtd/qinval.c
> +++ b/xen/drivers/passthrough/vtd/qinval.c
> @@ -217,6 +217,58 @@ static int invalidate_sync(struct iommu *iommu)
>      return 0;
>  }
>  
> +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> +                                         u16 seg, u8 bus, u8 devfn)
> +{
> +    struct domain *d = NULL;
> +    struct pci_dev *pdev;
> +
> +    if ( test_bit(did, iommu->domid_bitmap) )
> +        d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> +
> +    if ( d == NULL )
> +        return;

This should be accompanied by a comment explaining why
returning here without indicating any error is okay.

> +    pcidevs_lock();
> +
> +    for_each_pdev(d, pdev)
> +    {
> +        if ( ( pdev->seg == seg ) &&
> +             ( pdev->bus == bus ) &&
> +             ( pdev->devfn == devfn ) )

Stray blanks.

> +        {
> +            ASSERT(pdev->domain);
> +            list_del(&pdev->domain_list);
> +            pdev->domain = NULL;
> +            pci_hide_existing_device(pdev);
> +            break;
> +        }
> +    }
> +
> +    pcidevs_unlock();
> +
> +    if ( !is_hardware_domain(d) )
> +        domain_crash(d);

else printk();

> @@ -251,11 +303,13 @@ int qinval_device_iotlb(struct iommu *iommu,
>  }
>  
>  int qinval_device_iotlb_sync(struct iommu *iommu,
> -    u32 max_invs_pend, u16 sid, u16 size, u64 addr)
> +    u32 max_invs_pend, u16 did, u16 seg, u8 bus, u8 devfn, u16 size, u64 addr)
>  {
> +    u16 sid = PCI_BDF2(bus, devfn);

Pointless local variable (but being a matter of taste, the VT-d
maintainers will have the final say).

> --- a/xen/include/xen/pci.h
> +++ b/xen/include/xen/pci.h
> @@ -116,6 +116,7 @@ const unsigned long *pci_get_ro_map(u16 seg);
>  int pci_add_device(u16 seg, u8 bus, u8 devfn,
>                     const struct pci_dev_info *, nodeid_t node);
>  int pci_remove_device(u16 seg, u8 bus, u8 devfn);
> +void pci_hide_existing_device(struct pci_dev *pdev);
>  int pci_ro_device(int seg, int bus, int devfn);
>  int pci_hide_device(int bus, int devfn);

Please move the addition here.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation
  2016-04-05  9:09   ` Jan Beulich
@ 2016-04-07  1:49     ` Xu, Quan
  2016-04-07  1:51       ` Tian, Kevin
  0 siblings, 1 reply; 18+ messages in thread
From: Xu, Quan @ 2016-04-07  1:49 UTC (permalink / raw)
  To: Jan Beulich; +Cc: dario.faggioli, Wu, Feng, Tian, Kevin, xen-devel

On April 05, 2016 5:09pm, <JBeulich@suse.com> wrote:
> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> 
> The subject should mention "timeout", perhaps either in addition to or in place
> of "command line".
> 

I prefer "VT-d: add a timeout parameter for Queued Invalidation".

> > --- a/docs/misc/xen-command-line.markdown
> > +++ b/docs/misc/xen-command-line.markdown
> > @@ -1532,6 +1532,24 @@ Note that if **watchdog** option is also
> specified vpmu will be turned off.
> >  As the virtualisation is not 100% safe, don't use the vpmu flag on
> > production systems (see http://xenbits.xen.org/xsa/advisory-163.html)!
> >
> > +### vtd\_qi\_timeout (VT-d)
> > +> `= <integer>`
> > +
> > +> Default: `1`
> > +
> > +Specify the timeout of the VT-d Queued Invalidation in milliseconds.
> > +By default, the spin timeout is 1ms, which can be boot-time changed.
> 
> Especially the part after the comma makes little sense considering which file
> we're in.

Agreed.

> 
> > +In current code, VT-d Queued Invalidation includes Device-TLB, IOTLB,
> > +Context and IEC flush. If Device-TLB flush timed out, we would hide
> > +the target ATS device and crash the domain owning this ATS device.
> > +If impacted domain is hardware domain, just throw out a warning (done
> > +in queue\_invalidate\_wait). IOTLB, Context and IEC flush timeout are
> > +still in TODO-list.
> 
> Much of this doesn't seem to belong here either.
> 

Could I drop it?

> > +When you see error 'Queue invalidate wait descriptor timed out', try
> > +increasing the vtd\_qi\_timeout to 10ms or more.
> 
> Why 10ms? (If there's no specific reason, I think you'd better drop any explicit
> number.) 

Yes, no specific reason.

Also there's no reason the spell out the command line option again
> here - the context makes clear which value needs increasing.
> 
Agreed.


Then, the new description of xen-command-line.markdown:

+### vtd\_qi\_timeout (VT-d)
+> `= <integer>`
+
+> Default: `1`
+
+Specify the timeout of the VT-d Queued Invalidation in milliseconds.
+By default, the timeout is 1ms.
+
+When you see error 'Queue invalidate wait descriptor timed out', try
+increasing this value.
+

Any more suggestion?

Quan




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation
  2016-04-07  1:49     ` Xu, Quan
@ 2016-04-07  1:51       ` Tian, Kevin
  0 siblings, 0 replies; 18+ messages in thread
From: Tian, Kevin @ 2016-04-07  1:51 UTC (permalink / raw)
  To: Xu, Quan, Jan Beulich; +Cc: dario.faggioli, Wu, Feng, xen-devel

> From: Xu, Quan
> Sent: Thursday, April 07, 2016 9:49 AM
> 
> 
> >
> > > +In current code, VT-d Queued Invalidation includes Device-TLB, IOTLB,
> > > +Context and IEC flush. If Device-TLB flush timed out, we would hide
> > > +the target ATS device and crash the domain owning this ATS device.
> > > +If impacted domain is hardware domain, just throw out a warning (done
> > > +in queue\_invalidate\_wait). IOTLB, Context and IEC flush timeout are
> > > +still in TODO-list.
> >
> > Much of this doesn't seem to belong here either.
> >
> 
> Could I drop it?

yes, please do it. Above belongs to either a patch commit or code comment.

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-05  9:35   ` Jan Beulich
@ 2016-04-07  7:44     ` Xu, Quan
  2016-04-07 15:28       ` Jan Beulich
  2016-04-11  6:52       ` Tian, Kevin
  0 siblings, 2 replies; 18+ messages in thread
From: Xu, Quan @ 2016-04-07  7:44 UTC (permalink / raw)
  To: Jan Beulich; +Cc: dario.faggioli, Wu, Feng, Tian, Kevin, xen-devel

On April 05, 2016 5:35pm, Jan Beulich <JBeulich@suse.com> wrote:
> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> > The dev_invalidate_iotlb() scans ats_devices list to flush ATS
> > devices, and the invalidate_sync() is put after dev_invalidate_iotlb()
> > to synchronize with hardware for flush status. If we assign multiple
> > ATS devices to a domain, the flush status is about all these multiple
> > ATS devices. Once flush timeout expires, we couldn't find out which
> > one is the buggy ATS device.
> 
> Is that true? Or is that just a limitation of our implementation?
> 

IMO, both.
I hope vt-d maintainers help me double check it.

> > Then, The invalidate_sync() variant (We need to pass down the device's
> > SBDF to hide the ATS device) is put within dev_invalidate_iotlb() to
> > synchronize for the flush status one by one.
> 
> I don't think this is stating current state of things. So ...
> 
> > If flush timeout expires,
> > we could find out the buggy ATS device and hide it. However, for other
> > VT-d flush interfaces, the invalidate_sync() is still put after at present.
> > This is inconsistent.
> 
> ... taken together, what is inconsistent here needs to be described better, as well
> as what it is you do to eliminate the inconsistency. Note that you should not
> refer back (or imply knowledge of) the previous discussion on the earlier
> version.
> In any of that discussion is useful here, you need to summarize it instead.
> 

I will continue to summarize it and send out later.

> > --- a/xen/drivers/passthrough/vtd/extern.h
> > +++ b/xen/drivers/passthrough/vtd/extern.h
> > @@ -61,6 +61,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16
> > did,
> >
> >  int qinval_device_iotlb(struct iommu *iommu,
> >                          u32 max_invs_pend, u16 sid, u16 size, u64
> > addr);
> > +int qinval_device_iotlb_sync(struct iommu *iommu, u32 max_invs_pend,
> > +                             u16 sid, u16 size, u64 addr);
> 
> So are then both functions needed to be externally accessible?
> That would seem contrary to the last paragraph of the patch description.
> 

I was aware of this. I'd better make the qinval_device_iotlb() a static one in next v10.

[...]

> > +static int queue_invalidate_context_sync(struct iommu *iommu,
> 
> __must_check?
> 

Agreed.

[...]

> > +{
> > +    queue_invalidate_context(iommu, did, source_id,
> > +                             function_mask, granu);
> > +
> > +    return invalidate_sync(iommu);
> > +}
> 
> Further down you replace the only call to
> queue_invalidate_context() - why keep both functions instead of just making the
> existing one do the sync? (That would the likely also apply to
> qinval_device_iotlb() and others below.)
> 

It is optional.
 I think:
1. in the long term, we may need no _sync version.
2. At least, the current wrap looks good to me. e.g. queue_invalidate_context() is for context-cache Invalidate Descriptor, and the
invalidate_sync() is for Invalidation Wait Descriptor. It is much clearer.

> > @@ -338,23 +365,24 @@ static int flush_iotlb_qi(
> >
> >      if ( qi_ctrl->qinval_maddr != 0 )
> >      {
> > -        int rc;
> > -
> >          /* use queued invalidation */
> >          if (cap_write_drain(iommu->cap))
> >              dw = 1;
> >          if (cap_read_drain(iommu->cap))
> >              dr = 1;
> >          /* Need to conside the ih bit later */
> > -        queue_invalidate_iotlb(iommu,
> > -                               type >>
> DMA_TLB_FLUSH_GRANU_OFFSET, dr,
> > -                               dw, did, size_order, 0, addr);
> > +        ret = queue_invalidate_iotlb_sync(iommu,
> > +                  type >> DMA_TLB_FLUSH_GRANU_OFFSET, dr, dw, did,
> > +                  size_order, 0, addr);
> > +
> > +        /* TODO: Timeout error handling to be added later */
> 
> As of today queue_invalidate_wait() panics, so this comment is not very helpful
> as there is not timeout that could possibly be detected here.
> 

Okay, I will drop it.


> > +        if ( ret )
> > +            return ret;
> > +
> >          if ( flush_dev_iotlb )
> >              ret = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
> > -        rc = invalidate_sync(iommu);
> > -        if ( !ret )
> > -            ret = rc;
> >      }
> 
> I think leaving the existing logic as is would be better - best effort invalidation
> even when an error has occurred.
> 

I have an open:
As vt-d spec(:Queued Invalidation Ordering Considerations) said,
     1. If the Fence(FN) flag is 1 in a inv_wait_dsc, hardware must execute descriptors following the inv_wait_dsc only after wait command is completed.
     2. when a Device-TLB invalidation timeout is detected, hardware must not complete any pending inv_wait_dsc commands.
In current code, the Fence(FN) is always 1.
if a Device-TLB invalidation timeout is detected, this additional inv_wait_dsc is not completed.
__iiuc__, 
the new coming descriptors, in that queue, _might_ be not executed any more, waiting for this additional inv_wait_dsc which is not completed.
is it true?


> > --- a/xen/drivers/passthrough/vtd/x86/ats.c
> > +++ b/xen/drivers/passthrough/vtd/x86/ats.c
> > @@ -118,7 +118,6 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16
> did,
> >      {
> >          u16 sid = PCI_BDF2(pdev->bus, pdev->devfn);
> >          bool_t sbit;
> > -        int rc = 0;
> >
> >          /* Only invalidate devices that belong to this IOMMU */
> >          if ( pdev->iommu != iommu )
> > @@ -134,8 +133,8 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16
> did,
> >              /* invalidate all translations: sbit=1,bit_63=0,bit[62:12]=1 */
> >              sbit = 1;
> >              addr = (~0UL << PAGE_SHIFT_4K) & 0x7FFFFFFFFFFFFFFF;
> > -            rc = qinval_device_iotlb(iommu, pdev->ats_queue_depth,
> > -                                     sid, sbit, addr);
> > +            ret = qinval_device_iotlb_sync(iommu,
> pdev->ats_queue_depth,
> > +                                           sid, sbit, addr);
> >              break;
> >          case DMA_TLB_PSI_FLUSH:
> >              if ( !device_in_domain(iommu, pdev, did) ) @@ -154,16
> > +153,13 @@ int dev_invalidate_iotlb(struct iommu *iommu, u16 did,
> >                  addr |= (((u64)1 << (size_order - 1)) - 1) <<
> PAGE_SHIFT_4K;
> >              }
> >
> > -            rc = qinval_device_iotlb(iommu, pdev->ats_queue_depth,
> > -                                     sid, sbit, addr);
> > +            ret = qinval_device_iotlb_sync(iommu,
> pdev->ats_queue_depth,
> > +                                           sid, sbit, addr);
> >              break;
> >          default:
> >              dprintk(XENLOG_WARNING VTDPREFIX, "invalid vt-d flush
> type\n");
> >              return -EOPNOTSUPP;
> >          }
> > -
> > -        if ( !ret )
> > -            ret = rc;
> >      }
> 
> The removal of "rc" (which btw is unrelated to the purpose of your
> patch) means that if an earlier iteration encountering an error is followed by
> later successful ones, no error would get reported to the caller. Hence this part
> of the change needs to be undone.
> 

Agreed.
I tried to drop rc, as the ret was always zero in previous code.


Quan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-07  7:44     ` Xu, Quan
@ 2016-04-07 15:28       ` Jan Beulich
  2016-04-08  2:20         ` Xu, Quan
  2016-04-11  6:52       ` Tian, Kevin
  1 sibling, 1 reply; 18+ messages in thread
From: Jan Beulich @ 2016-04-07 15:28 UTC (permalink / raw)
  To: Quan Xu; +Cc: dario.faggioli, Feng Wu, Kevin Tian, xen-devel

>>> On 07.04.16 at 09:44, <quan.xu@intel.com> wrote:
> On April 05, 2016 5:35pm, Jan Beulich <JBeulich@suse.com> wrote:
>> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
>> > +{
>> > +    queue_invalidate_context(iommu, did, source_id,
>> > +                             function_mask, granu);
>> > +
>> > +    return invalidate_sync(iommu);
>> > +}
>> 
>> Further down you replace the only call to
>> queue_invalidate_context() - why keep both functions instead of just making 
> the
>> existing one do the sync? (That would the likely also apply to
>> qinval_device_iotlb() and others below.)
>> 
> 
> It is optional.
>  I think:
> 1. in the long term, we may need no _sync version.
> 2. At least, the current wrap looks good to me. e.g. 
> queue_invalidate_context() is for context-cache Invalidate Descriptor, and 
> the
> invalidate_sync() is for Invalidation Wait Descriptor. It is much clearer.

I don't really agree, but will leave it to the VT-d maintainers to judge.

>> > +        if ( ret )
>> > +            return ret;
>> > +
>> >          if ( flush_dev_iotlb )
>> >              ret = dev_invalidate_iotlb(iommu, did, addr, size_order, type);
>> > -        rc = invalidate_sync(iommu);
>> > -        if ( !ret )
>> > -            ret = rc;
>> >      }
>> 
>> I think leaving the existing logic as is would be better - best effort 
> invalidation
>> even when an error has occurred.
>> 
> 
> I have an open:
> As vt-d spec(:Queued Invalidation Ordering Considerations) said,
>      1. If the Fence(FN) flag is 1 in a inv_wait_dsc, hardware must execute 
> descriptors following the inv_wait_dsc only after wait command is completed.
>      2. when a Device-TLB invalidation timeout is detected, hardware must 
> not complete any pending inv_wait_dsc commands.
> In current code, the Fence(FN) is always 1.
> if a Device-TLB invalidation timeout is detected, this additional 
> inv_wait_dsc is not completed.
> __iiuc__, 
> the new coming descriptors, in that queue, _might_ be not executed any more, 
> waiting for this additional inv_wait_dsc which is not completed.
> is it true?

That's not a question to me, is it?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-07 15:28       ` Jan Beulich
@ 2016-04-08  2:20         ` Xu, Quan
  2016-04-11  7:25           ` Tian, Kevin
  0 siblings, 1 reply; 18+ messages in thread
From: Xu, Quan @ 2016-04-08  2:20 UTC (permalink / raw)
  To: Jan Beulich, Wu, Feng, Tian, Kevin; +Cc: dario.faggioli, xen-devel

On April 07, 2016 11:29pm, Jan Beulich <JBeulich@suse.com> wrote:
> >>> On 07.04.16 at 09:44, <quan.xu@intel.com> wrote:
> > On April 05, 2016 5:35pm, Jan Beulich <JBeulich@suse.com> wrote:
> >> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> >> > +{
> >> > +    queue_invalidate_context(iommu, did, source_id,
> >> > +                             function_mask, granu);
> >> > +
> >> > +    return invalidate_sync(iommu); }
> >>
> >> Further down you replace the only call to
> >> queue_invalidate_context() - why keep both functions instead of just
> >> making
> > the
> >> existing one do the sync? (That would the likely also apply to
> >> qinval_device_iotlb() and others below.)
> >>
> >
> > It is optional.
> >  I think:
> > 1. in the long term, we may need no _sync version.
> > 2. At least, the current wrap looks good to me. e.g.
> > queue_invalidate_context() is for context-cache Invalidate Descriptor,
> > and the
> > invalidate_sync() is for Invalidation Wait Descriptor. It is much clearer.
> 
> I don't really agree, but will leave it to the VT-d maintainers to judge.
> 

+to Kevin and Feng, I am open for it.


> >> > +        if ( ret )
> >> > +            return ret;
> >> > +
> >> >          if ( flush_dev_iotlb )
> >> >              ret = dev_invalidate_iotlb(iommu, did, addr, size_order,
> type);
> >> > -        rc = invalidate_sync(iommu);
> >> > -        if ( !ret )
> >> > -            ret = rc;
> >> >      }
> >>
> >> I think leaving the existing logic as is would be better - best
> >> effort
> > invalidation
> >> even when an error has occurred.
> >>
> >
> > I have an open:
> > As vt-d spec(:Queued Invalidation Ordering Considerations) said,
> >      1. If the Fence(FN) flag is 1 in a inv_wait_dsc, hardware must
> > execute descriptors following the inv_wait_dsc only after wait command is
> completed.
> >      2. when a Device-TLB invalidation timeout is detected, hardware
> > must not complete any pending inv_wait_dsc commands.
> > In current code, the Fence(FN) is always 1.
> > if a Device-TLB invalidation timeout is detected, this additional
> > inv_wait_dsc is not completed.
> > __iiuc__,
> > the new coming descriptors, in that queue, _might_ be not executed any
> > more, waiting for this additional inv_wait_dsc which is not completed.
> > is it true?
> 
> That's not a question to me, is it?

To community, but vt-d maintainers are someone who can explain to me.

Quan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue
  2016-04-05  9:47   ` Jan Beulich
@ 2016-04-10 14:28     ` Xu, Quan
  2016-04-11 16:17       ` Jan Beulich
  0 siblings, 1 reply; 18+ messages in thread
From: Xu, Quan @ 2016-04-10 14:28 UTC (permalink / raw)
  To: Jan Beulich; +Cc: dario.faggioli, Wu, Feng, Tian, Kevin, xen-devel

On April 05, 2016 5:48pm, Jan Beulich <JBeulich@suse.com> wrote:
> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> > +static void dev_invalidate_iotlb_timeout(struct iommu *iommu, u16 did,
> > +                                         u16 seg, u8 bus, u8 devfn)
> {
> > +    struct domain *d = NULL;
> > +    struct pci_dev *pdev;
> > +
> > +    if ( test_bit(did, iommu->domid_bitmap) )
> > +        d = rcu_lock_domain_by_id(iommu->domid_map[did]);
> > +
> > +    if ( d == NULL )
> > +        return;
> 
> This should be accompanied by a comment explaining why returning here
> without indicating any error is okay.
> 

I'd say:
"The domain has been freed, and the device no longer belongs to this domain."



> > +        {
> > +            ASSERT(pdev->domain);
> > +            list_del(&pdev->domain_list);
> > +            pdev->domain = NULL;
> > +            pci_hide_existing_device(pdev);
> > +            break;
> > +        }
> > +    }
> > +
> > +    pcidevs_unlock();
> > +
> > +    if ( !is_hardware_domain(d) )
> > +        domain_crash(d);
> 
> else printk();
> 

As I mentioned in commit log "If impacted domain is hardware domain, just throw out a warning (done in queue_invalidate_wait).",
I am not sure whether we really need a printk() at this point or not.

> > @@ -251,11 +303,13 @@ int qinval_device_iotlb(struct iommu *iommu,  }
> >
> >  int qinval_device_iotlb_sync(struct iommu *iommu,
> > -    u32 max_invs_pend, u16 sid, u16 size, u64 addr)
> > +    u32 max_invs_pend, u16 did, u16 seg, u8 bus, u8 devfn, u16 size,
> > + u64 addr)
> >  {
> > +    u16 sid = PCI_BDF2(bus, devfn);
> 
> Pointless local variable (but being a matter of taste, the VT-d maintainers will
> have the final say).
> 

I will drop this local variable.

> > --- a/xen/include/xen/pci.h
> > +++ b/xen/include/xen/pci.h
> > @@ -116,6 +116,7 @@ const unsigned long *pci_get_ro_map(u16 seg);  int
> > pci_add_device(u16 seg, u8 bus, u8 devfn,
> >                     const struct pci_dev_info *, nodeid_t node);  int
> > pci_remove_device(u16 seg, u8 bus, u8 devfn);
> > +void pci_hide_existing_device(struct pci_dev *pdev);
> >  int pci_ro_device(int seg, int bus, int devfn);  int
> > pci_hide_device(int bus, int devfn);
> 
> Please move the addition here.
> 


Agreed.


Quan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-07  7:44     ` Xu, Quan
  2016-04-07 15:28       ` Jan Beulich
@ 2016-04-11  6:52       ` Tian, Kevin
  2016-04-11  8:01         ` Xu, Quan
  1 sibling, 1 reply; 18+ messages in thread
From: Tian, Kevin @ 2016-04-11  6:52 UTC (permalink / raw)
  To: Xu, Quan, Jan Beulich; +Cc: dario.faggioli, Wu, Feng, xen-devel

> From: Xu, Quan
> Sent: Thursday, April 07, 2016 3:45 PM
> 
> On April 05, 2016 5:35pm, Jan Beulich <JBeulich@suse.com> wrote:
> > >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> > > The dev_invalidate_iotlb() scans ats_devices list to flush ATS
> > > devices, and the invalidate_sync() is put after dev_invalidate_iotlb()
> > > to synchronize with hardware for flush status. If we assign multiple
> > > ATS devices to a domain, the flush status is about all these multiple
> > > ATS devices. Once flush timeout expires, we couldn't find out which
> > > one is the buggy ATS device.
> >
> > Is that true? Or is that just a limitation of our implementation?
> >
> 
> IMO, both.
> I hope vt-d maintainers help me double check it.

Just a limitation of our implementation. Now dev_invalidate_iotlb puts
all possible IOTLB flush requests to the queue, and then invalidate_sync
pushes a wait descriptor w/ timeout to detect error. VT-d spec says one
or more descriptors may be fetched together by hardware. So when a 
timeout is triggered, we cannot tell which flush request actually has 
problem by reading queue head register. If we change the implementation
to one-invalidation-sync-per-request, then we can tell. I discussed with
Quan not to go that complexity though.

Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-08  2:20         ` Xu, Quan
@ 2016-04-11  7:25           ` Tian, Kevin
  2016-04-11  9:07             ` Xu, Quan
  0 siblings, 1 reply; 18+ messages in thread
From: Tian, Kevin @ 2016-04-11  7:25 UTC (permalink / raw)
  To: Xu, Quan, Jan Beulich, Wu, Feng; +Cc: dario.faggioli, xen-devel

> From: Xu, Quan
> Sent: Friday, April 08, 2016 10:21 AM
> 
> On April 07, 2016 11:29pm, Jan Beulich <JBeulich@suse.com> wrote:
> > >>> On 07.04.16 at 09:44, <quan.xu@intel.com> wrote:
> > > On April 05, 2016 5:35pm, Jan Beulich <JBeulich@suse.com> wrote:
> > >> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> > >> > +{
> > >> > +    queue_invalidate_context(iommu, did, source_id,
> > >> > +                             function_mask, granu);
> > >> > +
> > >> > +    return invalidate_sync(iommu); }
> > >>
> > >> Further down you replace the only call to
> > >> queue_invalidate_context() - why keep both functions instead of just
> > >> making
> > > the
> > >> existing one do the sync? (That would the likely also apply to
> > >> qinval_device_iotlb() and others below.)
> > >>
> > >
> > > It is optional.
> > >  I think:
> > > 1. in the long term, we may need no _sync version.
> > > 2. At least, the current wrap looks good to me. e.g.
> > > queue_invalidate_context() is for context-cache Invalidate Descriptor,
> > > and the
> > > invalidate_sync() is for Invalidation Wait Descriptor. It is much clearer.
> >
> > I don't really agree, but will leave it to the VT-d maintainers to judge.
> >
> 
> +to Kevin and Feng, I am open for it.

Let's just change existing one to _sync.

> 
> 
> > >> > +        if ( ret )
> > >> > +            return ret;
> > >> > +
> > >> >          if ( flush_dev_iotlb )
> > >> >              ret = dev_invalidate_iotlb(iommu, did, addr, size_order,
> > type);
> > >> > -        rc = invalidate_sync(iommu);
> > >> > -        if ( !ret )
> > >> > -            ret = rc;
> > >> >      }
> > >>
> > >> I think leaving the existing logic as is would be better - best
> > >> effort
> > > invalidation
> > >> even when an error has occurred.
> > >>
> > >
> > > I have an open:
> > > As vt-d spec(:Queued Invalidation Ordering Considerations) said,
> > >      1. If the Fence(FN) flag is 1 in a inv_wait_dsc, hardware must
> > > execute descriptors following the inv_wait_dsc only after wait command is
> > completed.
> > >      2. when a Device-TLB invalidation timeout is detected, hardware
> > > must not complete any pending inv_wait_dsc commands.
> > > In current code, the Fence(FN) is always 1.
> > > if a Device-TLB invalidation timeout is detected, this additional
> > > inv_wait_dsc is not completed.
> > > __iiuc__,
> > > the new coming descriptors, in that queue, _might_ be not executed any
> > > more, waiting for this additional inv_wait_dsc which is not completed.
> > > is it true?
> >
> > That's not a question to me, is it?
> 
> To community, but vt-d maintainers are someone who can explain to me.
> 

'not completed' here means 'abort', so your timeout check will be
hit since the status is never 'done' then.

But I'm not sure how your question is related to Jan's comment, which
looks reasonable to me. :-)

Thanks
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-11  6:52       ` Tian, Kevin
@ 2016-04-11  8:01         ` Xu, Quan
  0 siblings, 0 replies; 18+ messages in thread
From: Xu, Quan @ 2016-04-11  8:01 UTC (permalink / raw)
  To: Tian, Kevin, Jan Beulich; +Cc: dario.faggioli, Wu, Feng, xen-devel

On April 11, 2016 2:53pm, Tian, Kevin wrote:
> > From: Xu, Quan
> > Sent: Thursday, April 07, 2016 3:45 PM
> >
> > On April 05, 2016 5:35pm, Jan Beulich <JBeulich@suse.com> wrote:
> > > >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> > > > The dev_invalidate_iotlb() scans ats_devices list to flush ATS
> > > > devices, and the invalidate_sync() is put after
> > > > dev_invalidate_iotlb() to synchronize with hardware for flush
> > > > status. If we assign multiple ATS devices to a domain, the flush
> > > > status is about all these multiple ATS devices. Once flush timeout
> > > > expires, we couldn't find out which one is the buggy ATS device.
> > >
> > > Is that true? Or is that just a limitation of our implementation?
> > >
> >
> > IMO, both.
> > I hope vt-d maintainers help me double check it.
> 
> Just a limitation of our implementation. Now dev_invalidate_iotlb puts all
> possible IOTLB flush requests to the queue, and then invalidate_sync pushes a
> wait descriptor w/ timeout to detect error. VT-d spec says one or more
> descriptors may be fetched together by hardware. So when a timeout is
> triggered, we cannot tell which flush request actually has problem by reading
> queue head register. If we change the implementation to
> one-invalidation-sync-per-request, then we can tell. I discussed with Quan not
> to go that complexity though.
> 

Thanks for your correction!
I will enhance the commit log and send out later.

Quan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces
  2016-04-11  7:25           ` Tian, Kevin
@ 2016-04-11  9:07             ` Xu, Quan
  0 siblings, 0 replies; 18+ messages in thread
From: Xu, Quan @ 2016-04-11  9:07 UTC (permalink / raw)
  To: Tian, Kevin, Jan Beulich, Wu, Feng; +Cc: dario.faggioli, xen-devel

On April 11, 2016 3:25pm, Tian, Kevin <kevin.tian@intel.com> wrote:
> > From: Xu, Quan
> > Sent: Friday, April 08, 2016 10:21 AM
> >
> > On April 07, 2016 11:29pm, Jan Beulich <JBeulich@suse.com> wrote:
> > > >>> On 07.04.16 at 09:44, <quan.xu@intel.com> wrote:
> > > > On April 05, 2016 5:35pm, Jan Beulich <JBeulich@suse.com> wrote:
> > > >> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
> > > >> > +{
> > > >> > +    queue_invalidate_context(iommu, did, source_id,
> > > >> > +                             function_mask, granu);
> > > >> > +
> > > >> > +    return invalidate_sync(iommu); }
> > > >>
> > > >> Further down you replace the only call to
> > > >> queue_invalidate_context() - why keep both functions instead of
> > > >> just making
> > > > the
> > > >> existing one do the sync? (That would the likely also apply to
> > > >> qinval_device_iotlb() and others below.)
> > > >>
> > > >
> > > > It is optional.
> > > >  I think:
> > > > 1. in the long term, we may need no _sync version.
> > > > 2. At least, the current wrap looks good to me. e.g.
> > > > queue_invalidate_context() is for context-cache Invalidate
> > > > Descriptor, and the
> > > > invalidate_sync() is for Invalidation Wait Descriptor. It is much clearer.
> > >
> > > I don't really agree, but will leave it to the VT-d maintainers to judge.
> > >
> >
> > +to Kevin and Feng, I am open for it.
> 
> Let's just change existing one to _sync.
> 

Agreed.


> >
> >
> > > >> > +        if ( ret )
> > > >> > +            return ret;
> > > >> > +
> > > >> >          if ( flush_dev_iotlb )
> > > >> >              ret = dev_invalidate_iotlb(iommu, did, addr,
> > > >> > size_order,
> > > type);
> > > >> > -        rc = invalidate_sync(iommu);
> > > >> > -        if ( !ret )
> > > >> > -            ret = rc;
> > > >> >      }
> > > >>
> > > >> I think leaving the existing logic as is would be better - best
> > > >> effort
> > > > invalidation
> > > >> even when an error has occurred.
> > > >>
> > > >
> > > > I have an open:
> > > > As vt-d spec(:Queued Invalidation Ordering Considerations) said,
> > > >      1. If the Fence(FN) flag is 1 in a inv_wait_dsc, hardware
> > > > must execute descriptors following the inv_wait_dsc only after
> > > > wait command is
> > > completed.
> > > >      2. when a Device-TLB invalidation timeout is detected,
> > > > hardware must not complete any pending inv_wait_dsc commands.
> > > > In current code, the Fence(FN) is always 1.
> > > > if a Device-TLB invalidation timeout is detected, this additional
> > > > inv_wait_dsc is not completed.
> > > > __iiuc__,
> > > > the new coming descriptors, in that queue, _might_ be not executed
> > > > any more, waiting for this additional inv_wait_dsc which is not completed.
> > > > is it true?
> > >
> > > That's not a question to me, is it?
> >
> > To community, but vt-d maintainers are someone who can explain to me.
> >
> 
> 'not completed' here means 'abort', so your timeout check will be hit since the
> status is never 'done' then.
> 
> But I'm not sure how your question is related to Jan's comment, which looks
> reasonable to me. :-)
> 

I will take Jan's suggestion.
For this open, at least, to remind myself, I should consider software behavior and hardware behavior.
Let's ignore my open.

Quan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue
  2016-04-10 14:28     ` Xu, Quan
@ 2016-04-11 16:17       ` Jan Beulich
  0 siblings, 0 replies; 18+ messages in thread
From: Jan Beulich @ 2016-04-11 16:17 UTC (permalink / raw)
  To: Quan Xu; +Cc: dario.faggioli, Feng Wu, Kevin Tian, xen-devel

>>> On 10.04.16 at 16:28, <quan.xu@intel.com> wrote:
> On April 05, 2016 5:48pm, Jan Beulich <JBeulich@suse.com> wrote:
>> >>> On 01.04.16 at 16:47, <quan.xu@intel.com> wrote:
>> > +        {
>> > +            ASSERT(pdev->domain);
>> > +            list_del(&pdev->domain_list);
>> > +            pdev->domain = NULL;
>> > +            pci_hide_existing_device(pdev);
>> > +            break;
>> > +        }
>> > +    }
>> > +
>> > +    pcidevs_unlock();
>> > +
>> > +    if ( !is_hardware_domain(d) )
>> > +        domain_crash(d);
>> 
>> else printk();
>> 
> 
> As I mentioned in commit log "If impacted domain is hardware domain, just 
> throw out a warning (done in queue_invalidate_wait).",
> I am not sure whether we really need a printk() at this point or not.

I think the printk() belongs here; I do agree there's no need for two
printk()s on the same call tree.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2016-04-11 16:17 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-04-01 14:47 [PATCH v9 0/3] VT-d Device-TLB flush issue Quan Xu
2016-04-01 14:47 ` [PATCH v9 1/3] VT-d: add a command line parameter for Queued Invalidation Quan Xu
2016-04-05  9:09   ` Jan Beulich
2016-04-07  1:49     ` Xu, Quan
2016-04-07  1:51       ` Tian, Kevin
2016-04-01 14:47 ` [PATCH v9 2/3] VT-d: wrap a _sync version for all VT-d flush interfaces Quan Xu
2016-04-05  9:35   ` Jan Beulich
2016-04-07  7:44     ` Xu, Quan
2016-04-07 15:28       ` Jan Beulich
2016-04-08  2:20         ` Xu, Quan
2016-04-11  7:25           ` Tian, Kevin
2016-04-11  9:07             ` Xu, Quan
2016-04-11  6:52       ` Tian, Kevin
2016-04-11  8:01         ` Xu, Quan
2016-04-01 14:47 ` [PATCH v9 3/3] VT-d: Fix vt-d Device-TLB flush timeout issue Quan Xu
2016-04-05  9:47   ` Jan Beulich
2016-04-10 14:28     ` Xu, Quan
2016-04-11 16:17       ` Jan Beulich

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).