linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/3] iommu/vt-d: Some misc tweaks in SVA
@ 2021-01-21  1:45 Lu Baolu
  2021-01-21  1:45 ` [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows Lu Baolu
                   ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Lu Baolu @ 2021-01-21  1:45 UTC (permalink / raw)
  To: Joerg Roedel, Will Deacon
  Cc: Ashok Raj, Jacob Pan, Kevin Tian, Liu Yi L, iommu, linux-kernel,
	Lu Baolu

This includes some misc tweaks in the VT-d SVA implementation. I will
plan them for v5.12 if no objections.

Best regards,
baolu

Lu Baolu (3):
  iommu/vt-d: Add rate limited information when PRQ overflows
  iommu/vt-d: Allow devices to have more than 32 outstanding PRQ
  iommu/vt-d: Use INVALID response code instead of FAILURE

 drivers/iommu/intel/iommu.c |  3 ++-
 drivers/iommu/intel/svm.c   | 19 +++++++++----------
 include/linux/intel-svm.h   |  5 +++++
 3 files changed, 16 insertions(+), 11 deletions(-)

-- 
2.25.1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows
  2021-01-21  1:45 [PATCH 0/3] iommu/vt-d: Some misc tweaks in SVA Lu Baolu
@ 2021-01-21  1:45 ` Lu Baolu
  2021-01-22  6:38   ` Tian, Kevin
  2021-01-21  1:45 ` [PATCH 2/3] iommu/vt-d: Allow devices to have more than 32 outstanding PRQ Lu Baolu
  2021-01-21  1:45 ` [PATCH 3/3] iommu/vt-d: Use INVALID response code instead of FAILURE Lu Baolu
  2 siblings, 1 reply; 8+ messages in thread
From: Lu Baolu @ 2021-01-21  1:45 UTC (permalink / raw)
  To: Joerg Roedel, Will Deacon
  Cc: Ashok Raj, Jacob Pan, Kevin Tian, Liu Yi L, iommu, linux-kernel,
	Lu Baolu

So that the uses could get chances to know what happened.

Suggested-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel/svm.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
index 033b25886e57..f49fe715477b 100644
--- a/drivers/iommu/intel/svm.c
+++ b/drivers/iommu/intel/svm.c
@@ -895,6 +895,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)
 	struct intel_iommu *iommu = d;
 	struct intel_svm *svm = NULL;
 	int head, tail, handled = 0;
+	struct page_req_dsc *req;
 
 	/* Clear PPR bit before reading head/tail registers, to
 	 * ensure that we get a new interrupt if needed. */
@@ -904,7 +905,6 @@ static irqreturn_t prq_event_thread(int irq, void *d)
 	head = dmar_readq(iommu->reg + DMAR_PQH_REG) & PRQ_RING_MASK;
 	while (head != tail) {
 		struct vm_area_struct *vma;
-		struct page_req_dsc *req;
 		struct qi_desc resp;
 		int result;
 		vm_fault_t ret;
@@ -1042,8 +1042,14 @@ static irqreturn_t prq_event_thread(int irq, void *d)
 	 * Clear the page request overflow bit and wake up all threads that
 	 * are waiting for the completion of this handling.
 	 */
-	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO)
+	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
+		head = dmar_readq(iommu->reg + DMAR_PQH_REG) & PRQ_RING_MASK;
+		req = &iommu->prq[head / sizeof(*req)];
+		pr_warn_ratelimited("IOMMU: %s: Page request overflow: HEAD: %08llx %08llx",
+				    iommu->name, ((unsigned long long *)req)[0],
+				    ((unsigned long long *)req)[1]);
 		writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
+	}
 
 	if (!completion_done(&iommu->prq_complete))
 		complete(&iommu->prq_complete);
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 2/3] iommu/vt-d: Allow devices to have more than 32 outstanding PRQ
  2021-01-21  1:45 [PATCH 0/3] iommu/vt-d: Some misc tweaks in SVA Lu Baolu
  2021-01-21  1:45 ` [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows Lu Baolu
@ 2021-01-21  1:45 ` Lu Baolu
  2021-01-21  1:45 ` [PATCH 3/3] iommu/vt-d: Use INVALID response code instead of FAILURE Lu Baolu
  2 siblings, 0 replies; 8+ messages in thread
From: Lu Baolu @ 2021-01-21  1:45 UTC (permalink / raw)
  To: Joerg Roedel, Will Deacon
  Cc: Ashok Raj, Jacob Pan, Kevin Tian, Liu Yi L, iommu, linux-kernel,
	Lu Baolu

The minimum per-IOMMU PRQ queue size is one 4K page, this is more entries
than the hardcoded limit of 32 in the current VT-d code. Some devices can
support up to 512 outstanding PRQs but underutilized by this limit of 32.
Although, 32 gives some rough fairness when multiple devices share the same
IOMMU PRQ queue, but far from optimal for customized use cases.

Signed-off-by: Jacob Pan <jacob.jun.pan@linux.intel.com>
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel/iommu.c | 3 ++-
 drivers/iommu/intel/svm.c   | 4 ----
 include/linux/intel-svm.h   | 5 +++++
 3 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index be85af612bc1..7dee1cb9d6c1 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -33,6 +33,7 @@
 #include <linux/iommu.h>
 #include <linux/dma-iommu.h>
 #include <linux/intel-iommu.h>
+#include <linux/intel-svm.h>
 #include <linux/syscore_ops.h>
 #include <linux/tboot.h>
 #include <linux/dmi.h>
@@ -1527,7 +1528,7 @@ static void iommu_enable_dev_iotlb(struct device_domain_info *info)
 
 	if (info->pri_supported &&
 	    (info->pasid_enabled ? pci_prg_resp_pasid_required(pdev) : 1)  &&
-	    !pci_reset_pri(pdev) && !pci_enable_pri(pdev, 32))
+	    !pci_reset_pri(pdev) && !pci_enable_pri(pdev, PRQ_DEPTH))
 		info->pri_enabled = 1;
 #endif
 	if (info->ats_supported && pci_ats_page_aligned(pdev) &&
diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
index f49fe715477b..77509a0a863e 100644
--- a/drivers/iommu/intel/svm.c
+++ b/drivers/iommu/intel/svm.c
@@ -26,8 +26,6 @@
 static irqreturn_t prq_event_thread(int irq, void *d);
 static void intel_svm_drain_prq(struct device *dev, u32 pasid);
 
-#define PRQ_ORDER 0
-
 int intel_svm_enable_prq(struct intel_iommu *iommu)
 {
 	struct page *pages;
@@ -722,8 +720,6 @@ struct page_req_dsc {
 	u64 priv_data[2];
 };
 
-#define PRQ_RING_MASK	((0x1000 << PRQ_ORDER) - 0x20)
-
 static bool access_error(struct vm_area_struct *vma, struct page_req_dsc *req)
 {
 	unsigned long requested = 0;
diff --git a/include/linux/intel-svm.h b/include/linux/intel-svm.h
index 39d368a810b8..5f7b5abefdec 100644
--- a/include/linux/intel-svm.h
+++ b/include/linux/intel-svm.h
@@ -21,6 +21,11 @@ struct svm_dev_ops {
 #define SVM_REQ_EXEC	(1<<1)
 #define SVM_REQ_PRIV	(1<<0)
 
+/* Page Request Queue depth */
+#define PRQ_ORDER	0
+#define PRQ_RING_MASK	((0x1000 << PRQ_ORDER) - 0x20)
+#define PRQ_DEPTH	((0x1000 << PRQ_ORDER) >> 5)
+
 /*
  * The SVM_FLAG_PRIVATE_PASID flag requests a PASID which is *not* the "main"
  * PASID for the current process. Even if a PASID already exists, a new one
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [PATCH 3/3] iommu/vt-d: Use INVALID response code instead of FAILURE
  2021-01-21  1:45 [PATCH 0/3] iommu/vt-d: Some misc tweaks in SVA Lu Baolu
  2021-01-21  1:45 ` [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows Lu Baolu
  2021-01-21  1:45 ` [PATCH 2/3] iommu/vt-d: Allow devices to have more than 32 outstanding PRQ Lu Baolu
@ 2021-01-21  1:45 ` Lu Baolu
  2 siblings, 0 replies; 8+ messages in thread
From: Lu Baolu @ 2021-01-21  1:45 UTC (permalink / raw)
  To: Joerg Roedel, Will Deacon
  Cc: Ashok Raj, Jacob Pan, Kevin Tian, Liu Yi L, iommu, linux-kernel,
	Lu Baolu

The VT-d IOMMU response RESPONSE_FAILURE for a page request in below
cases:

- When it gets a Page_Request with no PASID;
- When it gets a Page_Request with PASID that is not in use for this
  device.

This is allowed by the spec, but IOMMU driver doesn't support such cases
today. When the device receives RESPONSE_FAILURE, it sends the device
state machine to HALT state. Now if we try to unload the driver, it hangs
since the device doesn't send any outbound transactions to host when the
driver is trying to clear things up. The only possible responses would be
for invalidation requests.

Let's use RESPONSE_INVALID instead for now, so that the device state
machine doesn't enter HALT state.

Suggested-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
---
 drivers/iommu/intel/svm.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
index 77509a0a863e..021f58899c16 100644
--- a/drivers/iommu/intel/svm.c
+++ b/drivers/iommu/intel/svm.c
@@ -907,10 +907,8 @@ static irqreturn_t prq_event_thread(int irq, void *d)
 		u64 address;
 
 		handled = 1;
-
 		req = &iommu->prq[head / sizeof(*req)];
-
-		result = QI_RESP_FAILURE;
+		result = QI_RESP_INVALID;
 		address = (u64)req->addr << VTD_PAGE_SHIFT;
 		if (!req->pasid_present) {
 			pr_err("%s: Page request without PASID: %08llx %08llx\n",
@@ -948,7 +946,6 @@ static irqreturn_t prq_event_thread(int irq, void *d)
 			rcu_read_unlock();
 		}
 
-		result = QI_RESP_INVALID;
 		/* Since we're using init_mm.pgd directly, we should never take
 		 * any faults on kernel addresses. */
 		if (!svm->mm)
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread

* RE: [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows
  2021-01-21  1:45 ` [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows Lu Baolu
@ 2021-01-22  6:38   ` Tian, Kevin
  2021-01-25  6:28     ` Lu Baolu
  0 siblings, 1 reply; 8+ messages in thread
From: Tian, Kevin @ 2021-01-22  6:38 UTC (permalink / raw)
  To: Lu Baolu, Joerg Roedel, Will Deacon
  Cc: Raj, Ashok, Jacob Pan, Liu, Yi L, iommu, linux-kernel

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Thursday, January 21, 2021 9:45 AM
> 
> So that the uses could get chances to know what happened.
> 
> Suggested-by: Ashok Raj <ashok.raj@intel.com>
> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> ---
>  drivers/iommu/intel/svm.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
> index 033b25886e57..f49fe715477b 100644
> --- a/drivers/iommu/intel/svm.c
> +++ b/drivers/iommu/intel/svm.c
> @@ -895,6 +895,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)
>  	struct intel_iommu *iommu = d;
>  	struct intel_svm *svm = NULL;
>  	int head, tail, handled = 0;
> +	struct page_req_dsc *req;
> 
>  	/* Clear PPR bit before reading head/tail registers, to
>  	 * ensure that we get a new interrupt if needed. */
> @@ -904,7 +905,6 @@ static irqreturn_t prq_event_thread(int irq, void *d)
>  	head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
> PRQ_RING_MASK;
>  	while (head != tail) {
>  		struct vm_area_struct *vma;
> -		struct page_req_dsc *req;
>  		struct qi_desc resp;
>  		int result;
>  		vm_fault_t ret;
> @@ -1042,8 +1042,14 @@ static irqreturn_t prq_event_thread(int irq, void
> *d)
>  	 * Clear the page request overflow bit and wake up all threads that
>  	 * are waiting for the completion of this handling.
>  	 */
> -	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO)
> +	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
> +		head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
> PRQ_RING_MASK;
> +		req = &iommu->prq[head / sizeof(*req)];
> +		pr_warn_ratelimited("IOMMU: %s: Page request overflow:
> HEAD: %08llx %08llx",
> +				    iommu->name, ((unsigned long long
> *)req)[0],
> +				    ((unsigned long long *)req)[1]);
>  		writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
> +	}
> 

Not about rate limiting but I think we may have a problem in above
logic. It is incorrect to always clear PRO when it's set w/o first checking
whether the overflow condition has been cleared. This code assumes
that if an overflow condition occurs it must have been cleared by earlier
loop when hitting this check. However since this code runs in a threaded 
context, the overflow condition could occur even after you reset the head 
to the tail (under some extreme condition). To be sane I think we'd better
read both head/tail again after seeing a pending PRO here and only clear 
PRO when it becomes a false indicator based on latest head/tail.

Thanks
Kevin

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows
  2021-01-22  6:38   ` Tian, Kevin
@ 2021-01-25  6:28     ` Lu Baolu
  2021-01-25  8:16       ` Tian, Kevin
  0 siblings, 1 reply; 8+ messages in thread
From: Lu Baolu @ 2021-01-25  6:28 UTC (permalink / raw)
  To: Tian, Kevin, Joerg Roedel, Will Deacon
  Cc: baolu.lu, Raj, Ashok, Jacob Pan, Liu, Yi L, iommu, linux-kernel

Hi Kevin,

On 2021/1/22 14:38, Tian, Kevin wrote:
>> From: Lu Baolu <baolu.lu@linux.intel.com>
>> Sent: Thursday, January 21, 2021 9:45 AM
>>
>> So that the uses could get chances to know what happened.
>>
>> Suggested-by: Ashok Raj <ashok.raj@intel.com>
>> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>> ---
>>   drivers/iommu/intel/svm.c | 10 ++++++++--
>>   1 file changed, 8 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
>> index 033b25886e57..f49fe715477b 100644
>> --- a/drivers/iommu/intel/svm.c
>> +++ b/drivers/iommu/intel/svm.c
>> @@ -895,6 +895,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)
>>   	struct intel_iommu *iommu = d;
>>   	struct intel_svm *svm = NULL;
>>   	int head, tail, handled = 0;
>> +	struct page_req_dsc *req;
>>
>>   	/* Clear PPR bit before reading head/tail registers, to
>>   	 * ensure that we get a new interrupt if needed. */
>> @@ -904,7 +905,6 @@ static irqreturn_t prq_event_thread(int irq, void *d)
>>   	head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
>> PRQ_RING_MASK;
>>   	while (head != tail) {
>>   		struct vm_area_struct *vma;
>> -		struct page_req_dsc *req;
>>   		struct qi_desc resp;
>>   		int result;
>>   		vm_fault_t ret;
>> @@ -1042,8 +1042,14 @@ static irqreturn_t prq_event_thread(int irq, void
>> *d)
>>   	 * Clear the page request overflow bit and wake up all threads that
>>   	 * are waiting for the completion of this handling.
>>   	 */
>> -	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO)
>> +	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
>> +		head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
>> PRQ_RING_MASK;
>> +		req = &iommu->prq[head / sizeof(*req)];
>> +		pr_warn_ratelimited("IOMMU: %s: Page request overflow:
>> HEAD: %08llx %08llx",
>> +				    iommu->name, ((unsigned long long
>> *)req)[0],
>> +				    ((unsigned long long *)req)[1]);
>>   		writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
>> +	}
>>
> 
> Not about rate limiting but I think we may have a problem in above
> logic. It is incorrect to always clear PRO when it's set w/o first checking
> whether the overflow condition has been cleared. This code assumes
> that if an overflow condition occurs it must have been cleared by earlier
> loop when hitting this check. However since this code runs in a threaded
> context, the overflow condition could occur even after you reset the head
> to the tail (under some extreme condition). To be sane I think we'd better
> read both head/tail again after seeing a pending PRO here and only clear
> PRO when it becomes a false indicator based on latest head/tail.
> 

Yes, agreed. We can check the head and tail and clear the overflow bit
until the queue is empty. The finial code looks like:

         /*
          * Clear the page request overflow bit and wake up all threads that
          * are waiting for the completion of this handling.
          */
         if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
                 head = dmar_readq(iommu->reg + DMAR_PQH_REG) & 
PRQ_RING_MASK;
                 tail = dmar_readq(iommu->reg + DMAR_PQT_REG) & 
PRQ_RING_MASK;
                 if (head == tail) {
                         req = &iommu->prq[head / sizeof(*req)];
                         pr_warn_ratelimited("IOMMU: %s: Page request 
overflow cleared: HEAD: %08llx %08llx",
                                             iommu->name, ((unsigned 
long long *)req)[0],
                                             ((unsigned long long 
*)req)[1]);
                         writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
                 }
         }

Thought?

> Thanks
> Kevin
> 

Best regards,
baolu

^ permalink raw reply	[flat|nested] 8+ messages in thread

* RE: [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows
  2021-01-25  6:28     ` Lu Baolu
@ 2021-01-25  8:16       ` Tian, Kevin
  2021-01-25  8:30         ` Lu Baolu
  0 siblings, 1 reply; 8+ messages in thread
From: Tian, Kevin @ 2021-01-25  8:16 UTC (permalink / raw)
  To: Lu Baolu, Joerg Roedel, Will Deacon
  Cc: Raj, Ashok, Jacob Pan, Liu, Yi L, iommu, linux-kernel

> From: Lu Baolu <baolu.lu@linux.intel.com>
> Sent: Monday, January 25, 2021 2:29 PM
> 
> Hi Kevin,
> 
> On 2021/1/22 14:38, Tian, Kevin wrote:
> >> From: Lu Baolu <baolu.lu@linux.intel.com>
> >> Sent: Thursday, January 21, 2021 9:45 AM
> >>
> >> So that the uses could get chances to know what happened.
> >>
> >> Suggested-by: Ashok Raj <ashok.raj@intel.com>
> >> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
> >> ---
> >>   drivers/iommu/intel/svm.c | 10 ++++++++--
> >>   1 file changed, 8 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
> >> index 033b25886e57..f49fe715477b 100644
> >> --- a/drivers/iommu/intel/svm.c
> >> +++ b/drivers/iommu/intel/svm.c
> >> @@ -895,6 +895,7 @@ static irqreturn_t prq_event_thread(int irq, void
> *d)
> >>   	struct intel_iommu *iommu = d;
> >>   	struct intel_svm *svm = NULL;
> >>   	int head, tail, handled = 0;
> >> +	struct page_req_dsc *req;
> >>
> >>   	/* Clear PPR bit before reading head/tail registers, to
> >>   	 * ensure that we get a new interrupt if needed. */
> >> @@ -904,7 +905,6 @@ static irqreturn_t prq_event_thread(int irq, void
> *d)
> >>   	head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
> >> PRQ_RING_MASK;
> >>   	while (head != tail) {
> >>   		struct vm_area_struct *vma;
> >> -		struct page_req_dsc *req;
> >>   		struct qi_desc resp;
> >>   		int result;
> >>   		vm_fault_t ret;
> >> @@ -1042,8 +1042,14 @@ static irqreturn_t prq_event_thread(int irq,
> void
> >> *d)
> >>   	 * Clear the page request overflow bit and wake up all threads that
> >>   	 * are waiting for the completion of this handling.
> >>   	 */
> >> -	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO)
> >> +	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
> >> +		head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
> >> PRQ_RING_MASK;
> >> +		req = &iommu->prq[head / sizeof(*req)];
> >> +		pr_warn_ratelimited("IOMMU: %s: Page request overflow:
> >> HEAD: %08llx %08llx",
> >> +				    iommu->name, ((unsigned long long
> >> *)req)[0],
> >> +				    ((unsigned long long *)req)[1]);
> >>   		writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
> >> +	}
> >>
> >
> > Not about rate limiting but I think we may have a problem in above
> > logic. It is incorrect to always clear PRO when it's set w/o first checking
> > whether the overflow condition has been cleared. This code assumes
> > that if an overflow condition occurs it must have been cleared by earlier
> > loop when hitting this check. However since this code runs in a threaded
> > context, the overflow condition could occur even after you reset the head
> > to the tail (under some extreme condition). To be sane I think we'd better
> > read both head/tail again after seeing a pending PRO here and only clear
> > PRO when it becomes a false indicator based on latest head/tail.
> >
> 
> Yes, agreed. We can check the head and tail and clear the overflow bit
> until the queue is empty. The finial code looks like:
> 
>          /*
>           * Clear the page request overflow bit and wake up all threads that
>           * are waiting for the completion of this handling.
>           */
>          if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
>                  head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
> PRQ_RING_MASK;
>                  tail = dmar_readq(iommu->reg + DMAR_PQT_REG) &
> PRQ_RING_MASK;
>                  if (head == tail) {
>                          req = &iommu->prq[head / sizeof(*req)];
>                          pr_warn_ratelimited("IOMMU: %s: Page request
> overflow cleared: HEAD: %08llx %08llx",
>                                              iommu->name, ((unsigned
> long long *)req)[0],
>                                              ((unsigned long long
> *)req)[1]);
>                          writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
>                  }
>          }
> 
> Thought?
> 

Just a small comment. Is it useful to also print a warning in the true
overflow condition which has to wait for next interrupt to be cleared?

Thanks
Kevin

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows
  2021-01-25  8:16       ` Tian, Kevin
@ 2021-01-25  8:30         ` Lu Baolu
  0 siblings, 0 replies; 8+ messages in thread
From: Lu Baolu @ 2021-01-25  8:30 UTC (permalink / raw)
  To: Tian, Kevin, Joerg Roedel, Will Deacon
  Cc: baolu.lu, Raj, Ashok, Jacob Pan, Liu, Yi L, iommu, linux-kernel

Hi Kevin,

On 2021/1/25 16:16, Tian, Kevin wrote:
>> From: Lu Baolu <baolu.lu@linux.intel.com>
>> Sent: Monday, January 25, 2021 2:29 PM
>>
>> Hi Kevin,
>>
>> On 2021/1/22 14:38, Tian, Kevin wrote:
>>>> From: Lu Baolu <baolu.lu@linux.intel.com>
>>>> Sent: Thursday, January 21, 2021 9:45 AM
>>>>
>>>> So that the uses could get chances to know what happened.
>>>>
>>>> Suggested-by: Ashok Raj <ashok.raj@intel.com>
>>>> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com>
>>>> ---
>>>>    drivers/iommu/intel/svm.c | 10 ++++++++--
>>>>    1 file changed, 8 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c
>>>> index 033b25886e57..f49fe715477b 100644
>>>> --- a/drivers/iommu/intel/svm.c
>>>> +++ b/drivers/iommu/intel/svm.c
>>>> @@ -895,6 +895,7 @@ static irqreturn_t prq_event_thread(int irq, void
>> *d)
>>>>    	struct intel_iommu *iommu = d;
>>>>    	struct intel_svm *svm = NULL;
>>>>    	int head, tail, handled = 0;
>>>> +	struct page_req_dsc *req;
>>>>
>>>>    	/* Clear PPR bit before reading head/tail registers, to
>>>>    	 * ensure that we get a new interrupt if needed. */
>>>> @@ -904,7 +905,6 @@ static irqreturn_t prq_event_thread(int irq, void
>> *d)
>>>>    	head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
>>>> PRQ_RING_MASK;
>>>>    	while (head != tail) {
>>>>    		struct vm_area_struct *vma;
>>>> -		struct page_req_dsc *req;
>>>>    		struct qi_desc resp;
>>>>    		int result;
>>>>    		vm_fault_t ret;
>>>> @@ -1042,8 +1042,14 @@ static irqreturn_t prq_event_thread(int irq,
>> void
>>>> *d)
>>>>    	 * Clear the page request overflow bit and wake up all threads that
>>>>    	 * are waiting for the completion of this handling.
>>>>    	 */
>>>> -	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO)
>>>> +	if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
>>>> +		head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
>>>> PRQ_RING_MASK;
>>>> +		req = &iommu->prq[head / sizeof(*req)];
>>>> +		pr_warn_ratelimited("IOMMU: %s: Page request overflow:
>>>> HEAD: %08llx %08llx",
>>>> +				    iommu->name, ((unsigned long long
>>>> *)req)[0],
>>>> +				    ((unsigned long long *)req)[1]);
>>>>    		writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
>>>> +	}
>>>>
>>>
>>> Not about rate limiting but I think we may have a problem in above
>>> logic. It is incorrect to always clear PRO when it's set w/o first checking
>>> whether the overflow condition has been cleared. This code assumes
>>> that if an overflow condition occurs it must have been cleared by earlier
>>> loop when hitting this check. However since this code runs in a threaded
>>> context, the overflow condition could occur even after you reset the head
>>> to the tail (under some extreme condition). To be sane I think we'd better
>>> read both head/tail again after seeing a pending PRO here and only clear
>>> PRO when it becomes a false indicator based on latest head/tail.
>>>
>>
>> Yes, agreed. We can check the head and tail and clear the overflow bit
>> until the queue is empty. The finial code looks like:
>>
>>           /*
>>            * Clear the page request overflow bit and wake up all threads that
>>            * are waiting for the completion of this handling.
>>            */
>>           if (readl(iommu->reg + DMAR_PRS_REG) & DMA_PRS_PRO) {
>>                   head = dmar_readq(iommu->reg + DMAR_PQH_REG) &
>> PRQ_RING_MASK;
>>                   tail = dmar_readq(iommu->reg + DMAR_PQT_REG) &
>> PRQ_RING_MASK;
>>                   if (head == tail) {
>>                           req = &iommu->prq[head / sizeof(*req)];
>>                           pr_warn_ratelimited("IOMMU: %s: Page request
>> overflow cleared: HEAD: %08llx %08llx",
>>                                               iommu->name, ((unsigned
>> long long *)req)[0],
>>                                               ((unsigned long long
>> *)req)[1]);
>>                           writel(DMA_PRS_PRO, iommu->reg + DMAR_PRS_REG);
>>                   }
>>           }
>>
>> Thought?
>>
> 
> Just a small comment. Is it useful to also print a warning in the true
> overflow condition which has to wait for next interrupt to be cleared?
> 

That's fine. :-)

> Thanks
> Kevin
> 

Best regards,
baolu

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2021-01-26  5:08 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-01-21  1:45 [PATCH 0/3] iommu/vt-d: Some misc tweaks in SVA Lu Baolu
2021-01-21  1:45 ` [PATCH 1/3] iommu/vt-d: Add rate limited information when PRQ overflows Lu Baolu
2021-01-22  6:38   ` Tian, Kevin
2021-01-25  6:28     ` Lu Baolu
2021-01-25  8:16       ` Tian, Kevin
2021-01-25  8:30         ` Lu Baolu
2021-01-21  1:45 ` [PATCH 2/3] iommu/vt-d: Allow devices to have more than 32 outstanding PRQ Lu Baolu
2021-01-21  1:45 ` [PATCH 3/3] iommu/vt-d: Use INVALID response code instead of FAILURE Lu Baolu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).