From: Dave Jiang <dave.jiang@intel.com>
To: vkoul@kernel.org
Cc: Konstantin Ananyev <konstantin.ananyev@intel.com>,
Konstantin Ananyev <konstantin.ananyev@intel.com>,
dmaengine@vger.kernel.org
Subject: [PATCH] dmaengine: idxd: assign MSIX vectors to each WQ rather than roundrobin
Date: Thu, 24 Jun 2021 13:39:33 -0700 [thread overview]
Message-ID: <162456717326.1130457.15258077196523268356.stgit@djiang5-desk3.ch.intel.com> (raw)
IOPS increased when changing MSIX vector to per WQ from roundrobin.
Allows descriptor to be completed by the submitter improves caching
locality.
Suggested-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
---
drivers/dma/idxd/idxd.h | 2 --
drivers/dma/idxd/submit.c | 36 ++++++++----------------------------
2 files changed, 8 insertions(+), 30 deletions(-)
diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h
index 5516ea31ff07..cdefd304d396 100644
--- a/drivers/dma/idxd/idxd.h
+++ b/drivers/dma/idxd/idxd.h
@@ -178,7 +178,6 @@ struct idxd_wq {
enum idxd_wq_state state;
unsigned long flags;
union wqcfg *wqcfg;
- u32 vec_ptr; /* interrupt steering */
struct dsa_hw_desc **hw_descs;
int num_descs;
union {
@@ -314,7 +313,6 @@ struct idxd_desc {
struct list_head list;
int id;
int cpu;
- unsigned int vector;
struct idxd_wq *wq;
};
diff --git a/drivers/dma/idxd/submit.c b/drivers/dma/idxd/submit.c
index 19afb62abaff..406f15a84f11 100644
--- a/drivers/dma/idxd/submit.c
+++ b/drivers/dma/idxd/submit.c
@@ -22,22 +22,13 @@ static struct idxd_desc *__get_desc(struct idxd_wq *wq, int idx, int cpu)
desc->hw->pasid = idxd->pasid;
/*
- * Descriptor completion vectors are 1...N for MSIX. We will round
- * robin through the N vectors.
+ * On host, MSIX vecotr 0 is used for misc interrupt. Therefore when we match
+ * vector 1:1 to the WQ id, we need to add 1
*/
- wq->vec_ptr = (wq->vec_ptr % idxd->num_wq_irqs) + 1;
- if (!idxd->int_handles) {
- desc->hw->int_handle = wq->vec_ptr;
- } else {
- desc->vector = wq->vec_ptr;
- /*
- * int_handles are only for descriptor completion. However for device
- * MSIX enumeration, vec 0 is used for misc interrupts. Therefore even
- * though we are rotating through 1...N for descriptor interrupts, we
- * need to acqurie the int_handles from 0..N-1.
- */
- desc->hw->int_handle = idxd->int_handles[desc->vector - 1];
- }
+ if (!idxd->int_handles)
+ desc->hw->int_handle = wq->id + 1;
+ else
+ desc->hw->int_handle = idxd->int_handles[wq->id];
return desc;
}
@@ -128,19 +119,8 @@ int idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc)
* Pending the descriptor to the lockless list for the irq_entry
* that we designated the descriptor to.
*/
- if (desc->hw->flags & IDXD_OP_FLAG_RCI) {
- int vec;
-
- /*
- * If the driver is on host kernel, it would be the value
- * assigned to interrupt handle, which is index for MSIX
- * vector. If it's guest then can't use the int_handle since
- * that is the index to IMS for the entire device. The guest
- * device local index will be used.
- */
- vec = !idxd->int_handles ? desc->hw->int_handle : desc->vector;
- llist_add(&desc->llnode, &idxd->irq_entries[vec].pending_llist);
- }
+ if (desc->hw->flags & IDXD_OP_FLAG_RCI)
+ llist_add(&desc->llnode, &idxd->irq_entries[wq->id + 1].pending_llist);
return 0;
}
next reply other threads:[~2021-06-24 20:39 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-24 20:39 Dave Jiang [this message]
2021-07-14 6:58 ` [PATCH] dmaengine: idxd: assign MSIX vectors to each WQ rather than roundrobin Vinod Koul
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=162456717326.1130457.15258077196523268356.stgit@djiang5-desk3.ch.intel.com \
--to=dave.jiang@intel.com \
--cc=dmaengine@vger.kernel.org \
--cc=konstantin.ananyev@intel.com \
--cc=vkoul@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).