* mptNsas MSI-X fixes
@ 2014-01-04 0:16 Martin K. Petersen
2014-01-04 0:16 ` [PATCH 1/2] [SCSI] mpt2sas: Rework the MSI-X grouping code Martin K. Petersen
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Martin K. Petersen @ 2014-01-04 0:16 UTC (permalink / raw)
To: linux-scsi
We found a couple of MSI-X related problems with mptNsas on very large
systems. The patches below contain fixes for mpt2sas and mpt3sas
respectively.
drivers/scsi/mpt2sas/mpt2sas_base.c | 64 ++++++++++++++++++++----------------------------------
drivers/scsi/mpt3sas/mpt3sas_base.c | 73 ++++++++++++++++++--------------------------------------------
2 files changed, 44 insertions(+), 93 deletions(-)
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] [SCSI] mpt2sas: Rework the MSI-X grouping code
2014-01-04 0:16 mptNsas MSI-X fixes Martin K. Petersen
@ 2014-01-04 0:16 ` Martin K. Petersen
2014-01-04 0:16 ` [PATCH 2/2] [SCSI] mpt3sas: " Martin K. Petersen
2014-06-25 9:10 ` mptNsas MSI-X fixes Sreekanth Reddy
2 siblings, 0 replies; 5+ messages in thread
From: Martin K. Petersen @ 2014-01-04 0:16 UTC (permalink / raw)
To: linux-scsi; +Cc: Martin K. Petersen, Sreekanth Reddy
From: "Martin K. Petersen" <martin.petersen@oracle.com>
On systems with a non power-of-two CPU count the existing MSI-X grouping
code failed to distribute interrupts correctly. Rework the code to
handle arbitrary processor counts.
Also remove the hardcoded upper limit on the number of processors so we
can boot on large systems.
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
CC: Sreekanth Reddy <Sreekanth.Reddy@lsi.com>
---
drivers/scsi/mpt2sas/mpt2sas_base.c | 64 +++++++++++++------------------------
1 file changed, 23 insertions(+), 41 deletions(-)
diff --git a/drivers/scsi/mpt2sas/mpt2sas_base.c b/drivers/scsi/mpt2sas/mpt2sas_base.c
index 3901edc35812..bfdbdf214169 100644
--- a/drivers/scsi/mpt2sas/mpt2sas_base.c
+++ b/drivers/scsi/mpt2sas/mpt2sas_base.c
@@ -1332,53 +1332,35 @@ _base_request_irq(struct MPT2SAS_ADAPTER *ioc, u8 index, u32 vector)
static void
_base_assign_reply_queues(struct MPT2SAS_ADAPTER *ioc)
{
- struct adapter_reply_queue *reply_q;
- int cpu_id;
- int cpu_grouping, loop, grouping, grouping_mod;
+ unsigned int cpu, nr_cpus, nr_msix, index = 0;
if (!_base_is_controller_msix_enabled(ioc))
return;
memset(ioc->cpu_msix_table, 0, ioc->cpu_msix_table_sz);
- /* when there are more cpus than available msix vectors,
- * then group cpus togeather on same irq
- */
- if (ioc->cpu_count > ioc->msix_vector_count) {
- grouping = ioc->cpu_count / ioc->msix_vector_count;
- grouping_mod = ioc->cpu_count % ioc->msix_vector_count;
- if (grouping < 2 || (grouping == 2 && !grouping_mod))
- cpu_grouping = 2;
- else if (grouping < 4 || (grouping == 4 && !grouping_mod))
- cpu_grouping = 4;
- else if (grouping < 8 || (grouping == 8 && !grouping_mod))
- cpu_grouping = 8;
- else
- cpu_grouping = 16;
- } else
- cpu_grouping = 0;
-
- loop = 0;
- reply_q = list_entry(ioc->reply_queue_list.next,
- struct adapter_reply_queue, list);
- for_each_online_cpu(cpu_id) {
- if (!cpu_grouping) {
- ioc->cpu_msix_table[cpu_id] = reply_q->msix_index;
- reply_q = list_entry(reply_q->list.next,
- struct adapter_reply_queue, list);
- } else {
- if (loop < cpu_grouping) {
- ioc->cpu_msix_table[cpu_id] =
- reply_q->msix_index;
- loop++;
- } else {
- reply_q = list_entry(reply_q->list.next,
- struct adapter_reply_queue, list);
- ioc->cpu_msix_table[cpu_id] =
- reply_q->msix_index;
- loop = 1;
- }
+
+ nr_cpus = num_online_cpus();
+ nr_msix = ioc->reply_queue_count = min(ioc->reply_queue_count,
+ ioc->facts.MaxMSIxVectors);
+ if (!nr_msix)
+ return;
+
+ cpu = cpumask_first(cpu_online_mask);
+
+ do {
+ unsigned int i, group = nr_cpus / nr_msix;
+
+ if (index < nr_cpus % nr_msix)
+ group++;
+
+ for (i = 0 ; i < group ; i++) {
+ ioc->cpu_msix_table[cpu] = index;
+ cpu = cpumask_next(cpu, cpu_online_mask);
}
- }
+
+ index++;
+
+ } while (cpu < nr_cpus);
}
/**
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] [SCSI] mpt3sas: Rework the MSI-X grouping code
2014-01-04 0:16 mptNsas MSI-X fixes Martin K. Petersen
2014-01-04 0:16 ` [PATCH 1/2] [SCSI] mpt2sas: Rework the MSI-X grouping code Martin K. Petersen
@ 2014-01-04 0:16 ` Martin K. Petersen
2014-06-25 9:10 ` mptNsas MSI-X fixes Sreekanth Reddy
2 siblings, 0 replies; 5+ messages in thread
From: Martin K. Petersen @ 2014-01-04 0:16 UTC (permalink / raw)
To: linux-scsi; +Cc: Martin K. Petersen, Sreekanth Reddy
From: "Martin K. Petersen" <martin.petersen@oracle.com>
On systems with a non power-of-two CPU count the existing MSI-X grouping
code failed to distribute interrupts correctly. Rework the code to
handle arbitrary processor counts.
Also remove the hardcoded upper limit on the number of processors so we
can boot on large systems.
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
Cc: Sreekanth Reddy <Sreekanth.Reddy@lsi.com>
---
drivers/scsi/mpt3sas/mpt3sas_base.c | 73 +++++++++++--------------------------
1 file changed, 21 insertions(+), 52 deletions(-)
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index fa785062e97b..35b95e61acd5 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -1624,66 +1624,35 @@ _base_request_irq(struct MPT3SAS_ADAPTER *ioc, u8 index, u32 vector)
static void
_base_assign_reply_queues(struct MPT3SAS_ADAPTER *ioc)
{
- struct adapter_reply_queue *reply_q;
- int cpu_id;
- int cpu_grouping, loop, grouping, grouping_mod;
- int reply_queue;
+ unsigned int cpu, nr_cpus, nr_msix, index = 0;
if (!_base_is_controller_msix_enabled(ioc))
return;
memset(ioc->cpu_msix_table, 0, ioc->cpu_msix_table_sz);
- /* NUMA Hardware bug workaround - drop to less reply queues */
- if (ioc->reply_queue_count > ioc->facts.MaxMSIxVectors) {
- ioc->reply_queue_count = ioc->facts.MaxMSIxVectors;
- reply_queue = 0;
- list_for_each_entry(reply_q, &ioc->reply_queue_list, list) {
- reply_q->msix_index = reply_queue;
- if (++reply_queue == ioc->reply_queue_count)
- reply_queue = 0;
- }
- }
+ nr_cpus = num_online_cpus();
+ nr_msix = ioc->reply_queue_count = min(ioc->reply_queue_count,
+ ioc->facts.MaxMSIxVectors);
+ if (!nr_msix)
+ return;
- /* when there are more cpus than available msix vectors,
- * then group cpus togeather on same irq
- */
- if (ioc->cpu_count > ioc->msix_vector_count) {
- grouping = ioc->cpu_count / ioc->msix_vector_count;
- grouping_mod = ioc->cpu_count % ioc->msix_vector_count;
- if (grouping < 2 || (grouping == 2 && !grouping_mod))
- cpu_grouping = 2;
- else if (grouping < 4 || (grouping == 4 && !grouping_mod))
- cpu_grouping = 4;
- else if (grouping < 8 || (grouping == 8 && !grouping_mod))
- cpu_grouping = 8;
- else
- cpu_grouping = 16;
- } else
- cpu_grouping = 0;
-
- loop = 0;
- reply_q = list_entry(ioc->reply_queue_list.next,
- struct adapter_reply_queue, list);
- for_each_online_cpu(cpu_id) {
- if (!cpu_grouping) {
- ioc->cpu_msix_table[cpu_id] = reply_q->msix_index;
- reply_q = list_entry(reply_q->list.next,
- struct adapter_reply_queue, list);
- } else {
- if (loop < cpu_grouping) {
- ioc->cpu_msix_table[cpu_id] =
- reply_q->msix_index;
- loop++;
- } else {
- reply_q = list_entry(reply_q->list.next,
- struct adapter_reply_queue, list);
- ioc->cpu_msix_table[cpu_id] =
- reply_q->msix_index;
- loop = 1;
- }
+ cpu = cpumask_first(cpu_online_mask);
+
+ do {
+ unsigned int i, group = nr_cpus / nr_msix;
+
+ if (index < nr_cpus % nr_msix)
+ group++;
+
+ for (i = 0 ; i < group ; i++) {
+ ioc->cpu_msix_table[cpu] = index;
+ cpu = cpumask_next(cpu, cpu_online_mask);
}
- }
+
+ index++;
+
+ } while (cpu < nr_cpus);
}
/**
--
1.8.3.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* RE: mptNsas MSI-X fixes
2014-01-04 0:16 mptNsas MSI-X fixes Martin K. Petersen
2014-01-04 0:16 ` [PATCH 1/2] [SCSI] mpt2sas: Rework the MSI-X grouping code Martin K. Petersen
2014-01-04 0:16 ` [PATCH 2/2] [SCSI] mpt3sas: " Martin K. Petersen
@ 2014-06-25 9:10 ` Sreekanth Reddy
2014-06-25 11:29 ` Christoph Hellwig
2 siblings, 1 reply; 5+ messages in thread
From: Sreekanth Reddy @ 2014-06-25 9:10 UTC (permalink / raw)
To: Martin K. Petersen, linux-scsi, James Bottomley; +Cc: sreekanth.reddy
Hi James,
This Patch set seems to be fine. Please consider this patch set as
Acked-by: Sreekanth Reddy <Sreekanth.reddy@avagotech.com>
Regards,
Sreekanth
>-----Original Message-----
>From: linux-scsi-owner@vger.kernel.org [mailto:linux-scsi-
>owner@vger.kernel.org] On Behalf Of Martin K. Petersen
>Sent: Saturday, January 04, 2014 5:47 AM
>To: linux-scsi@vger.kernel.org
>Subject: mptNsas MSI-X fixes
>
>We found a couple of MSI-X related problems with mptNsas on very large
>systems. The patches below contain fixes for mpt2sas and mpt3sas
>respectively.
>
> drivers/scsi/mpt2sas/mpt2sas_base.c | 64 ++++++++++++++++++++----------
>------------------------
> drivers/scsi/mpt3sas/mpt3sas_base.c | 73 ++++++++++++++++++-------------
>-------------------------------
> 2 files changed, 44 insertions(+), 93 deletions(-)
>
>--
>Martin K. Petersen Oracle Linux Engineering
>
>--
>To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body
>of a message to majordomo@vger.kernel.org More majordomo info at
>http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: mptNsas MSI-X fixes
2014-06-25 9:10 ` mptNsas MSI-X fixes Sreekanth Reddy
@ 2014-06-25 11:29 ` Christoph Hellwig
0 siblings, 0 replies; 5+ messages in thread
From: Christoph Hellwig @ 2014-06-25 11:29 UTC (permalink / raw)
To: Sreekanth Reddy; +Cc: Martin K. Petersen, linux-scsi, James Bottomley
On Wed, Jun 25, 2014 at 02:40:32PM +0530, Sreekanth Reddy wrote:
> Hi James,
>
> This Patch set seems to be fine. Please consider this patch set as
> Acked-by: Sreekanth Reddy <Sreekanth.reddy@avagotech.com>
Any reason you didn't simply include them in your resent driver
update series?
Also any chance you could start sending these series threaded?
Preferably you'd just use git-send-email to send the patches as that
gets everything right for you.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-06-25 11:29 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-04 0:16 mptNsas MSI-X fixes Martin K. Petersen
2014-01-04 0:16 ` [PATCH 1/2] [SCSI] mpt2sas: Rework the MSI-X grouping code Martin K. Petersen
2014-01-04 0:16 ` [PATCH 2/2] [SCSI] mpt3sas: " Martin K. Petersen
2014-06-25 9:10 ` mptNsas MSI-X fixes Sreekanth Reddy
2014-06-25 11:29 ` Christoph Hellwig
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.