* [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
@ 2010-06-10 16:59 Arthur Kepner
[not found] ` <20100610165921.GF22247-sJ/iWh9BUns@public.gmane.org>
0 siblings, 1 reply; 5+ messages in thread
From: Arthur Kepner @ 2010-06-10 16:59 UTC (permalink / raw)
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA
The mlx4_core driver allocates 'nreq' msix vectors (and irqs),
where:
nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
num_possible_cpus() + 1);
ConnectX HCAs support 512 event queues (4 reserved). On a system
with enough processors, we get:
mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again
Further attempts (by other drivers) to allocate interrupts fail,
because mlx4_core got 'em all.
How about this?
Signed-off-by: Arthur Kepner <akepner-sJ/iWh9BUns@public.gmane.org>
---
main.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c
index e3e0d54..0a316d0 100644
--- a/drivers/net/mlx4/main.c
+++ b/drivers/net/mlx4/main.c
@@ -68,6 +68,10 @@ static int msi_x = 1;
module_param(msi_x, int, 0444);
MODULE_PARM_DESC(msi_x, "attempt to use MSI-X if nonzero");
+static int max_msi_x_vec = 64;
+module_param(max_msi_x_vec, int, 0444);
+MODULE_PARM_DESC(max_msi_x_vec, "max MSI-X vectors we'll attempt to allocate");
+
#else /* CONFIG_PCI_MSI */
#define msi_x (0)
@@ -968,8 +972,10 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)
int i;
if (msi_x) {
+ nreq = min_t(int, num_possible_cpus() + 1, max_msi_x_vec);
nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
- num_possible_cpus() + 1);
+ nreq);
+
entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
if (!entries)
goto no_msi;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
[not found] ` <20100610165921.GF22247-sJ/iWh9BUns@public.gmane.org>
@ 2010-06-13 6:53 ` Eli Cohen
2010-06-14 15:23 ` Arthur Kepner
2010-06-17 14:53 ` Yevgeny Petrilin
1 sibling, 1 reply; 5+ messages in thread
From: Eli Cohen @ 2010-06-13 6:53 UTC (permalink / raw)
To: Arthur Kepner; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
On Thu, Jun 10, 2010 at 09:59:21AM -0700, Arthur Kepner wrote:
>
> The mlx4_core driver allocates 'nreq' msix vectors (and irqs),
> where:
>
> nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
> num_possible_cpus() + 1);
>
> ConnectX HCAs support 512 event queues (4 reserved). On a system
> with enough processors, we get:
>
> mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again
>
> Further attempts (by other drivers) to allocate interrupts fail,
> because mlx4_core got 'em all.
>
> How about this?
>
Hi Arthur,
how many CPU cores are in your system? What kernel version did you
use?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
2010-06-13 6:53 ` Eli Cohen
@ 2010-06-14 15:23 ` Arthur Kepner
0 siblings, 0 replies; 5+ messages in thread
From: Arthur Kepner @ 2010-06-14 15:23 UTC (permalink / raw)
To: Eli Cohen; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
On Sun, Jun 13, 2010 at 09:53:24AM +0300, Eli Cohen wrote:
> ....
> how many CPU cores are in your system? What kernel version did you
> use?
I'm almost certain that it was a 2048 core system (it's not available
right now for me to verify).
We used 2.6.32.12 (sles11 sp1).
--
Arthur
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
[not found] ` <20100610165921.GF22247-sJ/iWh9BUns@public.gmane.org>
2010-06-13 6:53 ` Eli Cohen
@ 2010-06-17 14:53 ` Yevgeny Petrilin
[not found] ` <E113D394D7C5DB4F8FF691FA7EE9DB4439A77CCD66-WQlSmcKwN8Te+A/uUDamNg@public.gmane.org>
1 sibling, 1 reply; 5+ messages in thread
From: Yevgeny Petrilin @ 2010-06-17 14:53 UTC (permalink / raw)
To: Arthur Kepner, linux-rdma-u79uwXL29TY76Z2rM5mHXA
> The mlx4_core driver allocates 'nreq' msix vectors (and irqs),
> where:
>
> nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
> num_possible_cpus() + 1);
>
> ConnectX HCAs support 512 event queues (4 reserved). On a system with enough processors, we get:
>
> mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again
>
> Further attempts (by other drivers) to allocate interrupts fail, because mlx4_core got 'em all.
>
> How about this?
Hi,
I think that this patch would do the job,
Anyway we are thinking of ways to change our interrupt allocation scheme.
--Yevgeny--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH/RFC] mlx4_core: module param to limit msix vec allocation
[not found] ` <E113D394D7C5DB4F8FF691FA7EE9DB4439A77CCD66-WQlSmcKwN8Te+A/uUDamNg@public.gmane.org>
@ 2010-06-17 21:27 ` Arthur Kepner
0 siblings, 0 replies; 5+ messages in thread
From: Arthur Kepner @ 2010-06-17 21:27 UTC (permalink / raw)
To: Yevgeny Petrilin; +Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA
On Thu, Jun 17, 2010 at 05:53:58PM +0300, Yevgeny Petrilin wrote:
> I think that this patch would do the job,
(Is that an ack?)
> Anyway we are thinking of ways to change our interrupt allocation scheme.
>
Would be interested to know what you've got in mind.
--
Arthur
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2010-06-17 21:27 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-06-10 16:59 [PATCH/RFC] mlx4_core: module param to limit msix vec allocation Arthur Kepner
[not found] ` <20100610165921.GF22247-sJ/iWh9BUns@public.gmane.org>
2010-06-13 6:53 ` Eli Cohen
2010-06-14 15:23 ` Arthur Kepner
2010-06-17 14:53 ` Yevgeny Petrilin
[not found] ` <E113D394D7C5DB4F8FF691FA7EE9DB4439A77CCD66-WQlSmcKwN8Te+A/uUDamNg@public.gmane.org>
2010-06-17 21:27 ` Arthur Kepner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.