From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755912AbeARKoY (ORCPT ); Thu, 18 Jan 2018 05:44:24 -0500 Received: from terminus.zytor.com ([65.50.211.136]:42065 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754788AbeARKoV (ORCPT ); Thu, 18 Jan 2018 05:44:21 -0500 Date: Thu, 18 Jan 2018 02:43:12 -0800 From: tip-bot for Thomas Gleixner Message-ID: Cc: hpa@zytor.com, keith.busch@intel.com, mingo@kernel.org, tglx@linutronix.de, linux-kernel@vger.kernel.org Reply-To: mingo@kernel.org, tglx@linutronix.de, linux-kernel@vger.kernel.org, hpa@zytor.com, keith.busch@intel.com In-Reply-To: References: To: linux-tip-commits@vger.kernel.org Subject: [tip:irq/urgent] irq/matrix: Spread interrupts on allocation Git-Commit-ID: a0c9259dc4e1923a98356967ce8b732da1979df8 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: a0c9259dc4e1923a98356967ce8b732da1979df8 Gitweb: https://git.kernel.org/tip/a0c9259dc4e1923a98356967ce8b732da1979df8 Author: Thomas Gleixner AuthorDate: Wed, 17 Jan 2018 16:01:47 +0100 Committer: Thomas Gleixner CommitDate: Thu, 18 Jan 2018 11:38:41 +0100 irq/matrix: Spread interrupts on allocation Keith reported an issue with vector space exhaustion on a server machine which is caused by the i40e driver allocating 168 MSI interrupts when the driver is initialized, even when most of these interrupts are not used at all. The x86 vector allocation code tries to avoid the immediate allocation with the reservation mode, but the card uses MSI and does not support MSI entry masking, which prevents reservation mode and requires immediate vector allocation. The matrix allocator is a bit naive and prefers the first CPU in the cpumask which describes the possible target CPUs for an allocation. That results in allocating all 168 vectors on CPU0 which later causes vector space exhaustion when the NVMe driver tries to allocate managed interrupts on each CPU for the per CPU queues. Avoid this by finding the CPU which has the lowest vector allocation count to spread out the non managed interrupt accross the possible target CPUs. Fixes: 2f75d9e1c905 ("genirq: Implement bitmap matrix allocator") Reported-by: Keith Busch Signed-off-by: Thomas Gleixner Tested-by: Keith Busch Link: https://lkml.kernel.org/r/alpine.DEB.2.20.1801171557330.1777@nanos --- kernel/irq/matrix.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/kernel/irq/matrix.c b/kernel/irq/matrix.c index 0ba0dd8..5187dfe 100644 --- a/kernel/irq/matrix.c +++ b/kernel/irq/matrix.c @@ -321,15 +321,23 @@ void irq_matrix_remove_reserved(struct irq_matrix *m) int irq_matrix_alloc(struct irq_matrix *m, const struct cpumask *msk, bool reserved, unsigned int *mapped_cpu) { - unsigned int cpu; + unsigned int cpu, best_cpu, maxavl = 0; + struct cpumap *cm; + unsigned int bit; + best_cpu = UINT_MAX; for_each_cpu(cpu, msk) { - struct cpumap *cm = per_cpu_ptr(m->maps, cpu); - unsigned int bit; + cm = per_cpu_ptr(m->maps, cpu); - if (!cm->online) + if (!cm->online || cm->available <= maxavl) continue; + best_cpu = cpu; + maxavl = cm->available; + } + + if (maxavl) { + cm = per_cpu_ptr(m->maps, best_cpu); bit = matrix_alloc_area(m, cm, 1, false); if (bit < m->alloc_end) { cm->allocated++; @@ -338,8 +346,8 @@ int irq_matrix_alloc(struct irq_matrix *m, const struct cpumask *msk, m->global_available--; if (reserved) m->global_reserved--; - *mapped_cpu = cpu; - trace_irq_matrix_alloc(bit, cpu, m, cm); + *mapped_cpu = best_cpu; + trace_irq_matrix_alloc(bit, best_cpu, m, cm); return bit; } }