From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753452AbeAQPBw (ORCPT + 1 other); Wed, 17 Jan 2018 10:01:52 -0500 Received: from Galois.linutronix.de ([146.0.238.70]:46194 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752068AbeAQPBv (ORCPT ); Wed, 17 Jan 2018 10:01:51 -0500 Date: Wed, 17 Jan 2018 16:01:47 +0100 (CET) From: Thomas Gleixner To: Keith Busch cc: LKML Subject: Re: [BUG 4.15-rc7] IRQ matrix management errors In-Reply-To: <20180117142440.GC7562@localhost.localdomain> Message-ID: References: <20180116061641.GB32639@localhost.localdomain> <20180116071145.GA5643@localhost.localdomain> <20180117022511.GD6259@localhost.localdomain> <20180117075500.GB7562@localhost.localdomain> <20180117142440.GC7562@localhost.localdomain> User-Agent: Alpine 2.20 (DEB 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Return-Path: On Wed, 17 Jan 2018, Keith Busch wrote: > On Wed, Jan 17, 2018 at 10:32:12AM +0100, Thomas Gleixner wrote: > > On Wed, 17 Jan 2018, Thomas Gleixner wrote: > > > That doesn't sound right. The vectors should be spread evenly accross the > > > CPUs. So ENOSPC should never happen. > > > > > > Can you please take snapshots of /sys/kernel/debug/irq/ between the > > > modprobe and modprobe -r steps? > > > > The allocation fails because CPU1 has exhausted it's vector space here: > > > > [002] d... 333.028216: irq_matrix_alloc_managed: bit=34 cpu=1 online=1 avl=0 alloc=202 managed=2 online_maps=112 global_avl=22085, global_rsvd=158, total_alloc=460 > > > > Now the interesting question is how that happens. > > The trace with "trace_events=irq_matrix" kernel parameter is attached, > ended shortly after an allocation failure. Which device is allocating gazillions of non-managed interrupts? NetworkManager-2208 [044] d... 8.648608: irq_matrix_alloc: bit=68 cpu=0 online=1 avl=168 alloc=35 managed=3 online_maps=112 global_avl=22359, global_rsvd=532, total_alloc=215 .... NetworkManager-2208 [044] d... 8.665114: irq_matrix_alloc: bit=237 cpu=0 online=1 avl=0 alloc=203 managed=3 online_maps=112 global_avl=22191, global_rsvd=364, total_alloc=383 That's 168 interrupts total. Enterprise grade insanity. The patch below should cure that by spreading them out on allocation. Thanks, tglx 8<------------------ diff --git a/kernel/irq/matrix.c b/kernel/irq/matrix.c index 0ba0dd8863a7..5831cc7db27d 100644 --- a/kernel/irq/matrix.c +++ b/kernel/irq/matrix.c @@ -321,29 +321,38 @@ void irq_matrix_remove_reserved(struct irq_matrix *m) int irq_matrix_alloc(struct irq_matrix *m, const struct cpumask *msk, bool reserved, unsigned int *mapped_cpu) { - unsigned int cpu; + unsigned int cpu, best_cpu, maxavl = 0; + struct cpumap *cm; + unsigned int bit; + best_cpu = UINT_MAX; for_each_cpu(cpu, msk) { - struct cpumap *cm = per_cpu_ptr(m->maps, cpu); - unsigned int bit; + cm = per_cpu_ptr(m->maps, cpu); - if (!cm->online) + if (!cm->online || cm->available <= maxavl) continue; - bit = matrix_alloc_area(m, cm, 1, false); - if (bit < m->alloc_end) { - cm->allocated++; - cm->available--; - m->total_allocated++; - m->global_available--; - if (reserved) - m->global_reserved--; - *mapped_cpu = cpu; - trace_irq_matrix_alloc(bit, cpu, m, cm); - return bit; - } + best_cpu = cpu; + maxavl = cm->available; } - return -ENOSPC; + + if (!maxavl) + return -ENOSPC; + + cm = per_cpu_ptr(m->maps, best_cpu); + bit = matrix_alloc_area(m, cm, 1, false); + if (bit >= m->alloc_end) + return -ENOSPC; + + cm->allocated++; + cm->available--; + m->total_allocated++; + m->global_available--; + if (reserved) + m->global_reserved--; + *mapped_cpu = best_cpu; + trace_irq_matrix_alloc(bit, best_cpu, m, cm); + return bit; } /**