From: Shung-Hsi Yu <shung-hsi.yu@suse.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Prarit Bhargava <prarit@redhat.com>,
linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
Borislav Petkov <bp@alien8.de>,
x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Marc Zyngier <maz@kernel.org>, Andy Lutomirski <luto@kernel.org>,
Ming Lei <ming.lei@redhat.com>, Peter Xu <peterx@redhat.com>
Subject: Re: "irq 4: Affinity broken due to vector space exhaustion." warning on restart of ttyS0 console
Date: Wed, 9 Dec 2020 14:33:04 +0800 [thread overview]
Message-ID: <20201209063304.GF23060@syu-laptop> (raw)
In-Reply-To: <871rh1gcck.fsf@nanos.tec.linutronix.de>
Hi Thomas,
On Tue, Nov 10, 2020 at 09:56:27PM +0100, Thomas Gleixner wrote:
> The real problem is irqbalanced aggressively exhausting the vector space
> of a _whole_ socket to the point that there is not a single vector left
> for serial. That's the problem you want to fix.
I believe this warning also gets triggered even when there's _no_ vector
exhaustion.
This seem to happen when the IRQ's affinity mask is set (wrongly) to CPUs on
a different NUMA node (e.g. cpumask_of_node(1) when the irqd->irq == 0).
$ lscpu
...
NUMA node0 CPU(s): 0-25,52-77
NUMA node1 CPU(s): 26-51,78-103
$ cat /sys/kernel/debug/tracing/trace
...
irqbalance-1994 [017] d... 74.912799: irq_matrix_alloc: bit=33 cpu=26 online=1 avl=198 alloc=3 managed=1 online_maps=104 global_avl=20687, global_rsvd=341, total_alloc=217
irqbalance-1994 [017] d... 74.912802: vector_alloc: irq=4 vector=33 reserved=0 ret=0
irqbalance-1994 [017] d... 74.912804: vector_update: irq=4 vector=33 cpu=26 prev_vector=33 prev_cpu=7
irqbalance-1994 [017] d... 74.912805: vector_config: irq=4 vector=33 cpu=26 apicdest=0x00000040
<idle>-0 [007] d.h. 74.970733: vector_free_moved: irq=4 cpu=7 vector=33 is_managed=0
<idle>-0 [007] d.h. 74.970738: irq_matrix_free: bit=33 cpu=7 online=1 avl=200 alloc=1 managed=1 online_maps=104 global_avl=20687, global_rsvd=341, total_alloc=217
...
(agetty)-3004 [047] d... 81.731231: vector_deactivate: irq=4 is_managed=0 can_reserve=1 reserve=0
(agetty)-3004 [047] d... 81.738035: vector_clear: irq=4 vector=33 cpu=26 prev_vector=0 prev_cpu=7
(agetty)-3004 [047] d... 81.738040: irq_matrix_free: bit=33 cpu=26 online=1 avl=199 alloc=2 managed=1 online_maps=104 global_avl=20689, global_rsvd=341, total_alloc=215
(agetty)-3004 [047] d... 81.738046: irq_matrix_reserve: online_maps=104 global_avl=20689, global_rsvd=342, total_alloc=215
(agetty)-3004 [047] d... 81.766739: vector_reserve: irq=4 ret=0
(agetty)-3004 [047] d... 81.766741: vector_config: irq=4 vector=239 cpu=0 apicdest=0x00000000
(agetty)-3004 [047] d... 81.777152: vector_activate: irq=4 is_managed=0 can_reserve=1 reserve=0
(agetty)-3004 [047] d... 81.777157: vector_alloc: irq=4 vector=0 reserved=1 ret=-22
----------------------------------------> irq_matrix_alloc() failed with
EINVAL because the cpumask
passed in is empty, which is a
result of affmask being
(ff,ffffc000,000fffff,fc000000)
and cpumask_of_node(node)
being
(00,00003fff,fff00000,03ffffff).
(agetty)-3004 [047] d... 81.789349: irq_matrix_alloc: bit=33 cpu=1 online=1 avl=199 alloc=2 managed=1 online_maps=104 global_avl=20688, global_rsvd=341, total_alloc=216
(agetty)-3004 [047] d... 81.789351: vector_alloc: irq=4 vector=33 reserved=1 ret=0
(agetty)-3004 [047] d... 81.789353: vector_update: irq=4 vector=33 cpu=1 prev_vector=0 prev_cpu=26
(agetty)-3004 [047] d... 81.789355: vector_config: irq=4 vector=33 cpu=1 apicdest=0x00000002
----------------------------------------> "irq 4: Affinity broken due to
vector space exhaustion."
warning shows up
(agetty)-3004 [047] d... 81.900783: irq_matrix_alloc: bit=33 cpu=26 online=1 avl=198 alloc=3 managed=1 online_maps=104 global_avl=20687, global_rsvd=341, total_alloc=217
(agetty)-3004 [047] d... 82.053535: vector_alloc: irq=4 vector=33 reserved=0 ret=0
(agetty)-3004 [047] d... 82.053536: vector_update: irq=4 vector=33 cpu=26 prev_vector=33 prev_cpu=1
(agetty)-3004 [047] d... 82.053538: vector_config: irq=4 vector=33 cpu=26 apicdest=0x00000040
Shung-Hsi Yu
next prev parent reply other threads:[~2020-12-09 6:34 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-10 19:24 "irq 4: Affinity broken due to vector space exhaustion." warning on restart of ttyS0 console Prarit Bhargava
2020-11-10 20:56 ` Thomas Gleixner
2020-11-11 16:14 ` Prarit Bhargava
2020-12-09 6:33 ` Shung-Hsi Yu [this message]
2020-12-09 7:10 ` Shung-Hsi Yu
2020-12-09 18:28 ` Thomas Gleixner
2020-12-10 6:18 ` Shung-Hsi Yu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201209063304.GF23060@syu-laptop \
--to=shung-hsi.yu@suse.com \
--cc=bp@alien8.de \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=maz@kernel.org \
--cc=ming.lei@redhat.com \
--cc=mingo@redhat.com \
--cc=peterx@redhat.com \
--cc=prarit@redhat.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).