From: Marc Zyngier <maz@kernel.org>
To: John Garry <john.garry@huawei.com>
Cc: Thomas Gleixner <tglx@linutronix.de>,
chenxiang <chenxiang66@hisilicon.com>,
Shameer Kolothum <shameerali.kolothum.thodi@huawei.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"liuqi (BA)" <liuqi115@huawei.com>, <wangxiongfeng2@huawei.com>,
David Decotigny <decot@google.com>
Subject: Re: PCI MSI issue for maxcpus=1
Date: Sat, 05 Mar 2022 15:40:25 +0000 [thread overview]
Message-ID: <87a6e4tnkm.wl-maz@kernel.org> (raw)
In-Reply-To: <1cbe7daa-8003-562b-06fa-5a50f7ee6ed2@huawei.com>
[+ David, who was chasing something similar]
Hi John,
On Fri, 04 Mar 2022 12:53:31 +0000,
John Garry <john.garry@huawei.com> wrote:
>
> > ...
>
> >
> > [ 7.961007] valid_col+0x14/0x24
> > [ 7.964223] its_send_single_command+0x4c/0x150
> > [ 7.968741] its_irq_domain_activate+0xc8/0x104
> > [ 7.973259] __irq_domain_activate_irq+0x5c/0xac
> > [ 7.977865] __irq_domain_activate_irq+0x38/0xac
> > [ 7.982471] irq_domain_activate_irq+0x3c/0x64
> > [ 7.986902] __msi_domain_alloc_irqs+0x1a8/0x2f4
> > [ 7.991507] msi_domain_alloc_irqs+0x20/0x2c
> > [ 7.995764] __pci_enable_msi_range+0x2ec/0x590
> > [ 8.000284] pci_alloc_irq_vectors_affinity+0xe0/0x140
> > [ 8.005410] hisi_sas_v3_probe+0x300/0xbe0
> > [ 8.009494] local_pci_probe+0x44/0xb0
> > [ 8.013232] work_for_cpu_fn+0x20/0x34
> > [ 8.016969] process_one_work+0x1d0/0x354
> > [ 8.020966] worker_thread+0x2c0/0x470
> > [ 8.024703] kthread+0x17c/0x190
> > [ 8.027920] ret_from_fork+0x10/0x20
> > [ 8.031485] ---[ end trace bb67cfc7eded7361 ]---
> >
>
> ...
>
> > Ah, of course. the CPU hasn't booted yet, so its collection isn't
> > mapped. I was hoping that the core code would keep the interrupt in
> > shutdown state, but it doesn't seem to be the case...
> >
> > > Apart from this, I assume that if another cpu comes online later in
> > > the affinity mask I would figure that we want to target the irq to
> > > that cpu (which I think we would not do here).
> >
> > That's probably also something that should come from core code, as
> > we're not really in a position to decide this in the ITS driver.
> > .
>
>
> Hi Marc,
>
> Have you had a chance to consider this issue further?
>
> So I think that x86 avoids this issue as it uses matrix.c, which
> handles CPUs being offline when selecting target CPUs for managed
> interrupts.
>
> So is your idea still that core code should keep the interrupt in
> shutdown state (for no CPUs online in affinity mask)?
Yup. I came up with this:
diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index 2bdfce5edafd..97e9eb9aecc6 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -823,6 +823,19 @@ static int msi_init_virq(struct irq_domain *domain, int virq, unsigned int vflag
if (!(vflags & VIRQ_ACTIVATE))
return 0;
+ if (!(vflags & VIRQ_CAN_RESERVE)) {
+ /*
+ * If the interrupt is managed but no CPU is available
+ * to service it, shut it down until better times.
+ */
+ if (irqd_affinity_is_managed(irqd) &&
+ !cpumask_intersects(irq_data_get_affinity_mask(irqd),
+ cpu_online_mask)) {
+ irqd_set_managed_shutdown(irqd);
+ return 0;
+ }
+ }
+
ret = irq_domain_activate_irq(irqd, vflags & VIRQ_CAN_RESERVE);
if (ret)
return ret;
With this in place, I get the following results (VM booted with 4
vcpus and maxcpus=1, the virtio device is using managed interrupts):
root@debian:~# cat /proc/interrupts
CPU0
10: 2298 GICv3 27 Level arch_timer
12: 84 GICv3 33 Level uart-pl011
49: 0 GICv3 41 Edge ACPI:Ged
50: 0 ITS-MSI 16384 Edge virtio0-config
51: 2088 ITS-MSI 16385 Edge virtio0-req.0
52: 0 ITS-MSI 16386 Edge virtio0-req.1
53: 0 ITS-MSI 16387 Edge virtio0-req.2
54: 0 ITS-MSI 16388 Edge virtio0-req.3
55: 11641 ITS-MSI 32768 Edge xhci_hcd
56: 0 ITS-MSI 32769 Edge xhci_hcd
IPI0: 0 Rescheduling interrupts
IPI1: 0 Function call interrupts
IPI2: 0 CPU stop interrupts
IPI3: 0 CPU stop (for crash dump) interrupts
IPI4: 0 Timer broadcast interrupts
IPI5: 0 IRQ work interrupts
IPI6: 0 CPU wake-up interrupts
Err: 0
root@debian:~# echo 1 >/sys/devices/system/cpu/cpu2/online
root@debian:~# cat /proc/interrupts
CPU0 CPU2
10: 2530 90 GICv3 27 Level arch_timer
12: 103 0 GICv3 33 Level uart-pl011
49: 0 0 GICv3 41 Edge ACPI:Ged
50: 0 0 ITS-MSI 16384 Edge virtio0-config
51: 2097 0 ITS-MSI 16385 Edge virtio0-req.0
52: 0 0 ITS-MSI 16386 Edge virtio0-req.1
53: 0 12 ITS-MSI 16387 Edge virtio0-req.2
54: 0 0 ITS-MSI 16388 Edge virtio0-req.3
55: 13487 0 ITS-MSI 32768 Edge xhci_hcd
56: 0 0 ITS-MSI 32769 Edge xhci_hcd
IPI0: 38 45 Rescheduling interrupts
IPI1: 3 3 Function call interrupts
IPI2: 0 0 CPU stop interrupts
IPI3: 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 Timer broadcast interrupts
IPI5: 0 0 IRQ work interrupts
IPI6: 0 0 CPU wake-up interrupts
Err: 0
Would this solve your problem?
Thanks,
M.
--
Without deviation from the norm, progress is not possible.
next prev parent reply other threads:[~2022-03-05 15:40 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-05 11:23 PCI MSI issue for maxcpus=1 John Garry
2022-01-06 15:49 ` Marc Zyngier
2022-01-07 11:24 ` John Garry
2022-01-16 12:07 ` Marc Zyngier
2022-01-17 9:14 ` Marc Zyngier
2022-01-17 11:59 ` John Garry
2022-01-24 11:22 ` Marc Zyngier
2022-03-04 12:53 ` John Garry
2022-03-05 15:40 ` Marc Zyngier [this message]
2022-03-07 13:48 ` John Garry
2022-03-07 14:01 ` Marc Zyngier
2022-03-07 14:03 ` Marc Zyngier
2022-03-08 1:37 ` David Decotigny
2022-03-08 3:57 ` Xiongfeng Wang
[not found] ` <87zgm0zfw7.wl-maz@kernel.org>
2022-03-10 3:19 ` Xiongfeng Wang
[not found] ` <87o82eyxmz.wl-maz@kernel.org>
2022-03-10 12:58 ` Xiongfeng Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87a6e4tnkm.wl-maz@kernel.org \
--to=maz@kernel.org \
--cc=chenxiang66@hisilicon.com \
--cc=decot@google.com \
--cc=john.garry@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=liuqi115@huawei.com \
--cc=shameerali.kolothum.thodi@huawei.com \
--cc=tglx@linutronix.de \
--cc=wangxiongfeng2@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.