From: YASUAKI ISHIMATSU <yasu.isimatu@gmail.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Kashyap Desai <kashyap.desai@broadcom.com>,
Hannes Reinecke <hare@suse.de>,
Marc Zyngier <marc.zyngier@arm.com>,
Christoph Hellwig <hch@lst.de>,
axboe@kernel.dk, mpe@ellerman.id.au, keith.busch@intel.com,
peterz@infradead.org, LKML <linux-kernel@vger.kernel.org>,
linux-scsi@vger.kernel.org,
Sumit Saxena <sumit.saxena@broadcom.com>,
Shivasharan Srikanteshwara
<shivasharan.srikanteshwara@broadcom.com>,
yasu.isimatu@gmail.com
Subject: Re: system hung up when offlining CPUs
Date: Tue, 10 Oct 2017 12:30:56 -0400 [thread overview]
Message-ID: <3d93387d-30eb-0434-2216-0e6435c633f8@gmail.com> (raw)
In-Reply-To: <alpine.DEB.2.20.1710042208400.2406@nanos>
Hi Thomas,
Sorry for the late reply.
I'll apply the patches and retest in this week.
Please wait a while.
Thanks,
Yasuaki Ishimatsu
On 10/04/2017 05:04 PM, Thomas Gleixner wrote:
> On Tue, 3 Oct 2017, Thomas Gleixner wrote:
>> Can you please apply the debug patch below.
>
> I found an issue with managed interrupts when the affinity mask of an
> managed interrupt spawns multiple CPUs. Explanation in the changelog
> below. I'm not sure that this cures the problems you have, but at least I
> could prove that it's not doing what it should do. The failure I'm seing is
> fixed, but I can't test that megasas driver due to -ENOHARDWARE.
>
> Can you please apply the patch below on top of Linus tree and retest?
>
> Please send me the outputs I asked you to provide last time in any case
> (success or fail).
>
> @block/scsi folks: Can you please run that through your tests as well?
>
> Thanks,
>
> tglx
>
> 8<-----------------------
> Subject: genirq/cpuhotplug: Enforce affinity setting on startup of managed irqs
> From: Thomas Gleixner <tglx@linutronix.de>
> Date: Wed, 04 Oct 2017 21:07:38 +0200
>
> Managed interrupts can end up in a stale state on CPU hotplug. If the
> interrupt is not targeting a single CPU, i.e. the affinity mask spawns
> multiple CPUs then the following can happen:
>
> After boot:
>
> dstate: 0x01601200
> IRQD_ACTIVATED
> IRQD_IRQ_STARTED
> IRQD_SINGLE_TARGET
> IRQD_AFFINITY_SET
> IRQD_AFFINITY_MANAGED
> node: 0
> affinity: 24-31
> effectiv: 24
> pending: 0
>
> After offlining CPU 31 - 24
>
> dstate: 0x01a31000
> IRQD_IRQ_DISABLED
> IRQD_IRQ_MASKED
> IRQD_SINGLE_TARGET
> IRQD_AFFINITY_SET
> IRQD_AFFINITY_MANAGED
> IRQD_MANAGED_SHUTDOWN
> node: 0
> affinity: 24-31
> effectiv: 24
> pending: 0
>
> Now CPU 25 gets onlined again, so it should get the effective interrupt
> affinity for this interruopt, but due to the x86 interrupt affinity setter
> restrictions this ends up after restarting the interrupt with:
>
> dstate: 0x01601300
> IRQD_ACTIVATED
> IRQD_IRQ_STARTED
> IRQD_SINGLE_TARGET
> IRQD_AFFINITY_SET
> IRQD_SETAFFINITY_PENDING
> IRQD_AFFINITY_MANAGED
> node: 0
> affinity: 24-31
> effectiv: 24
> pending: 24-31
>
> So the interrupt is still affine to CPU 24, which was the last CPU to go
> offline of that affinity set and the move to an online CPU within 24-31,
> in this case 25, is pending. This mechanism is x86/ia64 specific as those
> architectures cannot move interrupts from thread context and do this when
> an interrupt is actually handled. So the move is set to pending.
>
> Whats worse is that offlining CPU 25 again results in:
>
> dstate: 0x01601300
> IRQD_ACTIVATED
> IRQD_IRQ_STARTED
> IRQD_SINGLE_TARGET
> IRQD_AFFINITY_SET
> IRQD_SETAFFINITY_PENDING
> IRQD_AFFINITY_MANAGED
> node: 0
> affinity: 24-31
> effectiv: 24
> pending: 24-31
>
> This means the interrupt has not been shut down, because the outgoing CPU
> is not in the effective affinity mask, but of course nothing notices that
> the effective affinity mask is pointing at an offline CPU.
>
> In the case of restarting a managed interrupt the move restriction does not
> apply, so the affinity setting can be made unconditional. This needs to be
> done _before_ the interrupt is started up as otherwise the condition for
> moving it from thread context would not longer be fulfilled.
>
> With that change applied onlining CPU 25 after offlining 31-24 results in:
>
> dstate: 0x01600200
> IRQD_ACTIVATED
> IRQD_IRQ_STARTED
> IRQD_SINGLE_TARGET
> IRQD_AFFINITY_MANAGED
> node: 0
> affinity: 24-31
> effectiv: 25
> pending:
>
> And after offlining CPU 25:
>
> dstate: 0x01a30000
> IRQD_IRQ_DISABLED
> IRQD_IRQ_MASKED
> IRQD_SINGLE_TARGET
> IRQD_AFFINITY_MANAGED
> IRQD_MANAGED_SHUTDOWN
> node: 0
> affinity: 24-31
> effectiv: 25
> pending:
>
> which is the correct and expected result.
>
> To complete that, add some debug code to catch this kind of situation in
> the cpu offline code and warn about interrupt chips which allow affinity
> setting and do not update the effective affinity mask if that feature is
> enabled.
>
> Reported-by: YASUAKI ISHIMATSU <yasu.isimatu@gmail.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
>
> ---
> kernel/irq/chip.c | 2 +-
> kernel/irq/cpuhotplug.c | 28 +++++++++++++++++++++++++++-
> kernel/irq/manage.c | 17 +++++++++++++++++
> 3 files changed, 45 insertions(+), 2 deletions(-)
>
> --- a/kernel/irq/chip.c
> +++ b/kernel/irq/chip.c
> @@ -265,8 +265,8 @@ int irq_startup(struct irq_desc *desc, b
> irq_setup_affinity(desc);
> break;
> case IRQ_STARTUP_MANAGED:
> + irq_do_set_affinity(d, aff, false);
> ret = __irq_startup(desc);
> - irq_set_affinity_locked(d, aff, false);
> break;
> case IRQ_STARTUP_ABORT:
> return 0;
> --- a/kernel/irq/cpuhotplug.c
> +++ b/kernel/irq/cpuhotplug.c
> @@ -18,8 +18,34 @@
> static inline bool irq_needs_fixup(struct irq_data *d)
> {
> const struct cpumask *m = irq_data_get_effective_affinity_mask(d);
> + unsigned int cpu = smp_processor_id();
>
> - return cpumask_test_cpu(smp_processor_id(), m);
> +#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
> + /*
> + * The cpumask_empty() check is a workaround for interrupt chips,
> + * which do not implement effective affinity, but the architecture has
> + * enabled the config switch. Use the general affinity mask instead.
> + */
> + if (cpumask_empty(m))
> + m = irq_data_get_affinity_mask(d);
> +
> + /*
> + * Sanity check. If the mask is not empty when excluding the outgoing
> + * CPU then it must contain at least one online CPU. The outgoing CPU
> + * has been removed from the online mask already.
> + */
> + if (cpumask_any_but(m, cpu) < nr_cpu_ids &&
> + cpumask_any_and(m, cpu_online_mask) >= nr_cpu_ids) {
> + /*
> + * If this happens then there was a missed IRQ fixup at some
> + * point. Warn about it and enforce fixup.
> + */
> + pr_warn("Eff. affinity %*pbl of IRQ %u contains only offline CPUs after offlining CPU %u\n",
> + cpumask_pr_args(m), d->irq, cpu);
> + return true;
> + }
> +#endif
> + return cpumask_test_cpu(cpu, m);
> }
>
> static bool migrate_one_irq(struct irq_desc *desc)
> --- a/kernel/irq/manage.c
> +++ b/kernel/irq/manage.c
> @@ -168,6 +168,19 @@ void irq_set_thread_affinity(struct irq_
> set_bit(IRQTF_AFFINITY, &action->thread_flags);
> }
>
> +static void irq_validate_effective_affinity(struct irq_data *data)
> +{
> +#ifdef CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK
> + const struct cpumask *m = irq_data_get_effective_affinity_mask(data);
> + struct irq_chip *chip = irq_data_get_irq_chip(data);
> +
> + if (!cpumask_empty(m))
> + return;
> + pr_warn_once("irq_chip %s did not update eff. affinity mask of irq %u\n",
> + chip->name, data->irq);
> +#endif
> +}
> +
> int irq_do_set_affinity(struct irq_data *data, const struct cpumask *mask,
> bool force)
> {
> @@ -175,12 +188,16 @@ int irq_do_set_affinity(struct irq_data
> struct irq_chip *chip = irq_data_get_irq_chip(data);
> int ret;
>
> + if (!chip || !chip->irq_set_affinity)
> + return -EINVAL;
> +
> ret = chip->irq_set_affinity(data, mask, force);
> switch (ret) {
> case IRQ_SET_MASK_OK:
> case IRQ_SET_MASK_OK_DONE:
> cpumask_copy(desc->irq_common_data.affinity, mask);
> case IRQ_SET_MASK_OK_NOCOPY:
> + irq_validate_effective_affinity(data);
> irq_set_thread_affinity(desc);
> ret = 0;
> }
>
next prev parent reply other threads:[~2017-10-10 16:31 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-08-08 19:25 system hung up when offlining CPUs YASUAKI ISHIMATSU
2017-08-09 11:42 ` Marc Zyngier
2017-08-09 19:09 ` YASUAKI ISHIMATSU
2017-08-10 11:54 ` Marc Zyngier
2017-08-21 12:07 ` Christoph Hellwig
2017-08-21 13:18 ` Christoph Hellwig
2017-08-21 13:37 ` Marc Zyngier
2017-09-07 20:23 ` YASUAKI ISHIMATSU
2017-09-12 18:15 ` YASUAKI ISHIMATSU
2017-09-13 11:13 ` Hannes Reinecke
2017-09-13 11:35 ` Kashyap Desai
2017-09-13 13:33 ` Thomas Gleixner
2017-09-14 16:28 ` YASUAKI ISHIMATSU
2017-09-16 10:15 ` Thomas Gleixner
2017-09-16 15:02 ` Thomas Gleixner
2017-10-02 16:36 ` YASUAKI ISHIMATSU
2017-10-03 21:44 ` Thomas Gleixner
2017-10-04 21:04 ` Thomas Gleixner
2017-10-09 11:35 ` [tip:irq/urgent] genirq/cpuhotplug: Add sanity check for effective affinity mask tip-bot for Thomas Gleixner
2017-10-09 11:35 ` [tip:irq/urgent] genirq/cpuhotplug: Enforce affinity setting on startup of managed irqs tip-bot for Thomas Gleixner
2017-10-10 16:30 ` YASUAKI ISHIMATSU [this message]
2017-10-16 18:59 ` system hung up when offlining CPUs YASUAKI ISHIMATSU
2017-10-16 20:27 ` Thomas Gleixner
2017-10-30 9:08 ` Shivasharan Srikanteshwara
2017-11-01 0:47 ` Thomas Gleixner
2017-11-01 11:01 ` Hannes Reinecke
2017-10-04 21:10 ` Thomas Gleixner
-- strict thread matches above, loose matches on Subject: below --
2017-08-08 19:24 YASUAKI ISHIMATSU
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3d93387d-30eb-0434-2216-0e6435c633f8@gmail.com \
--to=yasu.isimatu@gmail.com \
--cc=axboe@kernel.dk \
--cc=hare@suse.de \
--cc=hch@lst.de \
--cc=kashyap.desai@broadcom.com \
--cc=keith.busch@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=marc.zyngier@arm.com \
--cc=mpe@ellerman.id.au \
--cc=peterz@infradead.org \
--cc=shivasharan.srikanteshwara@broadcom.com \
--cc=sumit.saxena@broadcom.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).