From: Heiner Kallweit <hkallweit1@gmail.com>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>,
Anna-Maria Gleixner <anna-maria@linutronix.de>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Grygorii Strashko <grygorii.strashko@ti.com>
Subject: Re: Fix 80d20d35af1e ("nohz: Fix local_timer_softirq_pending()") may have revealed another problem
Date: Fri, 28 Dec 2018 07:39:32 +0100 [thread overview]
Message-ID: <5aa51fc1-5a5c-0c61-5c28-0d9ca98e4514@gmail.com> (raw)
In-Reply-To: <d9647487-28db-d138-ae3e-3fd0d2fbe589@gmail.com>
On 28.12.2018 07:34, Heiner Kallweit wrote:
> On 28.12.2018 02:31, Frederic Weisbecker wrote:
>> On Fri, Dec 28, 2018 at 12:11:12AM +0100, Heiner Kallweit wrote:
>>>
> [...]
>>
>> Interesting, the softirq is raised from hardirq but it's not handled in the end of
>> the IRQ. Are you running threaded IRQS by any chance? If so I would expect ksoftirqd
>> to handle the pending work before we go idle. However I can imagine a small window
>> where such an expectation may not be met: if the softirq is raised after the ksoftirqd
>> thread is parked (CPUHP_AP_SMPBOOT_THREADS), which is right before we disable the CPU
>> (CPUHP_TEARDOWN_CPU).
>>
> I have a network driver (r8169) using NAPI which runs in softirq context AFAIK.
> For testing purposes I sometimes trigger system suspend via network, so there is
> network adapter activity when system suspends. Apart from that nothing really
> exciting:
> CPU0 CPU1 CPU2 CPU3
> 0: 43 0 0 0 IO-APIC 2-edge timer
> 1: 4 0 0 0 IO-APIC 1-edge i8042
> 8: 0 1 0 0 IO-APIC 8-fasteoi rtc0
> 9: 0 0 0 0 IO-APIC 9-fasteoi acpi
> 12: 0 0 0 5 IO-APIC 12-edge i8042
> 120: 0 0 0 0 PCI-MSI 311296-edge PCIe PME
> 121: 0 0 0 0 PCI-MSI 315392-edge PCIe PME
> 122: 0 0 0 0 PCI-MSI 327680-edge PCIe PME
> 123: 0 0 3328 0 PCI-MSI 294912-edge ahci[0000:00:12.0]
> 124: 0 133 0 0 PCI-MSI 344064-edge xhci_hcd
> 125: 0 0 32 0 PCI-MSI 245760-edge mei_me
> 127: 381 0 0 0 PCI-MSI 1572864-edge enp3s0
> 128: 0 0 0 236 PCI-MSI 32768-edge i915
> 129: 0 374 0 0 PCI-MSI 229376-edge snd_hda_intel:card0
>
>> I don't know if we can afford to ignore a softirq even at this late stage. We should
>> probably avoid leaking any. So here is a possible fix, if you don't mind trying:
>>
> I tested your patch and at least in the first minutes of testing couldn't reproduce
> the issue any longer. I tested manual system suspend and the following script you
> sent when we started to analyze the issue.
>
Also after some more time the issue didn't occur again. So it seems your analysis
was right and also the approach to fix it. Thanks!
Will let you know in case the issue should pop up again under special
circumstances.
> Heiner
>
> --------------------------------------------------------------------------
>
> #!/bin/bash
>
> do_hotplug()
> {
> for i in $(seq 1 $2)
> do
> echo $1 > /sys/devices/system/cpu/cpu$i/online
> done
> }
>
> LAST_CPU=$(($(nproc)-1))
>
> while true
> do
> do_hotplug 0 $LAST_CPU
> do_hotplug 1 $LAST_CPU
> done
>
next prev parent reply other threads:[~2018-12-28 6:39 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-16 6:13 Fix 80d20d35af1e ("nohz: Fix local_timer_softirq_pending()") may have revealed another problem Heiner Kallweit
2018-08-18 11:26 ` Thomas Gleixner
2018-08-18 22:34 ` Heiner Kallweit
2018-08-24 4:12 ` Frederic Weisbecker
2018-08-24 5:59 ` Heiner Kallweit
2018-08-24 8:01 ` Thomas Gleixner
2018-08-24 14:30 ` Frederic Weisbecker
2018-08-24 17:06 ` Heiner Kallweit
2018-08-28 2:25 ` Frederic Weisbecker
2018-09-27 16:05 ` Thomas Gleixner
2018-09-28 13:18 ` Frederic Weisbecker
2018-09-28 20:35 ` Heiner Kallweit
2018-10-15 20:58 ` Heiner Kallweit
2018-12-24 21:11 ` Heiner Kallweit
2018-12-27 6:53 ` Frederic Weisbecker
2018-12-27 23:11 ` Heiner Kallweit
2018-12-28 1:31 ` Frederic Weisbecker
2018-12-28 6:34 ` Heiner Kallweit
2018-12-28 6:39 ` Heiner Kallweit [this message]
2019-01-09 22:20 ` Heiner Kallweit
2019-01-11 21:36 ` Frederic Weisbecker
2019-01-16 6:24 ` Frederic Weisbecker
2019-01-16 18:42 ` Heiner Kallweit
2019-01-24 19:37 ` Heiner Kallweit
2019-02-14 19:05 ` Heiner Kallweit
2019-02-14 21:47 ` Thomas Gleixner
2019-02-14 22:33 ` Heiner Kallweit
2019-02-15 0:31 ` Frederic Weisbecker
2019-02-16 9:14 ` Heiner Kallweit
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5aa51fc1-5a5c-0c61-5c28-0d9ca98e4514@gmail.com \
--to=hkallweit1@gmail.com \
--cc=anna-maria@linutronix.de \
--cc=frederic@kernel.org \
--cc=grygorii.strashko@ti.com \
--cc=linux-kernel@vger.kernel.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).