From: "Christopher S. Aker" <caker@theshore.net>
To: Michal Hocko <mhocko@kernel.org>
Cc: Donald Buczek <buczek@molgen.mpg.de>,
dvteam@molgen.mpg.de, Paul Menzel <pmenzel@molgen.mpg.de>,
linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org,
Josh Triplett <josh@joshtriplett.org>
Subject: Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`
Date: Sun, 27 Nov 2016 00:32:52 -0500 [thread overview]
Message-ID: <E9180081-785C-4C1A-85F5-1B63960D2187@theshore.net> (raw)
In-Reply-To: <20161124101525.GB20668@dhcp22.suse.cz>
> On Nov 24, 2016, at 5:15 AM, Michal Hocko <mhocko@kernel.org> wrote:
>
>> * No rcu_* warnings on that machine with 4.7.2, but with 4.8.4 , 4.8.6 ,
>> 4.8.8 and now 4.9.0-rc5+Pauls patch
>
> I assume you haven't tried the Linus 4.8 kernel without any further
> stable patches? Just to be sure we are not talking about some later
> regression which found its way to the stable tree.
We are also seeing this frequently on our fleet since moving from 4.7.x to 4.8. This is from a machine running vanilla 4.8.6 just a few moments ago:
INFO: rcu_sched detected stalls on CPUs/tasks:
13-...: (420 ticks this GP) idle=ce1/140000000000000/0 softirq=225550784/225550904 fqs=87105
(detected by 26, t=600030 jiffies, g=68185325, c=68185324, q=344996)
Task dump for CPU 13:
kswapd1 R running task 12200 1840 2 0x00000808
0000000000000001 0000000000000034 000000000000012b 0000000000003139
ffff8b643fffb000 ffff8b028cee7cf8 ffff8b028cee7cf8 ffff8b028cee7d08
ffff8b028cee7d08 ffff8b028cee7d18 ffff8b028cee7d18 ffff8b0200000000
Call Trace:
[] ? shrink_node+0xcd/0x2f0
[] ? kswapd+0x304/0x710
[] ? mem_cgroup_shrink_node+0x160/0x160
[] ? kthread+0xc4/0xe0
[] ? ret_from_fork+0x1f/0x40
[] ? kthread_worker_fn+0x140/0x140
The machine will lag terribly during these occurrences .. some will eventually recover, some will spiral down and require a reboot.
-Chris
next prev parent reply other threads:[~2016-11-27 5:42 UTC|newest]
Thread overview: 94+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <24c226a5-1a4a-173e-8b4e-5107a2baac04@molgen.mpg.de>
2016-11-08 12:22 ` INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node` Paul Menzel
2016-11-08 17:03 ` Paul E. McKenney
2016-11-08 17:38 ` Paul Menzel
2016-11-08 18:39 ` Paul E. McKenney
2016-11-16 17:01 ` Paul Menzel
2016-11-16 17:30 ` Paul E. McKenney
2016-11-21 13:41 ` Michal Hocko
2016-11-21 14:01 ` Paul E. McKenney
2016-11-21 14:18 ` Michal Hocko
2016-11-21 14:29 ` Paul E. McKenney
2016-11-21 15:35 ` Donald Buczek
2016-11-24 10:15 ` Michal Hocko
2016-11-24 18:50 ` Donald Buczek
2016-11-27 9:37 ` Paul Menzel
2016-11-27 5:32 ` Christopher S. Aker [this message]
2016-11-27 9:19 ` Donald Buczek
2016-11-28 11:04 ` Michal Hocko
2016-11-28 12:26 ` Paul Menzel
2016-11-28 12:26 ` Paul Menzel
2016-11-30 10:28 ` Donald Buczek
2016-11-30 10:28 ` Donald Buczek
2016-11-30 11:09 ` Michal Hocko
2016-11-30 11:09 ` Michal Hocko
2016-11-30 11:43 ` Donald Buczek
2016-11-30 11:43 ` Donald Buczek
2016-12-02 9:14 ` Donald Buczek
2016-12-02 9:14 ` Donald Buczek
2016-12-06 8:32 ` Donald Buczek
2016-12-06 8:32 ` Donald Buczek
2016-11-30 11:53 ` Paul E. McKenney
2016-11-30 11:53 ` Paul E. McKenney
2016-11-30 11:54 ` Paul E. McKenney
2016-11-30 11:54 ` Paul E. McKenney
2016-11-30 12:31 ` Paul Menzel
2016-11-30 12:31 ` Paul Menzel
2016-11-30 14:31 ` Paul E. McKenney
2016-11-30 14:31 ` Paul E. McKenney
2016-11-30 13:19 ` Michal Hocko
2016-11-30 13:19 ` Michal Hocko
2016-11-30 14:29 ` Paul E. McKenney
2016-11-30 14:29 ` Paul E. McKenney
2016-11-30 16:38 ` Peter Zijlstra
2016-11-30 16:38 ` Peter Zijlstra
2016-11-30 17:02 ` Paul E. McKenney
2016-11-30 17:02 ` Paul E. McKenney
2016-11-30 17:05 ` Michal Hocko
2016-11-30 17:05 ` Michal Hocko
2016-11-30 17:23 ` Paul E. McKenney
2016-11-30 17:23 ` Paul E. McKenney
2016-11-30 17:34 ` Michal Hocko
2016-11-30 17:34 ` Michal Hocko
2016-11-30 17:50 ` Peter Zijlstra
2016-11-30 17:50 ` Peter Zijlstra
2016-11-30 19:40 ` Paul E. McKenney
2016-11-30 19:40 ` Paul E. McKenney
2016-12-01 5:30 ` Peter Zijlstra
2016-12-01 5:30 ` Peter Zijlstra
2016-12-01 12:40 ` Paul E. McKenney
2016-12-01 12:40 ` Paul E. McKenney
2016-12-01 16:36 ` Peter Zijlstra
2016-12-01 16:36 ` Peter Zijlstra
2016-12-01 16:59 ` Paul E. McKenney
2016-12-01 16:59 ` Paul E. McKenney
2016-12-01 18:09 ` Peter Zijlstra
2016-12-01 18:09 ` Peter Zijlstra
2016-12-01 18:42 ` Paul E. McKenney
2016-12-01 18:42 ` Paul E. McKenney
2016-12-01 18:49 ` Peter Zijlstra
2016-12-01 18:49 ` Peter Zijlstra
[not found] <d6981bac-8e97-b482-98c0-40949db03ca3@kernelpanic.ru>
[not found] ` <20161124133019.GE3612@linux.vnet.ibm.com>
[not found] ` <de88a72a-f861-b51f-9fb3-4265378702f1@kernelpanic.ru>
[not found] ` <20161125212000.GI31360@linux.vnet.ibm.com>
[not found] ` <20161128095825.GI14788@dhcp22.suse.cz>
[not found] ` <20161128105425.GY31360@linux.vnet.ibm.com>
[not found] ` <3a4242cb-0198-0a3b-97ae-536fb5ff83ec@kernelpanic.ru>
[not found] ` <20161128143435.GC3924@linux.vnet.ibm.com>
2016-11-28 14:40 ` Boris Zhmurov
2016-11-28 15:05 ` Paul E. McKenney
2016-11-28 19:16 ` Boris Zhmurov
2016-11-29 18:59 ` Paul E. McKenney
2016-11-30 17:41 ` Boris Zhmurov
2016-11-30 17:48 ` Michal Hocko
2016-11-30 18:12 ` Boris Zhmurov
2016-11-30 18:25 ` Michal Hocko
2016-11-30 18:26 ` Boris Zhmurov
2016-12-01 18:10 ` Boris Zhmurov
2016-12-01 19:39 ` Paul E. McKenney
2016-12-01 19:39 ` Paul E. McKenney
2016-12-02 9:37 ` Michal Hocko
2016-12-02 9:37 ` Michal Hocko
2016-12-02 13:52 ` Paul E. McKenney
2016-12-02 13:52 ` Paul E. McKenney
2016-12-02 16:39 ` Boris Zhmurov
2016-12-02 16:39 ` Boris Zhmurov
2016-12-02 16:44 ` Paul E. McKenney
2016-12-02 16:44 ` Paul E. McKenney
2016-12-02 17:02 ` Michal Hocko
2016-12-02 17:02 ` Michal Hocko
2016-12-02 17:15 ` Paul E. McKenney
2016-12-02 17:15 ` Paul E. McKenney
2016-11-30 19:42 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=E9180081-785C-4C1A-85F5-1B63960D2187@theshore.net \
--to=caker@theshore.net \
--cc=buczek@molgen.mpg.de \
--cc=dvteam@molgen.mpg.de \
--cc=josh@joshtriplett.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=mhocko@kernel.org \
--cc=pmenzel@molgen.mpg.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.