Re: RCU stalls when running out of memory on 3.14-rc4 w/ NFS and kernel threads priorities changed

[Florian Fainelli <f.fainelli@gmail.com>]

2014-03-04 21:34 GMT-08:00 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
> On Tue, Mar 04, 2014 at 07:55:03PM -0800, Florian Fainelli wrote:
>> 2014-03-04 17:43 GMT-08:00 Paul E. McKenney <paulmck@linux.vnet.ibm.com>:
>> > On Tue, Mar 04, 2014 at 05:16:27PM -0800, Florian Fainelli wrote:
>> >> 2014-03-04 17:03 GMT-08:00 Florian Fainelli <f.fainelli@gmail.com>:
>> >> > 2014-03-04 16:48 GMT-08:00 Eric Dumazet <eric.dumazet@gmail.com>:
>> >> >> On Tue, 2014-03-04 at 15:55 -0800, Florian Fainelli wrote:
>> >> >>> Hi all,
>> >> >>>
>> >> >>> I am seeing the following RCU stalls messages appearing on an ARMv7
>> >> >>> 4xCPUs system running 3.14-rc4:
>> >> >>>
>> >> >>> [   42.974327] INFO: rcu_sched detected stalls on CPUs/tasks:
>> >> >>> [   42.979839]  (detected by 0, t=2102 jiffies, g=4294967082,
>> >> >>> c=4294967081, q=516)
>> >> >>> [   42.987169] INFO: Stall ended before state dump start
>> >> >>>
>> >> >>> this is happening under the following conditions:
>> >> >>>
>> >> >>> - the attached bumper.c binary alters various kernel thread priorities
>> >> >>> based on the contents of bumpup.cfg and
>> >> >>> - malloc_crazy is running from a NFS share
>> >> >>> - malloc_crazy.c is running in a loop allocating chunks of memory but
>> >> >>> never freeing it
>> >> >>>
>> >> >>> when the priorities are altered, instead of getting the OOM killer to
>> >> >>> be invoked, the RCU stalls are happening. Taking NFS out of the
>> >> >>> equation does not allow me to reproduce the problem even with the
>> >> >>> priorities altered.
>> >> >>>
>> >> >>> This "problem" seems to have been there for quite a while now since I
>> >> >>> was able to get 3.8.13 to trigger that bug as well, with a slightly
>> >> >>> more detailed RCU debugging trace which points the finger at kswapd0.
>> >> >>>
>> >> >>> You should be able to get that reproduced under QEMU with the
>> >> >>> Versatile Express platform emulating a Cortex A15 CPU and the attached
>> >> >>> files.
>> >> >>>
>> >> >>> Any help or suggestions would be greatly appreciated. Thanks!
>> >> >>
>> >> >> Do you have a more complete trace, including stack traces ?
>> >> >
>> >> > Attatched is what I get out of SysRq-t, which is the only thing I have
>> >> > (note that the kernel is built with CONFIG_RCU_CPU_STALL_INFO=y):
>> >>
>> >> QEMU for Versatile Express w/ 2 CPUs yields something slightly
>> >> different than the real HW platform this is happening with, but it
>> >> does produce the RCU stall anyway:
>> >>
>> >> [  125.762946] BUG: soft lockup - CPU#1 stuck for 53s! [malloc_crazy:91]
>> >
>> > This soft-lockup condition can result in RCU CPU stall warnings.  Fix
>> > the problem causing the soft lockup, and I bet that your RCU CPU stall
>> > warnings go away.
>>
>> I definitively agree, which is why I was asking for help, as I think
>> the kernel thread priority change is what is causing the soft lockup
>> to appear, but nothing obvious jumps to mind when looking at the
>> trace.
>
> Is your hardware able to make the malloc_crazy CPU periodically dump
> its stack, perhaps in response to an NMI?  If not, another approach is
> to use ftrace -- though this will require a very high-priority task to
> turn tracing on and off reasonably quickly, unless you happen to have
> a very large amount of storage to hold the trace.
>
> What happens if you malloc() less intensively?  Does that avoid this
> problem?

It does yes, putting some arbitrary delays between the malloc() calls
does definitively help.

>The reason I ask is that you mentioned that avoiding NFS helped,
> and it is possible that NFS is increasing storage-access latencies and
> thus triggering another problem.  It is quite possible that slowing
> down the malloc()s would also help, and might allow you to observe what
> is happening more easily than when the system is driven fully to the
> lockup condition.
>
> Finally, what are you trying to achieve with this workload?  Does your
> production workload behave in this way?  Or is this an experimental
> investigation of some sort?

This is an experimental investigation, part of the problem being that
there were some expectations that altering priority of essential
kernel threads would "just work".

It seemed to me like even if we moved kthreadd to SCHED_RR, with
priority 2 (as shown by /proc/*/sched), we should still be at a more
favorable scheduling class than 'rcu_bh' and 'rcu_sched' which are on
SCHED_NORMAL. Interestingly, the issue only appears with 1 or 2 CPUs
online, as soon as the 4 are online I am no longer able to reproduce
it...
-- 
Florian