Memory exhaustion testing?

Memory exhaustion testing?

[Jesper Dangaard Brouer]

Hi MM-people,

How do you/we test the error paths when the system runs out of memory?

What kind of tools do you use?
or Any tricks to provoke this?

For testing my recent change to the SLUB allocator, I've implemented a
crude kernel module that tries to allocate all memory, so I can test the
error code-path in kmem_cache_alloc_bulk.

see:
 https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[David Rientjes]

On Thu, 12 Nov 2015, Jesper Dangaard Brouer wrote:

> Hi MM-people,
> 
> How do you/we test the error paths when the system runs out of memory?
> 
> What kind of tools do you use?
> or Any tricks to provoke this?
> 

Depends on the paths that you want to exercise when the system is out of 
memory :)  If it's just to trigger the oom killer, then no kernel module 
should be necessary if you're not limited by any cgroup and just disable 
swap and start off an anonymous memory hog that consumes all memory.

> For testing my recent change to the SLUB allocator, I've implemented a
> crude kernel module that tries to allocate all memory, so I can test the
> error code-path in kmem_cache_alloc_bulk.
> 

Trying to exercise certain paths under oom is difficult because the oom 
killer will usually quickly kill a process or you'll get hung up somewhere 
else that needs memory before the function you want to test.  This is why 
failslab had been used in the past, and does a good job at runtime 
testing.  My suggestion would just be to instrument the kernel to randomly 
fail as though the system is oom and ensure that it works.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[Tetsuo Handa]

On 2015/11/13 5:55, Jesper Dangaard Brouer wrote:
> Hi MM-people,
>
> How do you/we test the error paths when the system runs out of memory?
>
> What kind of tools do you use?
> or Any tricks to provoke this?

I use SystemTap for injecting memory allocation failure.

http://lkml.kernel.org/r/201503182136.EJC90660.QSFOVJFOLHFOtM@I-love.SAKURA.ne.jp

>
> For testing my recent change to the SLUB allocator, I've implemented a
> crude kernel module that tries to allocate all memory, so I can test the
> error code-path in kmem_cache_alloc_bulk.
>
> see:
>   https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c
>

I think you can test the error code-path in kmem_cache_alloc_bulk as well.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[Jesper Dangaard Brouer]

On Fri, 13 Nov 2015 14:54:37 -0800 (PST) David Rientjes <rientjes@google.com> wrote:

> [...]  This is why 
> failslab had been used in the past, and does a good job at runtime 
> testing.  

Thanks for mentioning CONFIG_FAILSLAB.  First I disregarded
"failslab" (I did notice it in the slub code) because it didn't
exercised the code path I wanted in kmem_cache_alloc_bulk().

But went to looking up the config setting I notice that we do have a
hole section for "Fault-injection".  Which is great, and what I was
looking for.

Menu config Location:
 -> Kernel hacking
  -> Fault-injection framework (FAULT_INJECTION [=y])

I think what I need can be covered by FAIL_PAGE_ALLOC, or should_fail_alloc_page().
I'll try and play a bit with it...

- - 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

[*] Fault-injection framework
[*]   Fault-injection capability for kmalloc
[*]   Fault-injection capabilitiy for alloc_pages()
[ ]   Fault-injection capability for disk IO
[ ]   Fault-injection capability for faking disk interrupts
[ ]   Fault-injection capability for futexes
[*]   Debugfs entries for fault-injection capabilities

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[Jesper Dangaard Brouer]

On Sat, 14 Nov 2015 17:23:15 +0900
Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> wrote:

> On 2015/11/13 5:55, Jesper Dangaard Brouer wrote:
> > Hi MM-people,
> >
> > How do you/we test the error paths when the system runs out of memory?
> >
> > What kind of tools do you use?
> > or Any tricks to provoke this?
> 
> I use SystemTap for injecting memory allocation failure.
> 
> http://lkml.kernel.org/r/201503182136.EJC90660.QSFOVJFOLHFOtM@I-love.SAKURA.ne.jp
> 
> >
> > For testing my recent change to the SLUB allocator, I've implemented a
> > crude kernel module that tries to allocate all memory, so I can test the
> > error code-path in kmem_cache_alloc_bulk.
> >
> > see:
> >   https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c
> >
> 
> I think you can test the error code-path in kmem_cache_alloc_bulk as
> well.

Yes, making __alloc_pages_nodemask() fail should propagate all the way
back into kmem_cache_alloc_bulk().

I do like your approach, but I think my use-case can be covered by
CONFIG_FAIL_PAGE_ALLOC (which like you, also hook into __alloc_pages_nodemask).
Although it seems I have more control with your approach, to filter in
which situations it should happen in.

Thanks for your input! :-)
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[Jesper Dangaard Brouer]

On Mon, 16 Nov 2015 15:24:40 +0100 Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> On Fri, 13 Nov 2015 14:54:37 -0800 (PST) David Rientjes <rientjes@google.com> wrote:
> 
> > [...]  This is why 
> > failslab had been used in the past, and does a good job at runtime 
> > testing.  
> 
> Thanks for mentioning CONFIG_FAILSLAB.  First I disregarded
> "failslab" (I did notice it in the slub code) because it didn't
> exercised the code path I wanted in kmem_cache_alloc_bulk().
> 
> But went to looking up the config setting I notice that we do have a
> hole section for "Fault-injection".  Which is great, and what I was
> looking for.
> 
> Menu config Location:
>  -> Kernel hacking
>   -> Fault-injection framework (FAULT_INJECTION [=y])
> 
> I think what I need can be covered by FAIL_PAGE_ALLOC, or should_fail_alloc_page().
> I'll try and play a bit with it...

I did manage to provoke/test the error path in kmem_cache_alloc_bulk(),
by using fault-injection framework "fail_page_alloc".

But was a little hard to trigger SLUB errors with this, because SLUB
retries after a failure, and second call to alloc_pages() is done with
lower order.

If order is lowered to zero, then should_fail_alloc_page() will skip it.
And just lowering /sys/kernel/debug/fail_page_alloc/min-order=0 is not
feasible as even fork starts to fail.  I managed to work-around this by
using "space" setting.

Created a script to ease this tricky invocation:
 https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail01_kmem_cache_alloc_bulk.sh

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[David Rientjes]

On Tue, 17 Nov 2015, Jesper Dangaard Brouer wrote:

> I did manage to provoke/test the error path in kmem_cache_alloc_bulk(),
> by using fault-injection framework "fail_page_alloc".
> 
> But was a little hard to trigger SLUB errors with this, because SLUB
> retries after a failure, and second call to alloc_pages() is done with
> lower order.
> 
> If order is lowered to zero, then should_fail_alloc_page() will skip it.
> And just lowering /sys/kernel/debug/fail_page_alloc/min-order=0 is not
> feasible as even fork starts to fail.  I managed to work-around this by
> using "space" setting.
> 
> Created a script to ease this tricky invocation:
>  https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail01_kmem_cache_alloc_bulk.sh
> 

Any chance you could proffer some of your scripts in the form of patches 
to the tools/testing directory?  Anything that can reliably trigger rarely 
executed code is always useful.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[Jesper Dangaard Brouer]

On Thu, 19 Nov 2015 12:40:50 -0800 (PST)
David Rientjes <rientjes@google.com> wrote:

> On Tue, 17 Nov 2015, Jesper Dangaard Brouer wrote:
> 
> > I did manage to provoke/test the error path in kmem_cache_alloc_bulk(),
> > by using fault-injection framework "fail_page_alloc".
> > 
> > But was a little hard to trigger SLUB errors with this, because SLUB
> > retries after a failure, and second call to alloc_pages() is done with
> > lower order.
> > 
> > If order is lowered to zero, then should_fail_alloc_page() will skip it.
> > And just lowering /sys/kernel/debug/fail_page_alloc/min-order=0 is not
> > feasible as even fork starts to fail.  I managed to work-around this by
> > using "space" setting.
> > 
> > Created a script to ease this tricky invocation:
> >  https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail01_kmem_cache_alloc_bulk.sh
> > 
> 
> Any chance you could proffer some of your scripts in the form of patches 
> to the tools/testing directory?  Anything that can reliably trigger rarely 
> executed code is always useful.

Perhaps that is a good idea.

I think should move the directory location in my git-repo
prototype-kernel[1] to reflect this directory layout, like I do with
real kernel stuff.  And when we are happy with the quality of the
scripts we can "move" it to the kernel.  (Like I did with my pktgen
tests[4], now located in samples/pktgen/).

A question; where should/could we place the kernel module
slab_bulk_test04_exhaust_mem[1] that my fail01 script depends on?


BTW, I've also added a script for testing NULL handling in normal
kmem_cache_alloc() call see[3].

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

[1] https://github.com/netoptimizer/prototype-kernel/

[2] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c

[3] https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail02_kmem_cache_alloc.sh

[4] https://github.com/netoptimizer/network-testing/tree/master/pktgen

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[David Rientjes]

On Fri, 20 Nov 2015, Jesper Dangaard Brouer wrote:

> > Any chance you could proffer some of your scripts in the form of patches 
> > to the tools/testing directory?  Anything that can reliably trigger rarely 
> > executed code is always useful.
> 
> Perhaps that is a good idea.
> 
> I think should move the directory location in my git-repo
> prototype-kernel[1] to reflect this directory layout, like I do with
> real kernel stuff.  And when we are happy with the quality of the
> scripts we can "move" it to the kernel.  (Like I did with my pktgen
> tests[4], now located in samples/pktgen/).
> 
> A question; where should/could we place the kernel module
> slab_bulk_test04_exhaust_mem[1] that my fail01 script depends on?
> 

I've had the same question because I'd like to add slab and page allocator 
benchmark modules originally developed by Christoph Lameter to the tree.  
Let's add Andrew.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Memory exhaustion testing?

[Andrew Morton]

On Fri, 20 Nov 2015 15:23:09 -0800 (PST) David Rientjes <rientjes@google.com> wrote:

> On Fri, 20 Nov 2015, Jesper Dangaard Brouer wrote:
> 
> > > Any chance you could proffer some of your scripts in the form of patches 
> > > to the tools/testing directory?  Anything that can reliably trigger rarely 
> > > executed code is always useful.
> > 
> > Perhaps that is a good idea.
> > 
> > I think should move the directory location in my git-repo
> > prototype-kernel[1] to reflect this directory layout, like I do with
> > real kernel stuff.  And when we are happy with the quality of the
> > scripts we can "move" it to the kernel.  (Like I did with my pktgen
> > tests[4], now located in samples/pktgen/).
> > 
> > A question; where should/could we place the kernel module
> > slab_bulk_test04_exhaust_mem[1] that my fail01 script depends on?
> > 
> 
> I've had the same question because I'd like to add slab and page allocator 
> benchmark modules originally developed by Christoph Lameter to the tree.  
> Let's add Andrew.

Well, fwiw the current approach is to build the testing module in lib/
(ls -l lib/*test*.c) and to modprobe it from selftests (grep -r
modprobe tools/testing/selftests).

Does that suit?  I guess we could build and install a module from mm/
if that's needed for some reason.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>