From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
To: mhocko@kernel.org, akpm@linux-foundation.org
Cc: mgorman@suse.de, rientjes@google.com,
torvalds@linux-foundation.org, oleg@redhat.com, hughd@google.com,
andrea@kernel.org, riel@redhat.com, linux-mm@kvack.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH 1/2] mm, oom: introduce oom reaper
Date: Fri, 18 Dec 2015 21:10:26 +0900 [thread overview]
Message-ID: <201512182110.FBH73485.LFOFtOOVSHFQMJ@I-love.SAKURA.ne.jp> (raw)
In-Reply-To: <20151217130223.GE18625@dhcp22.suse.cz>
Michal Hocko wrote:
> On Wed 16-12-15 16:50:35, Andrew Morton wrote:
> > On Tue, 15 Dec 2015 19:36:15 +0100 Michal Hocko <mhocko@kernel.org> wrote:
> [...]
> > > +static void oom_reap_vmas(struct mm_struct *mm)
> > > +{
> > > + int attempts = 0;
> > > +
> > > + while (attempts++ < 10 && !__oom_reap_vmas(mm))
> > > + schedule_timeout(HZ/10);
> >
> > schedule_timeout() in state TASK_RUNNING doesn't do anything. Use
> > msleep() or msleep_interruptible(). I can't decide which is more
> > appropriate - it only affects the load average display.
>
> Ups. You are right. I will go with msleep_interruptible(100).
>
I didn't know that. My testing was almost without oom_reap_vmas().
> > I guess it means that the __oom_reap_vmas() success rate is nice anud
> > high ;)
>
> I had a debugging trace_printks around this and there were no reties
> during my testing so I was probably lucky to not trigger the mmap_sem
> contention.
Yes, you are lucky that you did not hit the mmap_sem contention.
I retested with
static void oom_reap_vmas(struct mm_struct *mm)
{
int attempts = 0;
while (attempts++ < 10 && !__oom_reap_vmas(mm))
- schedule_timeout(HZ/10);
+ msleep_interruptible(100);
+ printk(KERN_WARNING "oom_reaper: attempts=%u\n", attempts);
/* Drop a reference taken by wake_oom_reaper */
mmdrop(mm);
}
and I can hit that attempts becomes 11 (i.e. oom_reap_vmas() gives up
waiting) if I ran a memory stressing program with many contending
mmap_sem readers and writers shown below.
----------
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sched.h>
#include <sys/mman.h>
static cpu_set_t set = { { 1 } }; /* Allow only CPU 0. */
static char filename[32] = { };
/* down_read(&mm->mmap_sem) requester. */
static int reader(void *unused)
{
const int fd = open(filename, O_RDONLY);
char buffer[128];
sched_setaffinity(0, sizeof(set), &set);
sleep(2);
while (pread(fd, buffer, sizeof(buffer), 0) > 0);
while (1)
pause();
return 0;
}
/* down_write(&mm->mmap_sem) requester. */
static int writer(void *unused)
{
const int fd = open("/proc/self/exe", O_RDONLY);
sched_setaffinity(0, sizeof(set), &set);
sleep(2);
while (1) {
void *ptr = mmap(NULL, 4096, PROT_READ, MAP_PRIVATE, fd, 0);
munmap(ptr, 4096);
}
return 0;
}
static void my_clone(int (*func) (void *))
{
char *stack = malloc(4096);
if (stack)
clone(func, stack + 4096,
CLONE_THREAD | CLONE_SIGHAND | CLONE_VM, NULL);
}
/* Memory consumer for invoking the OOM killer. */
static void memory_eater(void) {
char *buf = NULL;
unsigned long i;
unsigned long size = 0;
sleep(4);
for (size = 1048576; size < 512UL * (1 << 30); size <<= 1) {
char *cp = realloc(buf, size);
if (!cp) {
size >>= 1;
break;
}
buf = cp;
}
fprintf(stderr, "Start eating memory\n");
for (i = 0; i < size; i += 4096)
buf[i] = '\0'; /* Will cause OOM due to overcommit */
}
int main(int argc, char *argv[])
{
int i;
const pid_t pid = fork();
if (pid == 0) {
for (i = 0; i < 9; i++)
my_clone(writer);
writer(NULL);
_exit(0);
} else if (pid > 0) {
snprintf(filename, sizeof(filename), "/proc/%u/stat", pid);
for (i = 0; i < 1000; i++)
my_clone(reader);
}
memory_eater();
return *(char *) NULL; /* Not reached. */
}
----------
Complete log is at http://I-love.SAKURA.ne.jp/tmp/serial-20151218.txt.xz .
----------
[ 90.790847] Killed process 9560 (oom_reaper-test) total-vm:4312kB, anon-rss:124kB, file-rss:0kB, shmem-rss:0kB
[ 91.803154] oom_reaper: attempts=11
[ 100.701494] MemAlloc-Info: 509 stalling task, 0 dying task, 1 victim task.
[ 102.439082] Killed process 9559 (oom_reaper-test) total-vm:2170960kB, anon-rss:1564600kB, file-rss:0kB, shmem-rss:0kB
[ 102.441937] Killed process 9561 (oom_reaper-test) total-vm:2170960kB, anon-rss:1564776kB, file-rss:0kB, shmem-rss:0kB
[ 102.731326] oom_reaper: attempts=1
[ 125.420727] Killed process 10573 (oom_reaper-test) total-vm:4340kB, anon-rss:80kB, file-rss:0kB, shmem-rss:0kB
[ 126.440392] oom_reaper: attempts=11
[ 135.354193] MemAlloc-Info: 450 stalling task, 0 dying task, 0 victim task.
[ 240.023256] MemAlloc-Info: 1016 stalling task, 0 dying task, 0 victim task.
[ 302.246975] Killed process 10572 (oom_reaper-test) total-vm:2170960kB, anon-rss:1562128kB, file-rss:0kB, shmem-rss:0kB
[ 302.263515] oom_reaper: attempts=1
[ 382.961343] Killed process 11667 (oom_reaper-test) total-vm:4312kB, anon-rss:84kB, file-rss:0kB, shmem-rss:0kB
[ 383.980541] oom_reaper: attempts=11
[ 392.592658] MemAlloc-Info: 758 stalling task, 10 dying task, 1 victim task.
[ 399.497478] Killed process 11666 (oom_reaper-test) total-vm:2170960kB, anon-rss:1556072kB, file-rss:0kB, shmem-rss:0kB
[ 399.499101] Killed process 11668 (oom_reaper-test) total-vm:2170960kB, anon-rss:1556260kB, file-rss:0kB, shmem-rss:0kB
[ 399.778283] oom_reaper: attempts=1
[ 438.304082] Killed process 12680 (oom_reaper-test) total-vm:4324kB, anon-rss:120kB, file-rss:0kB, shmem-rss:0kB
[ 439.318951] oom_reaper: attempts=11
[ 445.581171] MemAlloc-Info: 796 stalling task, 0 dying task, 0 victim task.
[ 618.955215] MemAlloc-Info: 979 stalling task, 0 dying task, 0 victim task.
----------
Yes, this is an insane program. But what is important will be we prepare for
cases when oom_reap_vmas() gave up waiting. Silent hang up is annoying.
Like Andrew said
( http://lkml.kernel.org/r/20151216153513.e432dc70e035e5d07984710c@linux-foundation.org ),
I want to add a watchdog for printk()ing.
( http://lkml.kernel.org/r/201512170011.IAC73451.FLtFMSJHOQFVOO@I-love.SAKURA.ne.jp ).
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2015-12-18 12:10 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-15 18:36 [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko
2015-12-17 0:50 ` Andrew Morton
2015-12-17 13:02 ` Michal Hocko
2015-12-17 19:55 ` Linus Torvalds
2015-12-17 20:00 ` Andrew Morton
2015-12-18 11:54 ` Michal Hocko
2015-12-18 21:14 ` Andrew Morton
2015-12-21 8:38 ` Michal Hocko
2015-12-17 21:13 ` Andrew Morton
2015-12-18 12:11 ` Michal Hocko
2015-12-18 12:10 ` Tetsuo Handa [this message]
2015-12-20 7:14 ` Tetsuo Handa
2015-12-18 0:15 ` Andrew Morton
2015-12-18 11:48 ` Michal Hocko
2015-12-21 20:38 ` Paul Gortmaker
2016-01-06 9:10 ` Michal Hocko
2016-01-06 14:26 ` Paul Gortmaker
2016-01-06 15:00 ` Michal Hocko
2015-12-23 23:00 ` Ross Zwisler
2015-12-24 9:47 ` Michal Hocko
2015-12-24 11:06 ` Tetsuo Handa
2015-12-24 20:39 ` Ross Zwisler
2015-12-25 11:41 ` Michal Hocko
2015-12-24 20:44 ` Ross Zwisler
2015-12-25 11:35 ` Michal Hocko
2015-12-25 11:44 ` Michal Hocko
2016-01-06 15:42 [PATCH 0/2 -mm] oom reaper v4 Michal Hocko
2016-01-06 15:42 ` [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko
2016-01-07 11:23 ` Tetsuo Handa
2016-01-07 12:30 ` Michal Hocko
2016-01-11 22:54 ` Andrew Morton
2016-01-12 8:16 ` Michal Hocko
2016-01-28 1:28 ` David Rientjes
2016-01-28 21:42 ` Michal Hocko
2016-02-02 3:02 ` David Rientjes
2016-02-02 8:57 ` Michal Hocko
2016-02-02 11:48 ` Tetsuo Handa
2016-02-02 22:55 ` David Rientjes
2016-02-02 22:51 ` David Rientjes
2016-02-03 10:31 ` Tetsuo Handa
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=201512182110.FBH73485.LFOFtOOVSHFQMJ@I-love.SAKURA.ne.jp \
--to=penguin-kernel@i-love.sakura.ne.jp \
--cc=akpm@linux-foundation.org \
--cc=andrea@kernel.org \
--cc=hughd@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=mhocko@kernel.org \
--cc=oleg@redhat.com \
--cc=riel@redhat.com \
--cc=rientjes@google.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).