From: David Rientjes <rientjes@google.com>
To: Mikulas Patocka <mpatocka@redhat.com>
Cc: Michal Hocko <mhocko@kernel.org>,
Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
Ondrej Kozina <okozina@redhat.com>,
Jerome Marchand <jmarchan@redhat.com>,
Stanislav Kozina <skozina@redhat.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
dm-devel@redhat.com
Subject: Re: System freezes after OOM
Date: Thu, 14 Jul 2016 13:22:40 -0700 (PDT) [thread overview]
Message-ID: <alpine.DEB.2.10.1607141316240.68666@chino.kir.corp.google.com> (raw)
In-Reply-To: <alpine.LRH.2.02.1607140818250.15554@file01.intranet.prod.int.rdu2.redhat.com>
On Thu, 14 Jul 2016, Mikulas Patocka wrote:
> > schedule
> > schedule_timeout
> > io_schedule_timeout
> > mempool_alloc
> > __split_and_process_bio
> > dm_request
> > generic_make_request
> > submit_bio
> > mpage_readpages
> > ext4_readpages
> > __do_page_cache_readahead
> > ra_submit
> > filemap_fault
> > handle_mm_fault
> > __do_page_fault
> > do_page_fault
> > page_fault
>
> Device mapper should be able to proceed if there is no available memory.
> If it doesn't proceed, there is a bug in it.
>
The above stack trace has nothing to do with the device mapper with
pre-f9054c70d28b behavior. It simply is calling into mempool_alloc() and
no elements are being returned to the mempool that allow it to return.
Recall that in the above situation, the whole system is oom; nothing can
allocate memory. The oom killer has selected the above process to be oom
killed, so all other processes on the system trying to allocate memory
will stall in the page allocator waiting for this process to exit.
The natural response to this situation is to allow access to memory
reserves, if possible, so that mempool_alloc() may return. There is no
guarantee that _anything_ can return memory to the mempool, especially in
a system oom condition where nothing can make forward progress. Insisting
that should be guaranteed is not helpful.
> I'd like to ask - what device mapper targets did you use in this case? Are
> there some other deadlocked processes? (show sysrq-t, sysrq-w when this
> happened)
>
Every process on the system is deadlocked because they cannot get memory
through the page allocator until the above process exits. That is how the
oom killer works: select a process, kill it, give it access to memory
reserves so it may exit and free its memory, and wait.
> Did the machine lock up completely with that stacktrace, or was it just
> slowed down?
>
Hundreds of machines locked up and rebooted after a two hour watchdog
timeout.
> > That's obviously not the case if we have hundreds of machines timing out
> > after two hours waiting for that fault to succeed. The mempool interface
> > cannot require that users return elements to the pool synchronous with all
> > allocators so that we can happily loop forever, the only requirement on
>
> Mempool users must return objects to the mempool.
>
Not possible when the system is livelocked.
next prev parent reply other threads:[~2016-07-14 20:22 UTC|newest]
Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <57837CEE.1010609@redhat.com>
[not found] ` <f80dc690-7e71-26b2-59a2-5a1557d26713@redhat.com>
[not found] ` <9be09452-de7f-d8be-fd5d-4a80d1cd1ba3@redhat.com>
2016-07-11 15:43 ` System freezes after OOM Mikulas Patocka
2016-07-12 6:49 ` Michal Hocko
2016-07-12 23:44 ` Mikulas Patocka
2016-07-13 8:35 ` Jerome Marchand
2016-07-13 11:14 ` Michal Hocko
2016-07-13 14:21 ` Mikulas Patocka
2016-07-13 11:10 ` Michal Hocko
2016-07-13 12:50 ` Michal Hocko
2016-07-13 13:44 ` Milan Broz
2016-07-13 15:21 ` Mikulas Patocka
2016-07-14 9:09 ` Michal Hocko
2016-07-14 9:46 ` Milan Broz
2016-07-13 15:02 ` Mikulas Patocka
2016-07-14 10:51 ` [dm-devel] " Ondrej Kozina
2016-07-14 12:51 ` Michal Hocko
2016-07-14 14:00 ` Mikulas Patocka
2016-07-14 14:59 ` Michal Hocko
2016-07-14 15:25 ` Ondrej Kozina
2016-07-14 17:35 ` Mikulas Patocka
2016-07-15 8:35 ` Michal Hocko
2016-07-15 12:11 ` Mikulas Patocka
2016-07-15 12:22 ` Michal Hocko
2016-07-15 17:02 ` Mikulas Patocka
2016-07-18 7:22 ` Michal Hocko
2016-07-14 14:08 ` Ondrej Kozina
2016-07-14 15:31 ` Michal Hocko
2016-07-14 17:07 ` Ondrej Kozina
2016-07-14 17:36 ` Michal Hocko
2016-07-14 17:39 ` Michal Hocko
2016-07-15 11:42 ` Tetsuo Handa
2016-07-13 13:19 ` Tetsuo Handa
2016-07-13 13:39 ` Michal Hocko
2016-07-13 14:18 ` Mikulas Patocka
2016-07-13 14:56 ` Michal Hocko
2016-07-13 15:11 ` Mikulas Patocka
2016-07-13 23:53 ` David Rientjes
2016-07-14 11:01 ` Tetsuo Handa
2016-07-14 12:29 ` Mikulas Patocka
2016-07-14 20:26 ` David Rientjes
2016-07-14 21:40 ` Tetsuo Handa
2016-07-14 22:04 ` David Rientjes
2016-07-15 11:25 ` Mikulas Patocka
2016-07-15 21:21 ` David Rientjes
2016-07-14 12:27 ` Mikulas Patocka
2016-07-14 20:22 ` David Rientjes [this message]
2016-07-15 11:21 ` Mikulas Patocka
2016-07-15 21:25 ` David Rientjes
2016-07-15 21:39 ` Mikulas Patocka
2016-07-15 21:58 ` David Rientjes
2016-07-15 23:53 ` Mikulas Patocka
2016-07-18 15:14 ` Johannes Weiner
2016-07-14 15:29 ` Michal Hocko
2016-07-14 20:38 ` David Rientjes
2016-07-15 7:22 ` Michal Hocko
2016-07-15 8:23 ` Michal Hocko
2016-07-15 12:00 ` Mikulas Patocka
2016-07-15 21:47 ` David Rientjes
2016-07-18 7:39 ` Michal Hocko
2016-07-18 21:03 ` David Rientjes
2016-07-14 0:01 ` David Rientjes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.10.1607141316240.68666@chino.kir.corp.google.com \
--to=rientjes@google.com \
--cc=dm-devel@redhat.com \
--cc=jmarchan@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=mpatocka@redhat.com \
--cc=okozina@redhat.com \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=skozina@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).