LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Joel Fernandes <joelaf@google.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>,
	Zhaoyang Huang <huangzhaoyang@gmail.com>,
	Ingo Molnar <mingo@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	kernel-patch-test@lists.linaro.org,
	Andrew Morton <akpm@linux-foundation.org>,
	"open list:MEMORY MANAGEMENT" <linux-mm@kvack.org>,
	Vlastimil Babka <vbabka@suse.cz>
Subject: Re: [PATCH v1] kernel/trace:check the val against the available mem
Date: Wed, 4 Apr 2018 05:21:49 -0700
Message-ID: <CAJWu+oo8LE6ZzPaz0KgT4C5a3vYJS2SkohHMObj3pWN9BZMN=A@mail.gmail.com> (raw)
In-Reply-To: <20180404062039.GC6312@dhcp22.suse.cz>

On Tue, Apr 3, 2018 at 11:20 PM, Michal Hocko <mhocko@kernel.org> wrote:
> On Tue 03-04-18 18:56:27, Steven Rostedt wrote:
> [...]
>> From your earlier email:
>>
>> > Except that it doesn't work. si_mem_available is not really suitable for
>> > any allocation estimations. Its only purpose is to provide a very rough
>> > estimation for userspace. Any other use is basically abuse. The
>> > situation can change really quickly. Really it is really hard to be
>> > clever here with the volatility the memory allocations can cause.
>>
>> Now can you please explain to me why si_mem_available is not suitable
>> for my purpose.
>
> Several problems. It is overly optimistic especially when we are close
> to OOM. The available pagecache or slab reclaimable objects might be pinned
> long enough that your allocation based on that estimation will just make
> the situation worse and result in OOM. More importantly though, your
> allocations are GFP_KERNEL, right, that means that such an allocation
> will not reach to ZONE_MOVABLE or ZONE_HIGMEM (32b systems) while the
> pagecache will. So you will get an overestimate of how much you can
> allocate.

Yes, but right now it is assumed that there is all the memory in the
world to allocate. Clearly an overestimate is better than that. Right
now ftrace will just allocate memory until it actually fails to
allocate, at which point it may have caused other processes to be
likely to OOM. As Steve pointed out, it doesn't need to be accurate
but does solve the problem here.. (or some other similar method seems
to be needed to solve it).

>
> Really si_mem_available is for proc/meminfo and a rough estimate of the
> free memory because users tend to be confused by seeing MemFree too low
> and complaining that the system has eaten all their memory. I have some

By why is a rough estimate not good in this case, that's what I don't get.

> skepticism about how useful it is in practice apart from showing it in
> top or alike tools. The memory is simply not usable immediately or
> without an overall and visible effect on the whole system.

Sure there can be false positives but it does reduce the problem for
the most part I feel. May be we can use this as an interim solution
(better than leaving the issue hanging)? Or you could propose a
solution of how to get an estimate and prevent other tasks from
experiencing memory pressure for no reason.

Another thing we could do is check how much total memory there is on
the system and cap by that (which will prevent impossibly large
allocations) but that still doesn't address the problem completely.

thanks,

- Joel

  reply index

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-03-29 10:41 Zhaoyang Huang
2018-03-29 16:05 ` Steven Rostedt
2018-03-30  3:32   ` Zhaoyang Huang
2018-03-30 14:07     ` Steven Rostedt
2018-03-30  6:53 ` [Kernel-patch-test] " kbuild test robot
2018-03-30  6:54 ` kbuild test robot
2018-03-30 14:20 ` Steven Rostedt
2018-03-30 16:37   ` Joel Fernandes
2018-03-30 19:10     ` Steven Rostedt
2018-03-30 20:37       ` Joel Fernandes
2018-03-30 20:53   ` Matthew Wilcox
2018-03-30 21:30     ` Steven Rostedt
2018-03-30 21:42       ` Steven Rostedt
2018-03-30 23:38         ` Joel Fernandes
2018-03-31  1:41           ` Steven Rostedt
2018-03-31  2:18             ` Matthew Wilcox
2018-03-31  3:07               ` Steven Rostedt
2018-03-31  5:44                 ` Joel Fernandes
2018-04-02  0:52         ` Zhaoyang Huang
2018-04-03 11:06   ` Michal Hocko
2018-04-03 11:51     ` Steven Rostedt
2018-04-03 12:16       ` Michal Hocko
2018-04-03 12:23         ` Steven Rostedt
2018-04-03 12:35           ` Michal Hocko
2018-04-03 13:32             ` Steven Rostedt
2018-04-03 13:56               ` Michal Hocko
2018-04-03 14:17                 ` Steven Rostedt
2018-04-03 16:11                   ` Michal Hocko
2018-04-03 16:59                     ` Steven Rostedt
2018-04-03 22:56                     ` Steven Rostedt
2018-04-04  6:20                       ` Michal Hocko
2018-04-04 12:21                         ` Joel Fernandes [this message]
2018-04-04 12:59                         ` Steven Rostedt
2018-04-04 14:10                           ` Michal Hocko
2018-04-04 14:25                             ` Steven Rostedt
2018-04-04 14:42                               ` Michal Hocko
2018-04-04 15:04                                 ` Steven Rostedt
2018-04-04 15:27                                   ` Michal Hocko
2018-04-04 15:38                                     ` Steven Rostedt
2018-04-04  2:58                 ` Zhaoyang Huang
2018-04-04  6:23                   ` Michal Hocko
2018-04-04  9:29                     ` Zhaoyang Huang
2018-04-04 14:11                     ` Steven Rostedt
2018-04-04 14:23                       ` Michal Hocko
2018-04-04 14:31                         ` Steven Rostedt
2018-04-04 14:47                           ` Michal Hocko
2018-04-04 15:47                         ` Steven Rostedt
2018-04-05  2:58                           ` Matthew Wilcox
2018-04-05  4:12                             ` Joel Fernandes
2018-04-05 14:22                               ` Matthew Wilcox
2018-04-05 14:27                                 ` Michal Hocko
2018-04-05 14:34                                   ` Steven Rostedt
2018-04-05 15:13                                   ` Matthew Wilcox
2018-04-05 15:32                                     ` Michal Hocko
2018-04-05 16:15                                       ` Matthew Wilcox
2018-04-05 18:54                                         ` Michal Hocko
2018-04-05 20:15                                           ` __GFP_LOW Matthew Wilcox
2018-04-06  6:09                                             ` __GFP_LOW Michal Hocko
2018-04-08  4:27                                               ` __GFP_LOW Matthew Wilcox
2018-04-09  7:34                                                 ` __GFP_LOW Michal Hocko
2018-04-09 15:51                                                   ` __GFP_LOW Matthew Wilcox
2018-04-09 18:14                                                     ` __GFP_LOW Michal Hocko
     [not found]                                                       ` <CA+JonM0HG9kWb6-0iyDQ8UMxTeR-f=+ZL89t5DvvDULDC8Sfyw@mail.gmail.com>
2018-04-10 12:19                                                         ` __GFP_LOW Matthew Wilcox
2018-04-05 14:30                                 ` [PATCH v1] kernel/trace:check the val against the available mem Steven Rostedt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAJWu+oo8LE6ZzPaz0KgT4C5a3vYJS2SkohHMObj3pWN9BZMN=A@mail.gmail.com' \
    --to=joelaf@google.com \
    --cc=akpm@linux-foundation.org \
    --cc=huangzhaoyang@gmail.com \
    --cc=kernel-patch-test@lists.linaro.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    --cc=mingo@kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=vbabka@suse.cz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git
	git clone --mirror https://lore.kernel.org/lkml/10 lkml/git/10.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git