From: Dmitry Vyukov <dvyukov@google.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: syzbot <syzbot+bab151e82a4e973fa325@syzkaller.appspotmail.com>,
cgroups@vger.kernel.org, Johannes Weiner <hannes@cmpxchg.org>,
LKML <linux-kernel@vger.kernel.org>,
Linux-MM <linux-mm@kvack.org>,
syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
Vladimir Davydov <vdavydov.dev@gmail.com>
Subject: Re: WARNING in try_charge
Date: Mon, 6 Aug 2018 11:30:37 +0200 [thread overview]
Message-ID: <CACT4Y+Ystnwv4M6Uh+HBKbdADAnJ6otfR0GoA20crzqV+b2onQ@mail.gmail.com> (raw)
In-Reply-To: <20180806091552.GE19540@dhcp22.suse.cz>
On Mon, Aug 6, 2018 at 11:15 AM, Michal Hocko <mhocko@kernel.org> wrote:
> On Sat 04-08-18 06:33:02, syzbot wrote:
>> Hello,
>>
>> syzbot found the following crash on:
>>
>> HEAD commit: d1e0b8e0cb7a Add linux-next specific files for 20180725
>> git tree: linux-next
>> console output: https://syzkaller.appspot.com/x/log.txt?x=15a1c770400000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=eef3552c897e4d33
>> dashboard link: https://syzkaller.appspot.com/bug?extid=bab151e82a4e973fa325
>> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>>
>> Unfortunately, I don't have any reproducer for this crash yet.
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: syzbot+bab151e82a4e973fa325@syzkaller.appspotmail.com
>>
>> Killed process 23767 (syz-executor2) total-vm:70472kB, anon-rss:104kB,
>> file-rss:32768kB, shmem-rss:0kB
>> oom_reaper: reaped process 23767 (syz-executor2), now anon-rss:0kB,
>> file-rss:32000kB, shmem-rss:0kB
>
> More interesting stuff is higher in the kernel log
> : [ 366.435015] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ile0,task_memcg=/ile0,task=syz-executor3,pid=23766,uid=0
> : [ 366.449416] memory: usage 112kB, limit 0kB, failcnt 1605
>
> Are you sure you want to have hard limit set to 0?
syzkaller really does not mind to have it.
> : [ 366.454963] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
> : [ 366.461787] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
> : [ 366.467946] Memory cgroup stats for /ile0: cache:12KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
>
> There are only 3 pages charged to this memcg!
>
> : [ 366.487490] Tasks state (memory values in pages):
> : [ 366.492349] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
> : [ 366.501237] [ 23766] 0 23766 17620 8221 126976 0 0 syz-executor3
> : [ 366.510367] [ 23767] 0 23767 17618 8218 126976 0 0 syz-executor2
> : [ 366.519409] Memory cgroup out of memory: Kill process 23766 (syz-executor3) score 8252000 or sacrifice child
> : [ 366.529422] Killed process 23766 (syz-executor3) total-vm:70480kB, anon-rss:116kB, file-rss:32768kB, shmem-rss:0kB
> : [ 366.540456] oom_reaper: reaped process 23766 (syz-executor3), now anon-rss:0kB, file-rss:32000kB, shmem-rss:0kB
>
> The oom reaper cannot reclaim file backed memory from a large part. I
> assume this is are shared mappings which are living outside of memcg
> because of the counter.
>
> : [...]
> : [ 367.085870] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/ile0,task_memcg=/ile0,task=syz-executor2,pid=23767,uid=0
> : [ 367.100073] memory: usage 112kB, limit 0kB, failcnt 1615
> : [ 367.105549] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
> : [ 367.112428] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
> : [ 367.118593] Memory cgroup stats for /ile0: cache:12KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
> : [ 367.138136] Tasks state (memory values in pages):
> : [ 367.142986] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
> : [ 367.151889] [ 23766] 0 23766 17620 8002 126976 0 0 syz-executor3
> : [ 367.160946] [ 23767] 0 23767 17618 8218 126976 0 0 syz-executor2
> : [ 367.169994] Memory cgroup out of memory: Kill process 23767 (syz-executor2) score 8249000 or sacrifice child
> : [ 367.180119] Killed process 23767 (syz-executor2) total-vm:70472kB, anon-rss:104kB, file-rss:32768kB, shmem-rss:0kB
> : [ 367.192101] oom_reaper: reaped process 23767 (syz-executor2), now anon-rss:0kB, file-rss:32000kB, shmem-rss:0kB
> : [ 367.202986] ------------[ cut here ]------------
> : [ 367.207845] Memory cgroup charge failed because of no reclaimable memory! This looks like a misconfiguration or a kernel bug.
> : [ 367.207965] WARNING: CPU: 1 PID: 23767 at mm/memcontrol.c:1710 try_charge+0x734/0x1680
> : [ 367.227540] Kernel panic - not syncing: panic_on_warn set ...
>
> This is unexpected though. We have killed a task (23767) which is trying
> to charge the memory which means it should
> trigger the charge retry and that one should force the charge
>
> /*
> * Unlike in global OOM situations, memcg is not in a physical
> * memory shortage. Allow dying and OOM-killed tasks to
> * bypass the last charges so that they can exit quickly and
> * free their memory.
> */
> if (unlikely(tsk_is_oom_victim(current) ||
> fatal_signal_pending(current) ||
> current->flags & PF_EXITING))
> goto force;
>
> There doesn't seem to be any other sign of OOM killer invocation which
> could then indeed lead to the warning as there is no other task to kill
> (both syz-executor[23] have been killed and oom_reaped already). So I
> would be curious what happened between 367.180119 which was the last
> successful oom invocation and 367.207845. An additional printk in
> mem_cgroup_out_of_memory might tell us more.
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 4603ad75c9a9..852cd3dbdcd9 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -1388,6 +1388,8 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
> bool ret;
>
> mutex_lock(&oom_lock);
> + pr_info("task=%s pid=%d invoked memcg oom killer. oom_victim=%d\n",
> + current->comm, current->pid, tsk_is_oom_victim(current));
> ret = out_of_memory(&oc);
> mutex_unlock(&oom_lock);
> return ret;
>
> Anyway your memcg setup is indeed misconfigured. Memcg with 0 hard limit
> and basically no memory charged by existing tasks is not going to fly
> and the warning is exactly to call that out.
Please-please-please do not mix kernel bugs and notices to user into
the same bucket:
https://lore.kernel.org/patchwork/patch/949071/
next prev parent reply other threads:[~2018-08-06 9:31 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-08-04 13:33 WARNING in try_charge syzbot
2018-08-04 13:45 ` Tetsuo Handa
2018-08-05 11:33 ` Tetsuo Handa
2018-08-05 8:14 ` syzbot
2018-08-06 9:15 ` Michal Hocko
2018-08-06 9:30 ` Dmitry Vyukov [this message]
2018-08-06 9:48 ` Michal Hocko
2018-08-06 10:34 ` Dmitry Vyukov
2018-08-06 11:02 ` Michal Hocko
2018-08-06 11:57 ` Dmitry Vyukov
2018-08-06 14:21 ` Michal Hocko
2018-08-06 14:58 ` Dmitry Vyukov
2018-08-06 17:30 ` Michal Hocko
2018-08-06 17:53 ` Dmitry Vyukov
2018-08-06 15:07 ` Dmitry Vyukov
2018-08-06 15:31 ` Johannes Weiner
2018-08-06 10:39 ` Dmitry Vyukov
2018-08-06 10:47 ` Tetsuo Handa
2018-08-06 11:09 ` Michal Hocko
2018-08-06 11:27 ` syzbot
2018-08-06 11:32 ` Michal Hocko
2018-08-06 11:58 ` Dmitry Vyukov
2018-08-06 14:41 ` Tetsuo Handa
2018-08-06 14:58 ` Michal Hocko
2018-08-06 15:12 ` Tetsuo Handa
2018-08-06 14:54 ` David Howells
2018-08-06 15:04 ` Tetsuo Handa
2018-08-06 11:00 ` syzbot
2018-08-06 15:32 ` Tetsuo Handa
2018-08-06 15:42 ` syzbot
2018-08-06 16:02 ` Tetsuo Handa
2018-08-06 17:44 ` Michal Hocko
2018-08-06 17:49 ` Dmitry Vyukov
2018-08-06 17:56 ` Michal Hocko
2018-08-06 18:13 ` Michal Hocko
2018-08-06 18:23 ` syzbot
2018-08-06 18:55 ` Michal Hocko
2018-08-06 19:12 ` syzbot
2018-08-06 19:45 ` Michal Hocko
2018-08-06 19:46 ` Michal Hocko
2018-08-07 11:18 ` Dmitry Vyukov
2018-08-07 11:25 ` Michal Hocko
2018-08-06 18:39 ` Michal Hocko
2018-08-06 20:26 ` Tetsuo Handa
2018-08-06 20:34 ` Michal Hocko
2018-08-06 20:46 ` Tetsuo Handa
2018-08-06 20:55 ` Michal Hocko
2018-08-06 21:50 ` Tetsuo Handa
2018-08-07 10:19 ` Tetsuo Handa
2018-08-09 13:57 ` Tetsuo Handa
2018-08-09 15:07 ` Michal Hocko
2018-08-09 21:05 ` Tetsuo Handa
2018-08-09 15:34 ` Johannes Weiner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CACT4Y+Ystnwv4M6Uh+HBKbdADAnJ6otfR0GoA20crzqV+b2onQ@mail.gmail.com \
--to=dvyukov@google.com \
--cc=cgroups@vger.kernel.org \
--cc=hannes@cmpxchg.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=syzbot+bab151e82a4e973fa325@syzkaller.appspotmail.com \
--cc=syzkaller-bugs@googlegroups.com \
--cc=vdavydov.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).