From: Ross Zwisler <zwisler@gmail.com> To: Michal Hocko <mhocko@kernel.org> Cc: Andrew Morton <akpm@linux-foundation.org>, Mel Gorman <mgorman@suse.de>, Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>, David Rientjes <rientjes@google.com>, Linus Torvalds <torvalds@linux-foundation.org>, Oleg Nesterov <oleg@redhat.com>, Hugh Dickins <hughd@google.com>, Andrea Argangeli <andrea@kernel.org>, Rik van Riel <riel@redhat.com>, linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>, Ross Zwisler <ross.zwisler@linux.intel.com> Subject: Re: [PATCH 1/2] mm, oom: introduce oom reaper Date: Thu, 24 Dec 2015 13:44:03 -0700 [thread overview] Message-ID: <CAOxpaSXRxJGqL3Fxz5280KZy6xG0ZGwyrf-7i6LArSC0eJsv2A@mail.gmail.com> (raw) In-Reply-To: <20151224094758.GA22760@dhcp22.suse.cz> On Thu, Dec 24, 2015 at 2:47 AM, Michal Hocko <mhocko@kernel.org> wrote: > On Wed 23-12-15 16:00:09, Ross Zwisler wrote: > [...] >> While running xfstests on next-20151223 I hit a pair of kernel BUGs >> that bisected to this commit: >> >> 1eb3a80d8239 ("mm, oom: introduce oom reaper") > > Thank you for the report and the bisection. > >> Here is a BUG produced by generic/029 when run against XFS: >> >> [ 235.751723] ------------[ cut here ]------------ >> [ 235.752194] kernel BUG at mm/filemap.c:208! > > This is VM_BUG_ON_PAGE(page_mapped(page), page), right? Could you attach > the full kernel log? It all smells like a race when OOM reaper tears > down the mapping and there is a truncate still in progress. But hitting > the BUG_ON just because of that doesn't make much sense to me. OOM > reaper is essentially MADV_DONTNEED. I have to think about this some > more, though, but I am in a holiday mode until early next year so please > bear with me. The two stack traces were gathered with next-20151223, so the line numbers may have moved around a bit when compared to the actual "mm, oom: introduce oom reaper" commit. > [...] >> [ 235.765638] Call Trace: >> [ 235.765903] [<ffffffff811c8493>] delete_from_page_cache+0x63/0xd0 >> [ 235.766513] [<ffffffff811dc3e5>] truncate_inode_page+0xa5/0x120 >> [ 235.767088] [<ffffffff811dc648>] truncate_inode_pages_range+0x1a8/0x7f0 >> [ 235.767725] [<ffffffff81021459>] ? sched_clock+0x9/0x10 >> [ 235.768239] [<ffffffff810db37c>] ? local_clock+0x1c/0x20 >> [ 235.768779] [<ffffffff811feba4>] ? unmap_mapping_range+0x64/0x130 >> [ 235.769385] [<ffffffff811febb4>] ? unmap_mapping_range+0x74/0x130 >> [ 235.770010] [<ffffffff810f5c3f>] ? up_write+0x1f/0x40 >> [ 235.770501] [<ffffffff811febb4>] ? unmap_mapping_range+0x74/0x130 >> [ 235.771092] [<ffffffff811dcd58>] truncate_pagecache+0x48/0x70 >> [ 235.771646] [<ffffffff811dcdb2>] truncate_setsize+0x32/0x40 >> [ 235.772276] [<ffffffff8148e972>] xfs_setattr_size+0x232/0x470 >> [ 235.772839] [<ffffffff8148ec64>] xfs_vn_setattr+0xb4/0xc0 >> [ 235.773369] [<ffffffff8127af87>] notify_change+0x237/0x350 >> [ 235.773945] [<ffffffff81257c87>] do_truncate+0x77/0xc0 >> [ 235.774446] [<ffffffff8125800f>] do_sys_ftruncate.constprop.15+0xef/0x150 >> [ 235.775156] [<ffffffff812580ae>] SyS_ftruncate+0xe/0x10 >> [ 235.775650] [<ffffffff81a527b2>] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 235.776257] Code: 5f 5d c3 48 8b 43 20 48 8d 78 ff a8 01 48 0f 44 >> fb 8b 47 48 85 c0 0f 88 2b 01 00 00 48 c7 c6 a8 57 f0 81 48 89 df e8 >> fa 1a 03 00 <0f> 0b 4c 89 ce 44 89 fa 4c 89 e7 4c 89 45 b0 4c 89 4d b8 >> e8 32 >> [ 235.778695] RIP [<ffffffff811c81f6>] __delete_from_page_cache+0x206/0x440 >> [ 235.779350] RSP <ffff8800bab83b60> >> [ 235.779694] ---[ end trace fac9dd65c4cdd828 ]--- >> >> And a different BUG produced by generic/095, also with XFS: >> >> [ 609.398897] ------------[ cut here ]------------ >> [ 609.399843] kernel BUG at mm/truncate.c:629! > > Hmm, I do not see any BUG_ON at this line. But there is > BUG_ON(page_mapped(page)) at line 620. Ditto - check out next-20151223 for real line numbers. >> [ 609.400666] invalid opcode: 0000 [#1] SMP >> [ 609.401512] Modules linked in: nd_pmem nd_btt nd_e820 libnvdimm >> [ 609.402719] CPU: 4 PID: 26782 Comm: fio Tainted: G W > > There was a warning before this triggered. The full kernel log would be > helpful as well. Sure, I can gather full kernel logs, but it'll probably after the new year. > [...] >> [ 609.425325] Call Trace: >> [ 609.425797] [<ffffffff811dc307>] invalidate_inode_pages2+0x17/0x20 >> [ 609.426971] [<ffffffff81482167>] xfs_file_read_iter+0x297/0x300 >> [ 609.428097] [<ffffffff81259ac9>] __vfs_read+0xc9/0x100 >> [ 609.429073] [<ffffffff8125a319>] vfs_read+0x89/0x130 >> [ 609.430010] [<ffffffff8125b418>] SyS_read+0x58/0xd0 >> [ 609.430943] [<ffffffff81a527b2>] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 609.432139] Code: 85 d8 fe ff ff 01 00 00 00 f6 c4 40 0f 84 59 ff >> ff ff 49 8b 47 20 48 8d 78 ff a8 01 49 0f 44 ff 8b 47 48 85 c0 0f 88 >> bd 01 00 00 <0f> 0b 4d 3b 67 08 0f 85 70 ff ff ff 49 f7 07 00 18 00 00 >> 74 15 > [...] >> My test setup is a qemu guest machine with a pair of 4 GiB PMEM >> ramdisk test devices, one for the xfstest test disk and one for the >> scratch disk. > > Is this just a plain ramdisk device or it needs a special configuration? Just a plain PMEM ram disk with DAX turned off. Configuration instructions for PMEM can be found here: https://nvdimm.wiki.kernel.org/
WARNING: multiple messages have this Message-ID (diff)
From: Ross Zwisler <zwisler@gmail.com> To: Michal Hocko <mhocko@kernel.org> Cc: Andrew Morton <akpm@linux-foundation.org>, Mel Gorman <mgorman@suse.de>, Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>, David Rientjes <rientjes@google.com>, Linus Torvalds <torvalds@linux-foundation.org>, Oleg Nesterov <oleg@redhat.com>, Hugh Dickins <hughd@google.com>, Andrea Argangeli <andrea@kernel.org>, Rik van Riel <riel@redhat.com>, linux-mm@kvack.org, LKML <linux-kernel@vger.kernel.org>, Ross Zwisler <ross.zwisler@linux.intel.com> Subject: Re: [PATCH 1/2] mm, oom: introduce oom reaper Date: Thu, 24 Dec 2015 13:44:03 -0700 [thread overview] Message-ID: <CAOxpaSXRxJGqL3Fxz5280KZy6xG0ZGwyrf-7i6LArSC0eJsv2A@mail.gmail.com> (raw) In-Reply-To: <20151224094758.GA22760@dhcp22.suse.cz> On Thu, Dec 24, 2015 at 2:47 AM, Michal Hocko <mhocko@kernel.org> wrote: > On Wed 23-12-15 16:00:09, Ross Zwisler wrote: > [...] >> While running xfstests on next-20151223 I hit a pair of kernel BUGs >> that bisected to this commit: >> >> 1eb3a80d8239 ("mm, oom: introduce oom reaper") > > Thank you for the report and the bisection. > >> Here is a BUG produced by generic/029 when run against XFS: >> >> [ 235.751723] ------------[ cut here ]------------ >> [ 235.752194] kernel BUG at mm/filemap.c:208! > > This is VM_BUG_ON_PAGE(page_mapped(page), page), right? Could you attach > the full kernel log? It all smells like a race when OOM reaper tears > down the mapping and there is a truncate still in progress. But hitting > the BUG_ON just because of that doesn't make much sense to me. OOM > reaper is essentially MADV_DONTNEED. I have to think about this some > more, though, but I am in a holiday mode until early next year so please > bear with me. The two stack traces were gathered with next-20151223, so the line numbers may have moved around a bit when compared to the actual "mm, oom: introduce oom reaper" commit. > [...] >> [ 235.765638] Call Trace: >> [ 235.765903] [<ffffffff811c8493>] delete_from_page_cache+0x63/0xd0 >> [ 235.766513] [<ffffffff811dc3e5>] truncate_inode_page+0xa5/0x120 >> [ 235.767088] [<ffffffff811dc648>] truncate_inode_pages_range+0x1a8/0x7f0 >> [ 235.767725] [<ffffffff81021459>] ? sched_clock+0x9/0x10 >> [ 235.768239] [<ffffffff810db37c>] ? local_clock+0x1c/0x20 >> [ 235.768779] [<ffffffff811feba4>] ? unmap_mapping_range+0x64/0x130 >> [ 235.769385] [<ffffffff811febb4>] ? unmap_mapping_range+0x74/0x130 >> [ 235.770010] [<ffffffff810f5c3f>] ? up_write+0x1f/0x40 >> [ 235.770501] [<ffffffff811febb4>] ? unmap_mapping_range+0x74/0x130 >> [ 235.771092] [<ffffffff811dcd58>] truncate_pagecache+0x48/0x70 >> [ 235.771646] [<ffffffff811dcdb2>] truncate_setsize+0x32/0x40 >> [ 235.772276] [<ffffffff8148e972>] xfs_setattr_size+0x232/0x470 >> [ 235.772839] [<ffffffff8148ec64>] xfs_vn_setattr+0xb4/0xc0 >> [ 235.773369] [<ffffffff8127af87>] notify_change+0x237/0x350 >> [ 235.773945] [<ffffffff81257c87>] do_truncate+0x77/0xc0 >> [ 235.774446] [<ffffffff8125800f>] do_sys_ftruncate.constprop.15+0xef/0x150 >> [ 235.775156] [<ffffffff812580ae>] SyS_ftruncate+0xe/0x10 >> [ 235.775650] [<ffffffff81a527b2>] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 235.776257] Code: 5f 5d c3 48 8b 43 20 48 8d 78 ff a8 01 48 0f 44 >> fb 8b 47 48 85 c0 0f 88 2b 01 00 00 48 c7 c6 a8 57 f0 81 48 89 df e8 >> fa 1a 03 00 <0f> 0b 4c 89 ce 44 89 fa 4c 89 e7 4c 89 45 b0 4c 89 4d b8 >> e8 32 >> [ 235.778695] RIP [<ffffffff811c81f6>] __delete_from_page_cache+0x206/0x440 >> [ 235.779350] RSP <ffff8800bab83b60> >> [ 235.779694] ---[ end trace fac9dd65c4cdd828 ]--- >> >> And a different BUG produced by generic/095, also with XFS: >> >> [ 609.398897] ------------[ cut here ]------------ >> [ 609.399843] kernel BUG at mm/truncate.c:629! > > Hmm, I do not see any BUG_ON at this line. But there is > BUG_ON(page_mapped(page)) at line 620. Ditto - check out next-20151223 for real line numbers. >> [ 609.400666] invalid opcode: 0000 [#1] SMP >> [ 609.401512] Modules linked in: nd_pmem nd_btt nd_e820 libnvdimm >> [ 609.402719] CPU: 4 PID: 26782 Comm: fio Tainted: G W > > There was a warning before this triggered. The full kernel log would be > helpful as well. Sure, I can gather full kernel logs, but it'll probably after the new year. > [...] >> [ 609.425325] Call Trace: >> [ 609.425797] [<ffffffff811dc307>] invalidate_inode_pages2+0x17/0x20 >> [ 609.426971] [<ffffffff81482167>] xfs_file_read_iter+0x297/0x300 >> [ 609.428097] [<ffffffff81259ac9>] __vfs_read+0xc9/0x100 >> [ 609.429073] [<ffffffff8125a319>] vfs_read+0x89/0x130 >> [ 609.430010] [<ffffffff8125b418>] SyS_read+0x58/0xd0 >> [ 609.430943] [<ffffffff81a527b2>] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 609.432139] Code: 85 d8 fe ff ff 01 00 00 00 f6 c4 40 0f 84 59 ff >> ff ff 49 8b 47 20 48 8d 78 ff a8 01 49 0f 44 ff 8b 47 48 85 c0 0f 88 >> bd 01 00 00 <0f> 0b 4d 3b 67 08 0f 85 70 ff ff ff 49 f7 07 00 18 00 00 >> 74 15 > [...] >> My test setup is a qemu guest machine with a pair of 4 GiB PMEM >> ramdisk test devices, one for the xfstest test disk and one for the >> scratch disk. > > Is this just a plain ramdisk device or it needs a special configuration? Just a plain PMEM ram disk with DAX turned off. Configuration instructions for PMEM can be found here: https://nvdimm.wiki.kernel.org/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2015-12-24 20:44 UTC|newest] Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-12-15 18:36 [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko 2015-12-15 18:36 ` Michal Hocko 2015-12-17 0:50 ` Andrew Morton 2015-12-17 0:50 ` Andrew Morton 2015-12-17 13:02 ` Michal Hocko 2015-12-17 13:02 ` Michal Hocko 2015-12-17 19:55 ` Linus Torvalds 2015-12-17 19:55 ` Linus Torvalds 2015-12-17 20:00 ` Andrew Morton 2015-12-17 20:00 ` Andrew Morton 2015-12-18 11:54 ` Michal Hocko 2015-12-18 11:54 ` Michal Hocko 2015-12-18 21:14 ` Andrew Morton 2015-12-18 21:14 ` Andrew Morton 2015-12-21 8:38 ` Michal Hocko 2015-12-21 8:38 ` Michal Hocko 2015-12-17 21:13 ` Andrew Morton 2015-12-17 21:13 ` Andrew Morton 2015-12-18 12:11 ` Michal Hocko 2015-12-18 12:11 ` Michal Hocko 2015-12-18 12:10 ` Tetsuo Handa 2015-12-18 12:10 ` Tetsuo Handa 2015-12-20 7:14 ` Tetsuo Handa 2015-12-20 7:14 ` Tetsuo Handa 2015-12-18 0:15 ` Andrew Morton 2015-12-18 0:15 ` Andrew Morton 2015-12-18 11:48 ` Michal Hocko 2015-12-18 11:48 ` Michal Hocko 2015-12-21 20:38 ` Paul Gortmaker 2015-12-21 20:38 ` Paul Gortmaker 2016-01-06 9:10 ` Michal Hocko 2016-01-06 9:10 ` Michal Hocko 2016-01-06 14:26 ` Paul Gortmaker 2016-01-06 14:26 ` Paul Gortmaker 2016-01-06 15:00 ` Michal Hocko 2016-01-06 15:00 ` Michal Hocko 2015-12-23 23:00 ` Ross Zwisler 2015-12-23 23:00 ` Ross Zwisler 2015-12-24 9:47 ` Michal Hocko 2015-12-24 9:47 ` Michal Hocko 2015-12-24 11:06 ` Tetsuo Handa 2015-12-24 11:06 ` Tetsuo Handa 2015-12-24 20:39 ` Ross Zwisler 2015-12-24 20:39 ` Ross Zwisler 2015-12-25 11:41 ` Michal Hocko 2015-12-25 11:41 ` Michal Hocko 2015-12-24 20:44 ` Ross Zwisler [this message] 2015-12-24 20:44 ` Ross Zwisler 2015-12-25 11:35 ` Michal Hocko 2015-12-25 11:35 ` Michal Hocko 2015-12-25 11:44 ` Michal Hocko 2015-12-25 11:44 ` Michal Hocko 2016-01-06 15:42 [PATCH 0/2 -mm] oom reaper v4 Michal Hocko 2016-01-06 15:42 ` [PATCH 1/2] mm, oom: introduce oom reaper Michal Hocko 2016-01-06 15:42 ` Michal Hocko 2016-01-07 11:23 ` Tetsuo Handa 2016-01-07 11:23 ` Tetsuo Handa 2016-01-07 12:30 ` Michal Hocko 2016-01-07 12:30 ` Michal Hocko 2016-01-11 22:54 ` Andrew Morton 2016-01-11 22:54 ` Andrew Morton 2016-01-12 8:16 ` Michal Hocko 2016-01-12 8:16 ` Michal Hocko 2016-01-28 1:28 ` David Rientjes 2016-01-28 1:28 ` David Rientjes 2016-01-28 21:42 ` Michal Hocko 2016-01-28 21:42 ` Michal Hocko 2016-02-02 3:02 ` David Rientjes 2016-02-02 3:02 ` David Rientjes 2016-02-02 8:57 ` Michal Hocko 2016-02-02 8:57 ` Michal Hocko 2016-02-02 11:48 ` Tetsuo Handa 2016-02-02 11:48 ` Tetsuo Handa 2016-02-02 22:55 ` David Rientjes 2016-02-02 22:55 ` David Rientjes 2016-02-02 22:51 ` David Rientjes 2016-02-02 22:51 ` David Rientjes 2016-02-03 10:31 ` Tetsuo Handa 2016-02-03 10:31 ` Tetsuo Handa
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CAOxpaSXRxJGqL3Fxz5280KZy6xG0ZGwyrf-7i6LArSC0eJsv2A@mail.gmail.com \ --to=zwisler@gmail.com \ --cc=akpm@linux-foundation.org \ --cc=andrea@kernel.org \ --cc=hughd@google.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mgorman@suse.de \ --cc=mhocko@kernel.org \ --cc=oleg@redhat.com \ --cc=penguin-kernel@i-love.sakura.ne.jp \ --cc=riel@redhat.com \ --cc=rientjes@google.com \ --cc=ross.zwisler@linux.intel.com \ --cc=torvalds@linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.