From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752502AbbLXUoH (ORCPT ); Thu, 24 Dec 2015 15:44:07 -0500 Received: from mail-qg0-f67.google.com ([209.85.192.67]:34459 "EHLO mail-qg0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751075AbbLXUoF (ORCPT ); Thu, 24 Dec 2015 15:44:05 -0500 MIME-Version: 1.0 In-Reply-To: <20151224094758.GA22760@dhcp22.suse.cz> References: <1450204575-13052-1-git-send-email-mhocko@kernel.org> <20151224094758.GA22760@dhcp22.suse.cz> Date: Thu, 24 Dec 2015 13:44:03 -0700 Message-ID: Subject: Re: [PATCH 1/2] mm, oom: introduce oom reaper From: Ross Zwisler To: Michal Hocko Cc: Andrew Morton , Mel Gorman , Tetsuo Handa , David Rientjes , Linus Torvalds , Oleg Nesterov , Hugh Dickins , Andrea Argangeli , Rik van Riel , linux-mm@kvack.org, LKML , Ross Zwisler Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Dec 24, 2015 at 2:47 AM, Michal Hocko wrote: > On Wed 23-12-15 16:00:09, Ross Zwisler wrote: > [...] >> While running xfstests on next-20151223 I hit a pair of kernel BUGs >> that bisected to this commit: >> >> 1eb3a80d8239 ("mm, oom: introduce oom reaper") > > Thank you for the report and the bisection. > >> Here is a BUG produced by generic/029 when run against XFS: >> >> [ 235.751723] ------------[ cut here ]------------ >> [ 235.752194] kernel BUG at mm/filemap.c:208! > > This is VM_BUG_ON_PAGE(page_mapped(page), page), right? Could you attach > the full kernel log? It all smells like a race when OOM reaper tears > down the mapping and there is a truncate still in progress. But hitting > the BUG_ON just because of that doesn't make much sense to me. OOM > reaper is essentially MADV_DONTNEED. I have to think about this some > more, though, but I am in a holiday mode until early next year so please > bear with me. The two stack traces were gathered with next-20151223, so the line numbers may have moved around a bit when compared to the actual "mm, oom: introduce oom reaper" commit. > [...] >> [ 235.765638] Call Trace: >> [ 235.765903] [] delete_from_page_cache+0x63/0xd0 >> [ 235.766513] [] truncate_inode_page+0xa5/0x120 >> [ 235.767088] [] truncate_inode_pages_range+0x1a8/0x7f0 >> [ 235.767725] [] ? sched_clock+0x9/0x10 >> [ 235.768239] [] ? local_clock+0x1c/0x20 >> [ 235.768779] [] ? unmap_mapping_range+0x64/0x130 >> [ 235.769385] [] ? unmap_mapping_range+0x74/0x130 >> [ 235.770010] [] ? up_write+0x1f/0x40 >> [ 235.770501] [] ? unmap_mapping_range+0x74/0x130 >> [ 235.771092] [] truncate_pagecache+0x48/0x70 >> [ 235.771646] [] truncate_setsize+0x32/0x40 >> [ 235.772276] [] xfs_setattr_size+0x232/0x470 >> [ 235.772839] [] xfs_vn_setattr+0xb4/0xc0 >> [ 235.773369] [] notify_change+0x237/0x350 >> [ 235.773945] [] do_truncate+0x77/0xc0 >> [ 235.774446] [] do_sys_ftruncate.constprop.15+0xef/0x150 >> [ 235.775156] [] SyS_ftruncate+0xe/0x10 >> [ 235.775650] [] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 235.776257] Code: 5f 5d c3 48 8b 43 20 48 8d 78 ff a8 01 48 0f 44 >> fb 8b 47 48 85 c0 0f 88 2b 01 00 00 48 c7 c6 a8 57 f0 81 48 89 df e8 >> fa 1a 03 00 <0f> 0b 4c 89 ce 44 89 fa 4c 89 e7 4c 89 45 b0 4c 89 4d b8 >> e8 32 >> [ 235.778695] RIP [] __delete_from_page_cache+0x206/0x440 >> [ 235.779350] RSP >> [ 235.779694] ---[ end trace fac9dd65c4cdd828 ]--- >> >> And a different BUG produced by generic/095, also with XFS: >> >> [ 609.398897] ------------[ cut here ]------------ >> [ 609.399843] kernel BUG at mm/truncate.c:629! > > Hmm, I do not see any BUG_ON at this line. But there is > BUG_ON(page_mapped(page)) at line 620. Ditto - check out next-20151223 for real line numbers. >> [ 609.400666] invalid opcode: 0000 [#1] SMP >> [ 609.401512] Modules linked in: nd_pmem nd_btt nd_e820 libnvdimm >> [ 609.402719] CPU: 4 PID: 26782 Comm: fio Tainted: G W > > There was a warning before this triggered. The full kernel log would be > helpful as well. Sure, I can gather full kernel logs, but it'll probably after the new year. > [...] >> [ 609.425325] Call Trace: >> [ 609.425797] [] invalidate_inode_pages2+0x17/0x20 >> [ 609.426971] [] xfs_file_read_iter+0x297/0x300 >> [ 609.428097] [] __vfs_read+0xc9/0x100 >> [ 609.429073] [] vfs_read+0x89/0x130 >> [ 609.430010] [] SyS_read+0x58/0xd0 >> [ 609.430943] [] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 609.432139] Code: 85 d8 fe ff ff 01 00 00 00 f6 c4 40 0f 84 59 ff >> ff ff 49 8b 47 20 48 8d 78 ff a8 01 49 0f 44 ff 8b 47 48 85 c0 0f 88 >> bd 01 00 00 <0f> 0b 4d 3b 67 08 0f 85 70 ff ff ff 49 f7 07 00 18 00 00 >> 74 15 > [...] >> My test setup is a qemu guest machine with a pair of 4 GiB PMEM >> ramdisk test devices, one for the xfstest test disk and one for the >> scratch disk. > > Is this just a plain ramdisk device or it needs a special configuration? Just a plain PMEM ram disk with DAX turned off. Configuration instructions for PMEM can be found here: https://nvdimm.wiki.kernel.org/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qg0-f49.google.com (mail-qg0-f49.google.com [209.85.192.49]) by kanga.kvack.org (Postfix) with ESMTP id 43BCB82F99 for ; Thu, 24 Dec 2015 15:44:04 -0500 (EST) Received: by mail-qg0-f49.google.com with SMTP id o11so102849292qge.2 for ; Thu, 24 Dec 2015 12:44:04 -0800 (PST) Received: from mail-qg0-x243.google.com (mail-qg0-x243.google.com. [2607:f8b0:400d:c04::243]) by mx.google.com with ESMTPS id o7si46562646qho.30.2015.12.24.12.44.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 24 Dec 2015 12:44:03 -0800 (PST) Received: by mail-qg0-x243.google.com with SMTP id o11so1323564qge.0 for ; Thu, 24 Dec 2015 12:44:03 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <20151224094758.GA22760@dhcp22.suse.cz> References: <1450204575-13052-1-git-send-email-mhocko@kernel.org> <20151224094758.GA22760@dhcp22.suse.cz> Date: Thu, 24 Dec 2015 13:44:03 -0700 Message-ID: Subject: Re: [PATCH 1/2] mm, oom: introduce oom reaper From: Ross Zwisler Content-Type: text/plain; charset=UTF-8 Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Andrew Morton , Mel Gorman , Tetsuo Handa , David Rientjes , Linus Torvalds , Oleg Nesterov , Hugh Dickins , Andrea Argangeli , Rik van Riel , linux-mm@kvack.org, LKML , Ross Zwisler On Thu, Dec 24, 2015 at 2:47 AM, Michal Hocko wrote: > On Wed 23-12-15 16:00:09, Ross Zwisler wrote: > [...] >> While running xfstests on next-20151223 I hit a pair of kernel BUGs >> that bisected to this commit: >> >> 1eb3a80d8239 ("mm, oom: introduce oom reaper") > > Thank you for the report and the bisection. > >> Here is a BUG produced by generic/029 when run against XFS: >> >> [ 235.751723] ------------[ cut here ]------------ >> [ 235.752194] kernel BUG at mm/filemap.c:208! > > This is VM_BUG_ON_PAGE(page_mapped(page), page), right? Could you attach > the full kernel log? It all smells like a race when OOM reaper tears > down the mapping and there is a truncate still in progress. But hitting > the BUG_ON just because of that doesn't make much sense to me. OOM > reaper is essentially MADV_DONTNEED. I have to think about this some > more, though, but I am in a holiday mode until early next year so please > bear with me. The two stack traces were gathered with next-20151223, so the line numbers may have moved around a bit when compared to the actual "mm, oom: introduce oom reaper" commit. > [...] >> [ 235.765638] Call Trace: >> [ 235.765903] [] delete_from_page_cache+0x63/0xd0 >> [ 235.766513] [] truncate_inode_page+0xa5/0x120 >> [ 235.767088] [] truncate_inode_pages_range+0x1a8/0x7f0 >> [ 235.767725] [] ? sched_clock+0x9/0x10 >> [ 235.768239] [] ? local_clock+0x1c/0x20 >> [ 235.768779] [] ? unmap_mapping_range+0x64/0x130 >> [ 235.769385] [] ? unmap_mapping_range+0x74/0x130 >> [ 235.770010] [] ? up_write+0x1f/0x40 >> [ 235.770501] [] ? unmap_mapping_range+0x74/0x130 >> [ 235.771092] [] truncate_pagecache+0x48/0x70 >> [ 235.771646] [] truncate_setsize+0x32/0x40 >> [ 235.772276] [] xfs_setattr_size+0x232/0x470 >> [ 235.772839] [] xfs_vn_setattr+0xb4/0xc0 >> [ 235.773369] [] notify_change+0x237/0x350 >> [ 235.773945] [] do_truncate+0x77/0xc0 >> [ 235.774446] [] do_sys_ftruncate.constprop.15+0xef/0x150 >> [ 235.775156] [] SyS_ftruncate+0xe/0x10 >> [ 235.775650] [] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 235.776257] Code: 5f 5d c3 48 8b 43 20 48 8d 78 ff a8 01 48 0f 44 >> fb 8b 47 48 85 c0 0f 88 2b 01 00 00 48 c7 c6 a8 57 f0 81 48 89 df e8 >> fa 1a 03 00 <0f> 0b 4c 89 ce 44 89 fa 4c 89 e7 4c 89 45 b0 4c 89 4d b8 >> e8 32 >> [ 235.778695] RIP [] __delete_from_page_cache+0x206/0x440 >> [ 235.779350] RSP >> [ 235.779694] ---[ end trace fac9dd65c4cdd828 ]--- >> >> And a different BUG produced by generic/095, also with XFS: >> >> [ 609.398897] ------------[ cut here ]------------ >> [ 609.399843] kernel BUG at mm/truncate.c:629! > > Hmm, I do not see any BUG_ON at this line. But there is > BUG_ON(page_mapped(page)) at line 620. Ditto - check out next-20151223 for real line numbers. >> [ 609.400666] invalid opcode: 0000 [#1] SMP >> [ 609.401512] Modules linked in: nd_pmem nd_btt nd_e820 libnvdimm >> [ 609.402719] CPU: 4 PID: 26782 Comm: fio Tainted: G W > > There was a warning before this triggered. The full kernel log would be > helpful as well. Sure, I can gather full kernel logs, but it'll probably after the new year. > [...] >> [ 609.425325] Call Trace: >> [ 609.425797] [] invalidate_inode_pages2+0x17/0x20 >> [ 609.426971] [] xfs_file_read_iter+0x297/0x300 >> [ 609.428097] [] __vfs_read+0xc9/0x100 >> [ 609.429073] [] vfs_read+0x89/0x130 >> [ 609.430010] [] SyS_read+0x58/0xd0 >> [ 609.430943] [] entry_SYSCALL_64_fastpath+0x12/0x76 >> [ 609.432139] Code: 85 d8 fe ff ff 01 00 00 00 f6 c4 40 0f 84 59 ff >> ff ff 49 8b 47 20 48 8d 78 ff a8 01 49 0f 44 ff 8b 47 48 85 c0 0f 88 >> bd 01 00 00 <0f> 0b 4d 3b 67 08 0f 85 70 ff ff ff 49 f7 07 00 18 00 00 >> 74 15 > [...] >> My test setup is a qemu guest machine with a pair of 4 GiB PMEM >> ramdisk test devices, one for the xfstest test disk and one for the >> scratch disk. > > Is this just a plain ramdisk device or it needs a special configuration? Just a plain PMEM ram disk with DAX turned off. Configuration instructions for PMEM can be found here: https://nvdimm.wiki.kernel.org/ -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org