* kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] @ 2020-10-22 0:30 Qian Cai 2020-10-22 0:49 ` Matthew Wilcox 0 siblings, 1 reply; 14+ messages in thread From: Qian Cai @ 2020-10-22 0:30 UTC (permalink / raw) To: Christoph Hellwig, Darrick J. Wong, Matthew Wilcox (Oracle) Cc: linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm Today's linux-next starts to trigger this wondering if anyone has any clue. [ 9765.086947][T48578] LTP: starting iogen01 (export LTPROOT; rwtest -N iogen01 -i 120s -s read,write -Da -Dv -n 2 500b:$TMPDIR/doio.f1.$$ 1000b:$TMPDIR/doio.f2.$$) [ 9839.423703][T97227] ------------[ cut here ]------------ [ 9839.429819][T97227] kernel BUG at mm/page-writeback.c:2241! [ 9839.435459][T97227] invalid opcode: 0000 [#1] SMP KASAN PTI [ 9839.441066][T97227] CPU: 56 PID: 97227 Comm: doio Tainted: G IO 5.9.0-next-20201021 #1 [ 9839.450251][T97227] Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 11/13/2019 [ 9839.459532][T97227] RIP: 0010:write_cache_pages+0x95f/0xeb0 [ 9839.465137][T97227] Code: 03 80 3c 02 00 0f 85 e5 04 00 00 49 8b 46 08 48 c7 c6 40 fb ca 9c 48 8d 50 ff a8 01 4c 0f 45 f2 4c 89 f7 e8 33 e3 08 00 0f 0b <0f> 0b 3d 00 00 08 00 0f 84 c3 00 00 00 48 8b 54 24 30 48 c1 ea 03 [ 9839.484715][T97227] RSP: 0018:ffffc9003063f610 EFLAGS: 00010282 [ 9839.490672][T97227] RAX: 01bfffc00000803f RBX: ffffea0024e68500 RCX: ffffffff9b8d232e [ 9839.498547][T97227] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffea0024e68500 [ 9839.506422][T97227] RBP: ffffc9003063f708 R08: fffff940049cd0a1 R09: fffff940049cd0a1 [ 9839.514297][T97227] R10: ffffea0024e68507 R11: fffff940049cd0a0 R12: ffffc9003063fa20 [ 9839.522171][T97227] R13: ffffea0024e68500 R14: ffffea0024e68500 R15: dffffc0000000000 [ 9839.530044][T97227] FS: 00007f23ef12a740(0000) GS:ffff88a01f280000(0000) knlGS:0000000000000000 [ 9839.538878][T97227] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9839.545355][T97227] CR2: 0000000001c79000 CR3: 0000000c15786004 CR4: 00000000007706e0 [ 9839.553229][T97227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 9839.561104][T97227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 9839.568976][T97227] PKRU: 55555554 [ 9839.572395][T97227] Call Trace: [ 9839.575559][T97227] ? iomap_writepage_map+0x23a0/0x23a0 [ 9839.580900][T97227] ? clear_page_dirty_for_io+0x990/0x990 [ 9839.586420][T97227] ? rcu_read_lock_sched_held+0x9c/0xd0 [ 9839.591850][T97227] ? rcu_read_lock_bh_held+0xb0/0xb0 [ 9839.597021][T97227] ? find_held_lock+0x33/0x1c0 [ 9839.601670][T97227] ? xfs_vm_writepages+0xc2/0x130 [ 9839.606575][T97227] ? lock_downgrade+0x700/0x700 [ 9839.611305][T97227] ? rcu_read_unlock+0x40/0x40 [ 9839.615949][T97227] ? do_raw_spin_lock+0x121/0x290 [ 9839.620854][T97227] ? rwlock_bug.part.1+0x90/0x90 [ 9839.625669][T97227] iomap_writepages+0x3f/0xb0 iomap_writepages at fs/iomap/buffered-io.c:1576 [ 9839.630225][T97227] xfs_vm_writepages+0xd7/0x130 [ 9839.634955][T97227] ? xfs_vm_readahead+0x10/0x10 [ 9839.639686][T97227] ? find_held_lock+0x33/0x1c0 [ 9839.644327][T97227] do_writepages+0xcd/0x250 do_writepages at mm/page-writeback.c:2355 [ 9839.648707][T97227] ? page_writeback_cpu_online+0x10/0x10 [ 9839.654224][T97227] ? do_raw_spin_lock+0x121/0x290 [ 9839.659129][T97227] ? rwlock_bug.part.1+0x90/0x90 [ 9839.663945][T97227] ? rcu_read_lock_bh_held+0xb0/0xb0 [ 9839.669113][T97227] __filemap_fdatawrite_range+0x250/0x310 __filemap_fdatawrite_range at mm/filemap.c:423 [ 9839.674717][T97227] ? delete_from_page_cache_batch+0xaa0/0xaa0 [ 9839.680669][T97227] ? rcu_read_lock_bh_held+0xb0/0xb0 [ 9839.685836][T97227] ? rcu_read_lock_sched_held+0x9c/0xd0 [ 9839.691265][T97227] file_write_and_wait_range+0x85/0xe0 file_write_and_wait_range at mm/filemap.c:761 [ 9839.696607][T97227] xfs_file_fsync+0x192/0x710 fs_file_fsync at fs/xfs/xfs_file.c:105 [ 9839.701163][T97227] ? xfs_file_read_iter+0x490/0x490 [ 9839.706242][T97227] ? up_write+0x148/0x460 [ 9839.710446][T97227] ? iomap_write_begin+0xde0/0xde0 [ 9839.715438][T97227] xfs_file_buffered_aio_write+0x82a/0xa30 generic_write_sync at include/linux/fs.h:2727 (inlined by) xfs_file_buffered_aio_write at fs/xfs/xfs_file.c:684 [ 9839.721129][T97227] ? xfs_file_aio_write_checks+0x620/0x620 [ 9839.726820][T97227] ? lockdep_hardirqs_on_prepare+0x3d0/0x3d0 [ 9839.732691][T97227] new_sync_write+0x3aa/0x610 [ 9839.737247][T97227] ? new_sync_read+0x600/0x600 [ 9839.741888][T97227] ? vfs_write+0x36c/0x5b0 [ 9839.746181][T97227] ? rcu_read_lock_any_held+0xcd/0xf0 [ 9839.751433][T97227] vfs_write+0x3e9/0x5b0 ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-22 0:30 kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] Qian Cai @ 2020-10-22 0:49 ` Matthew Wilcox 2020-10-22 13:23 ` William Kucharski ` (2 more replies) 0 siblings, 3 replies; 14+ messages in thread From: Matthew Wilcox @ 2020-10-22 0:49 UTC (permalink / raw) To: Qian Cai Cc: Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: > Today's linux-next starts to trigger this wondering if anyone has any clue. I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE to try to get a clue about it. Good to know it's not the THP patches since they aren't in linux-next. I don't understand how it can happen. We have the page locked, and then we do: if (PageWriteback(page)) { if (wbc->sync_mode != WB_SYNC_NONE) wait_on_page_writeback(page); else goto continue_unlock; } VM_BUG_ON_PAGE(PageWriteback(page), page); Nobody should be able to put this page under writeback while we have it locked ... right? The page can be redirtied by the code that's supposed to be writing it back, but I don't see how anyone can make PageWriteback true while we're holding the page lock. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-22 0:49 ` Matthew Wilcox @ 2020-10-22 13:23 ` William Kucharski 2020-10-22 16:46 ` Matthew Wilcox 2020-10-22 15:35 ` Qian Cai 2020-10-26 9:49 ` Jan Kara 2 siblings, 1 reply; 14+ messages in thread From: William Kucharski @ 2020-10-22 13:23 UTC (permalink / raw) To: Matthew Wilcox Cc: Qian Cai, Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm > On Oct 21, 2020, at 6:49 PM, Matthew Wilcox <willy@infradead.org> wrote: > > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: >> Today's linux-next starts to trigger this wondering if anyone has any clue. > > I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE > to try to get a clue about it. Good to know it's not the THP patches > since they aren't in linux-next. > > I don't understand how it can happen. We have the page locked, and then we do: > > if (PageWriteback(page)) { > if (wbc->sync_mode != WB_SYNC_NONE) > wait_on_page_writeback(page); > else > goto continue_unlock; > } > > VM_BUG_ON_PAGE(PageWriteback(page), page); > > Nobody should be able to put this page under writeback while we have it > locked ... right? The page can be redirtied by the code that's supposed > to be writing it back, but I don't see how anyone can make PageWriteback > true while we're holding the page lock. Looking at __test_set_page_writeback(), I see that it (and most other callers to lock_page_memcg()) do the following: lock_page_memcg(page) /* do other stuff */ ret = TestSetPageWriteback(page); /* do more stuff */ unlock_page_memcg(page) yet lock_page_memcg() does have a few cases where it can (silently) return NULL to indicate an error. Only test_clear_page_writeback() actually saves off the return value (but it too never bothers to check whether it is NULL or not.) Could it be one of those error conditions is occurring leading to no lock actually being taken? The conditions would be extremely rare, but it feels wrong not to check somewhere: struct page *head = compound_head(page); /* rmap on tail pages */ [ ... ] if (mem_cgroup_disabled()) return NULL; again: memcg = head->mem_cgroup; if (unlikely(!memcg)) return NULL; ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-22 13:23 ` William Kucharski @ 2020-10-22 16:46 ` Matthew Wilcox 0 siblings, 0 replies; 14+ messages in thread From: Matthew Wilcox @ 2020-10-22 16:46 UTC (permalink / raw) To: William Kucharski Cc: Qian Cai, Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm On Thu, Oct 22, 2020 at 07:23:33AM -0600, William Kucharski wrote: > > > > On Oct 21, 2020, at 6:49 PM, Matthew Wilcox <willy@infradead.org> wrote: > > > > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: > >> Today's linux-next starts to trigger this wondering if anyone has any clue. > > > > I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE > > to try to get a clue about it. Good to know it's not the THP patches > > since they aren't in linux-next. > > > > I don't understand how it can happen. We have the page locked, and then we do: > > > > if (PageWriteback(page)) { > > if (wbc->sync_mode != WB_SYNC_NONE) > > wait_on_page_writeback(page); > > else > > goto continue_unlock; > > } > > > > VM_BUG_ON_PAGE(PageWriteback(page), page); > > > > Nobody should be able to put this page under writeback while we have it > > locked ... right? The page can be redirtied by the code that's supposed > > to be writing it back, but I don't see how anyone can make PageWriteback > > true while we're holding the page lock. > > Looking at __test_set_page_writeback(), I see that it (and most other > callers to lock_page_memcg()) do the following: lock_page_memcg() is, unfortunately, completely unrelated to lock_page(). I believe all callers of __test_set_page_writeback() have the page lock held already, but I'm going to put in an assert to that effect. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-22 0:49 ` Matthew Wilcox 2020-10-22 13:23 ` William Kucharski @ 2020-10-22 15:35 ` Qian Cai 2020-10-22 17:12 ` Matthew Wilcox 2020-10-26 9:49 ` Jan Kara 2 siblings, 1 reply; 14+ messages in thread From: Qian Cai @ 2020-10-22 15:35 UTC (permalink / raw) To: Matthew Wilcox Cc: Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm On Thu, 2020-10-22 at 01:49 +0100, Matthew Wilcox wrote: > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: > > Today's linux-next starts to trigger this wondering if anyone has any clue. > > I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE > to try to get a clue about it. Good to know it's not the THP patches > since they aren't in linux-next. > > I don't understand how it can happen. We have the page locked, and then we > do: > > if (PageWriteback(page)) { > if (wbc->sync_mode != WB_SYNC_NONE) > wait_on_page_writeback(page); > else > goto continue_unlock; > } > > VM_BUG_ON_PAGE(PageWriteback(page), page); > > Nobody should be able to put this page under writeback while we have it > locked ... right? The page can be redirtied by the code that's supposed > to be writing it back, but I don't see how anyone can make PageWriteback > true while we're holding the page lock. It happened again on today's linux-next: [ 7613.579890][T55770] page:00000000a4b35e02 refcount:3 mapcount:0 mapping:00000000457ceb87 index:0x3e pfn:0x1cef4e [ 7613.590594][T55770] aops:xfs_address_space_operations ino:805d85a dentry name:"doio.f1.55762" [ 7613.599192][T55770] flags: 0xbfffc0000000bf(locked|waiters|referenced|uptodate|dirty|lru|active) [ 7613.608596][T55770] raw: 00bfffc0000000bf ffffea0005027d48 ffff88810eaec030 ffff888231f3a6a8 [ 7613.617101][T55770] raw: 000000000000003e 0000000000000000 00000003ffffffff ffff888143724000 [ 7613.625590][T55770] page dumped because: VM_BUG_ON_PAGE(PageWriteback(page)) [ 7613.632695][T55770] page->mem_cgroup:ffff888143724000 ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-22 15:35 ` Qian Cai @ 2020-10-22 17:12 ` Matthew Wilcox 2020-10-30 12:08 ` Qian Cai 0 siblings, 1 reply; 14+ messages in thread From: Matthew Wilcox @ 2020-10-22 17:12 UTC (permalink / raw) To: Qian Cai Cc: Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm On Thu, Oct 22, 2020 at 11:35:26AM -0400, Qian Cai wrote: > On Thu, 2020-10-22 at 01:49 +0100, Matthew Wilcox wrote: > > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: > > > Today's linux-next starts to trigger this wondering if anyone has any clue. > > > > I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE > > to try to get a clue about it. Good to know it's not the THP patches > > since they aren't in linux-next. > > > > I don't understand how it can happen. We have the page locked, and then we > > do: > > > > if (PageWriteback(page)) { > > if (wbc->sync_mode != WB_SYNC_NONE) > > wait_on_page_writeback(page); > > else > > goto continue_unlock; > > } > > > > VM_BUG_ON_PAGE(PageWriteback(page), page); > > > > Nobody should be able to put this page under writeback while we have it > > locked ... right? The page can be redirtied by the code that's supposed > > to be writing it back, but I don't see how anyone can make PageWriteback > > true while we're holding the page lock. > > It happened again on today's linux-next: > > [ 7613.579890][T55770] page:00000000a4b35e02 refcount:3 mapcount:0 mapping:00000000457ceb87 index:0x3e pfn:0x1cef4e > [ 7613.590594][T55770] aops:xfs_address_space_operations ino:805d85a dentry name:"doio.f1.55762" > [ 7613.599192][T55770] flags: 0xbfffc0000000bf(locked|waiters|referenced|uptodate|dirty|lru|active) > [ 7613.608596][T55770] raw: 00bfffc0000000bf ffffea0005027d48 ffff88810eaec030 ffff888231f3a6a8 > [ 7613.617101][T55770] raw: 000000000000003e 0000000000000000 00000003ffffffff ffff888143724000 > [ 7613.625590][T55770] page dumped because: VM_BUG_ON_PAGE(PageWriteback(page)) > [ 7613.632695][T55770] page->mem_cgroup:ffff888143724000 Seems like it reproduces for you pretty quickly. I have no luck ;-( Can you add this? +++ b/mm/page-writeback.c @@ -2774,6 +2774,7 @@ int __test_set_page_writeback(struct page *page, bool keep_write) struct address_space *mapping = page_mapping(page); int ret, access_ret; + VM_BUG_ON_PAGE(!PageLocked(page), page); lock_page_memcg(page); if (mapping && mapping_use_writeback_tags(mapping)) { XA_STATE(xas, &mapping->i_pages, page_index(page)); This is the only place (afaict) that sets PageWriteback, so that will tell us whether someone is setting Writeback without holding the lock, or whether we're suffering from a spurious wakeup. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-22 17:12 ` Matthew Wilcox @ 2020-10-30 12:08 ` Qian Cai 0 siblings, 0 replies; 14+ messages in thread From: Qian Cai @ 2020-10-30 12:08 UTC (permalink / raw) To: Matthew Wilcox Cc: Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm On Thu, 2020-10-22 at 18:12 +0100, Matthew Wilcox wrote: > On Thu, Oct 22, 2020 at 11:35:26AM -0400, Qian Cai wrote: > > On Thu, 2020-10-22 at 01:49 +0100, Matthew Wilcox wrote: > > > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: > > > > Today's linux-next starts to trigger this wondering if anyone has any > > > > clue. > > > > > > I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE > > > to try to get a clue about it. Good to know it's not the THP patches > > > since they aren't in linux-next. > > > > > > I don't understand how it can happen. We have the page locked, and then > > > we > > > do: > > > > > > if (PageWriteback(page)) { > > > if (wbc->sync_mode != WB_SYNC_NONE) > > > wait_on_page_writeback(page); > > > else > > > goto continue_unlock; > > > } > > > > > > VM_BUG_ON_PAGE(PageWriteback(page), page); > > > > > > Nobody should be able to put this page under writeback while we have it > > > locked ... right? The page can be redirtied by the code that's supposed > > > to be writing it back, but I don't see how anyone can make PageWriteback > > > true while we're holding the page lock. > > > > It happened again on today's linux-next: > > > > [ 7613.579890][T55770] page:00000000a4b35e02 refcount:3 mapcount:0 > > mapping:00000000457ceb87 index:0x3e pfn:0x1cef4e > > [ 7613.590594][T55770] aops:xfs_address_space_operations ino:805d85a dentry > > name:"doio.f1.55762" > > [ 7613.599192][T55770] flags: > > 0xbfffc0000000bf(locked|waiters|referenced|uptodate|dirty|lru|active) > > [ 7613.608596][T55770] raw: 00bfffc0000000bf ffffea0005027d48 > > ffff88810eaec030 ffff888231f3a6a8 > > [ 7613.617101][T55770] raw: 000000000000003e 0000000000000000 > > 00000003ffffffff ffff888143724000 > > [ 7613.625590][T55770] page dumped because: > > VM_BUG_ON_PAGE(PageWriteback(page)) > > [ 7613.632695][T55770] page->mem_cgroup:ffff888143724000 > > Seems like it reproduces for you pretty quickly. I have no luck ;-( > > Can you add this? It turns out I had no luck for the last a few days. I'll keep running and report back if it triggers again. > > +++ b/mm/page-writeback.c > @@ -2774,6 +2774,7 @@ int __test_set_page_writeback(struct page *page, bool > keep_write) > struct address_space *mapping = page_mapping(page); > int ret, access_ret; > > + VM_BUG_ON_PAGE(!PageLocked(page), page); > lock_page_memcg(page); > if (mapping && mapping_use_writeback_tags(mapping)) { > XA_STATE(xas, &mapping->i_pages, page_index(page)); > > This is the only place (afaict) that sets PageWriteback, so that will > tell us whether someone is setting Writeback without holding the lock, > or whether we're suffering from a spurious wakeup. > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-22 0:49 ` Matthew Wilcox 2020-10-22 13:23 ` William Kucharski 2020-10-22 15:35 ` Qian Cai @ 2020-10-26 9:49 ` Jan Kara 2020-10-26 13:13 ` Matthew Wilcox 2 siblings, 1 reply; 14+ messages in thread From: Jan Kara @ 2020-10-26 9:49 UTC (permalink / raw) To: Matthew Wilcox Cc: Qian Cai, Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm On Thu 22-10-20 01:49:06, Matthew Wilcox wrote: > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: > > Today's linux-next starts to trigger this wondering if anyone has any clue. > > I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE > to try to get a clue about it. Good to know it's not the THP patches > since they aren't in linux-next. > > I don't understand how it can happen. We have the page locked, and then we do: > > if (PageWriteback(page)) { > if (wbc->sync_mode != WB_SYNC_NONE) > wait_on_page_writeback(page); > else > goto continue_unlock; > } > > VM_BUG_ON_PAGE(PageWriteback(page), page); > > Nobody should be able to put this page under writeback while we have it > locked ... right? The page can be redirtied by the code that's supposed > to be writing it back, but I don't see how anyone can make PageWriteback > true while we're holding the page lock. FWIW here's very similar report for ext4 [1] and I strongly suspect this started happening after Linus' rewrite of the page bit waiting logic. Linus thinks it's preexisting bug which just got exposed by his changes (which is possible). I've been searching a culprit for some time but so far I failed. It's good to know it isn't ext4 specific so we should be searching in the generic code ;). So far I was concentrating more on ext4 bits... Honza [1] https://lore.kernel.org/lkml/000000000000d3a33205add2f7b2@google.com/ -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-26 9:49 ` Jan Kara @ 2020-10-26 13:13 ` Matthew Wilcox 2020-10-26 13:55 ` Jens Axboe 0 siblings, 1 reply; 14+ messages in thread From: Matthew Wilcox @ 2020-10-26 13:13 UTC (permalink / raw) To: Jan Kara Cc: Qian Cai, Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, Jens Axboe, linux-mm On Mon, Oct 26, 2020 at 10:49:48AM +0100, Jan Kara wrote: > On Thu 22-10-20 01:49:06, Matthew Wilcox wrote: > > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: > > > Today's linux-next starts to trigger this wondering if anyone has any clue. > > > > I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE > > to try to get a clue about it. Good to know it's not the THP patches > > since they aren't in linux-next. > > > > I don't understand how it can happen. We have the page locked, and then we do: > > > > if (PageWriteback(page)) { > > if (wbc->sync_mode != WB_SYNC_NONE) > > wait_on_page_writeback(page); > > else > > goto continue_unlock; > > } > > > > VM_BUG_ON_PAGE(PageWriteback(page), page); > > > > Nobody should be able to put this page under writeback while we have it > > locked ... right? The page can be redirtied by the code that's supposed > > to be writing it back, but I don't see how anyone can make PageWriteback > > true while we're holding the page lock. > > FWIW here's very similar report for ext4 [1] and I strongly suspect this > started happening after Linus' rewrite of the page bit waiting logic. Linus > thinks it's preexisting bug which just got exposed by his changes (which is > possible). I've been searching a culprit for some time but so far I failed. > It's good to know it isn't ext4 specific so we should be searching in the > generic code ;). So far I was concentrating more on ext4 bits... > > Honza > > [1] https://lore.kernel.org/lkml/000000000000d3a33205add2f7b2@google.com/ Oh good, I was wondering if it was an XFS bug ;-) I hope Qian gets it to reproduce soon with the assert because that will tell us whether it's a spurious wakeup or someone calling SetPageWriteback without holding the page lock. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-26 13:13 ` Matthew Wilcox @ 2020-10-26 13:55 ` Jens Axboe 2020-10-26 14:26 ` Qian Cai 2020-10-26 14:51 ` Qian Cai 0 siblings, 2 replies; 14+ messages in thread From: Jens Axboe @ 2020-10-26 13:55 UTC (permalink / raw) To: Matthew Wilcox, Jan Kara Cc: Qian Cai, Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, linux-mm On 10/26/20 7:13 AM, Matthew Wilcox wrote: > On Mon, Oct 26, 2020 at 10:49:48AM +0100, Jan Kara wrote: >> On Thu 22-10-20 01:49:06, Matthew Wilcox wrote: >>> On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote: >>>> Today's linux-next starts to trigger this wondering if anyone has any clue. >>> >>> I've seen that occasionally too. I changed that BUG_ON to VM_BUG_ON_PAGE >>> to try to get a clue about it. Good to know it's not the THP patches >>> since they aren't in linux-next. >>> >>> I don't understand how it can happen. We have the page locked, and then we do: >>> >>> if (PageWriteback(page)) { >>> if (wbc->sync_mode != WB_SYNC_NONE) >>> wait_on_page_writeback(page); >>> else >>> goto continue_unlock; >>> } >>> >>> VM_BUG_ON_PAGE(PageWriteback(page), page); >>> >>> Nobody should be able to put this page under writeback while we have it >>> locked ... right? The page can be redirtied by the code that's supposed >>> to be writing it back, but I don't see how anyone can make PageWriteback >>> true while we're holding the page lock. >> >> FWIW here's very similar report for ext4 [1] and I strongly suspect this >> started happening after Linus' rewrite of the page bit waiting logic. Linus >> thinks it's preexisting bug which just got exposed by his changes (which is >> possible). I've been searching a culprit for some time but so far I failed. >> It's good to know it isn't ext4 specific so we should be searching in the >> generic code ;). So far I was concentrating more on ext4 bits... >> >> Honza >> >> [1] https://lore.kernel.org/lkml/000000000000d3a33205add2f7b2@google.com/ > > Oh good, I was wondering if it was an XFS bug ;-) > > I hope Qian gets it to reproduce soon with the assert because that will > tell us whether it's a spurious wakeup or someone calling SetPageWriteback > without holding the page lock. I've tried to reproduce this as well, to no avail. Qian, could you perhaps detail the setup? What kind of storage, kernel config, compiler, etc. -- Jens Axboe ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-26 13:55 ` Jens Axboe @ 2020-10-26 14:26 ` Qian Cai 2020-11-04 15:16 ` Jan Kara 2020-10-26 14:51 ` Qian Cai 1 sibling, 1 reply; 14+ messages in thread From: Qian Cai @ 2020-10-26 14:26 UTC (permalink / raw) To: Jens Axboe, Matthew Wilcox, Jan Kara Cc: Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, linux-mm On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote: > I've tried to reproduce this as well, to no avail. Qian, could you perhaps > detail the setup? What kind of storage, kernel config, compiler, etc. > So far I have only been able to reproduce on this Intel platform: HPE DL560 gen10 Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz 131072 MB memory, 1000 GB disk space (smartpqi nvme) It was running some CPU and memory hotplug, KVM and then LTP workloads (syscalls, mm, and fs). Finally, it was always this LTP test case to trigger it: # export LTPROOT; rwtest -N iogen01 -i 120s -s read,write -Da -Dv -n 2 500b:$TMPDIR/doio.f1.$$ 1000b:$TMPDIR/doio.f2.$$ https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/fs/doio/rwtest gcc-8.3.1-5.1.el8.x86_64 .config: https://git.code.tencent.com/cail/linux-mm/blob/master/x86.config == storage information == [ 46.131150] smartpqi 0000:23:00.0: Online Firmware Activation enabled [ 46.138495] smartpqi 0000:23:00.0: Serial Management Protocol enabled [ 46.145701] smartpqi 0000:23:00.0: New Soft Reset Handshake enabled [ 46.152705] smartpqi 0000:23:00.0: RAID IU Timeout enabled [ 46.158934] smartpqi 0000:23:00.0: TMF IU Timeout enabled [ 46.168740] scsi host0: smartpqi [ 47.750425] nvme nvme1: 31/0/0 default/read/poll queues [ 47.826211] scsi 0:0:0:0: Direct-Access ATA MM1000GEFQV HPG8 PQ: 0 ANSI: 6 [ 47.841752] smartpqi 0000:23:00.0: added 0:0:0:0 0000000000000000 Direct-Access ATA MM1000GEFQV AIO+ qd=32 [ 47.941249] nvme1n1: p1 [ 47.943078] scsi 0:0:1:0: Enclosure HPE Smart Adapter 2.62 PQ: 0 ANSI: 5 [ 47.958506] smartpqi 0000:23:00.0: added 0:0:1:0 51402ec001f36448 Enclosure HPE Smart Adapter AIO- [ 47.962844] nvme nvme0: 31/0/0 default/read/poll queues [ 48.015511] scsi 0:2:0:0: RAID HPE P408i-a SR Gen10 2.62 PQ: 0 ANSI: 5 [ 48.029736] smartpqi 0000:23:00.0: added 0:2:0:0 0000000000000000 RAID HPE P408i-a SR Gen10 [ 48.042711] smartpqi 0000:4e:00.0: Microsemi Smart Family Controller found [ 48.149820] nvme0n1: p1 [ 48.956194] smartpqi 0000:4e:00.0: Online Firmware Activation enabled [ 48.963399] smartpqi 0000:4e:00.0: Serial Management Protocol enabled [ 48.970625] smartpqi 0000:4e:00.0: New Soft Reset Handshake enabled [ 48.977645] smartpqi 0000:4e:00.0: RAID IU Timeout enabled [ 48.983873] smartpqi 0000:4e:00.0: TMF IU Timeout enabled [ 48.994687] scsi host1: smartpqi [ 50.612955] scsi 1:0:0:0: Enclosure HPE Smart Adapter 2.62 PQ: 0 ANSI: 5 [ 50.628219] smartpqi 0000:4e:00.0: added 1:0:0:0 51402ec01040ffc8 Enclosure HPE Smart Adapter AIO- [ 50.681859] scsi 1:2:0:0: RAID HPE E208i-p SR Gen10 2.62 PQ: 0 ANSI: 5 [ 50.695843] smartpqi 0000:4e:00.0: added 1:2:0:0 0000000000000000 RAID HPE E208i-p SR Gen10 [ 50.856683] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB) [ 50.865195] sd 0:0:0:0: [sda] 4096-byte physical blocks [ 50.871354] sd 0:0:0:0: [sda] Write Protect is off [ 50.876956] sd 0:0:0:0: [sda] Mode Sense: 46 00 10 08 [ 50.877299] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA [ 50.898824] sda: sda1 sda2 sda3 [ 50.943835] sd 0:0:0:0: [sda] Attached SCSI disk ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-26 14:26 ` Qian Cai @ 2020-11-04 15:16 ` Jan Kara 2020-11-04 15:40 ` Qian Cai 0 siblings, 1 reply; 14+ messages in thread From: Jan Kara @ 2020-11-04 15:16 UTC (permalink / raw) To: Qian Cai Cc: Jens Axboe, Matthew Wilcox, Jan Kara, Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, linux-mm On Mon 26-10-20 10:26:26, Qian Cai wrote: > On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote: > > I've tried to reproduce this as well, to no avail. Qian, could you perhaps > > detail the setup? What kind of storage, kernel config, compiler, etc. > > > > So far I have only been able to reproduce on this Intel platform: > > HPE DL560 gen10 > Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz > 131072 MB memory, 1000 GB disk space (smartpqi nvme) Did you try running with the debug patch Matthew sent? Any results? Honza -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-11-04 15:16 ` Jan Kara @ 2020-11-04 15:40 ` Qian Cai 0 siblings, 0 replies; 14+ messages in thread From: Qian Cai @ 2020-11-04 15:40 UTC (permalink / raw) To: Jan Kara Cc: Jens Axboe, Matthew Wilcox, Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, linux-mm On Wed, 2020-11-04 at 16:16 +0100, Jan Kara wrote: > On Mon 26-10-20 10:26:26, Qian Cai wrote: > > On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote: > > > I've tried to reproduce this as well, to no avail. Qian, could you perhaps > > > detail the setup? What kind of storage, kernel config, compiler, etc. > > > > > > > So far I have only been able to reproduce on this Intel platform: > > > > HPE DL560 gen10 > > Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz > > 131072 MB memory, 1000 GB disk space (smartpqi nvme) > > Did you try running with the debug patch Matthew sent? Any results? Running every day, but no luck so far. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] 2020-10-26 13:55 ` Jens Axboe 2020-10-26 14:26 ` Qian Cai @ 2020-10-26 14:51 ` Qian Cai 1 sibling, 0 replies; 14+ messages in thread From: Qian Cai @ 2020-10-26 14:51 UTC (permalink / raw) To: Jens Axboe, Matthew Wilcox, Jan Kara Cc: Christoph Hellwig, Darrick J. Wong, linux-xfs, linux-fsdevel, linux-kernel, linux-mm On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote: > I've tried to reproduce this as well, to no avail. Qian, could you perhaps > detail the setup? What kind of storage, kernel config, compiler, etc. This should work: https://gitlab.com/cailca/linux-mm/-/blob/master/x86.config ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2020-11-04 15:40 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-10-22 0:30 kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ] Qian Cai 2020-10-22 0:49 ` Matthew Wilcox 2020-10-22 13:23 ` William Kucharski 2020-10-22 16:46 ` Matthew Wilcox 2020-10-22 15:35 ` Qian Cai 2020-10-22 17:12 ` Matthew Wilcox 2020-10-30 12:08 ` Qian Cai 2020-10-26 9:49 ` Jan Kara 2020-10-26 13:13 ` Matthew Wilcox 2020-10-26 13:55 ` Jens Axboe 2020-10-26 14:26 ` Qian Cai 2020-11-04 15:16 ` Jan Kara 2020-11-04 15:40 ` Qian Cai 2020-10-26 14:51 ` Qian Cai
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).