kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Qian Cai]

Today's linux-next starts to trigger this wondering if anyone has any clue.

[ 9765.086947][T48578] LTP: starting iogen01 (export LTPROOT; rwtest -N iogen01 -i 120s -s read,write -Da -Dv -n 2 500b:$TMPDIR/doio.f1.$$ 1000b:$TMPDIR/doio.f2.$$)
[ 9839.423703][T97227] ------------[ cut here ]------------
[ 9839.429819][T97227] kernel BUG at mm/page-writeback.c:2241!
[ 9839.435459][T97227] invalid opcode: 0000 [#1] SMP KASAN PTI
[ 9839.441066][T97227] CPU: 56 PID: 97227 Comm: doio Tainted: G          IO      5.9.0-next-20201021 #1
[ 9839.450251][T97227] Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 11/13/2019
[ 9839.459532][T97227] RIP: 0010:write_cache_pages+0x95f/0xeb0
[ 9839.465137][T97227] Code: 03 80 3c 02 00 0f 85 e5 04 00 00 49 8b 46 08 48 c7 c6 40 fb ca 9c 48 8d 50 ff a8 01 4c 0f 45 f2 4c 89 f7 e8 33 e3 08 00 0f 0b <0f> 0b 3d 00 00 08 00 0f 84 c3 00 00 00 48 8b 54 24 30 48 c1 ea 03
[ 9839.484715][T97227] RSP: 0018:ffffc9003063f610 EFLAGS: 00010282
[ 9839.490672][T97227] RAX: 01bfffc00000803f RBX: ffffea0024e68500 RCX: ffffffff9b8d232e
[ 9839.498547][T97227] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffea0024e68500
[ 9839.506422][T97227] RBP: ffffc9003063f708 R08: fffff940049cd0a1 R09: fffff940049cd0a1
[ 9839.514297][T97227] R10: ffffea0024e68507 R11: fffff940049cd0a0 R12: ffffc9003063fa20
[ 9839.522171][T97227] R13: ffffea0024e68500 R14: ffffea0024e68500 R15: dffffc0000000000
[ 9839.530044][T97227] FS:  00007f23ef12a740(0000) GS:ffff88a01f280000(0000) knlGS:0000000000000000
[ 9839.538878][T97227] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 9839.545355][T97227] CR2: 0000000001c79000 CR3: 0000000c15786004 CR4: 00000000007706e0
[ 9839.553229][T97227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9839.561104][T97227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 9839.568976][T97227] PKRU: 55555554
[ 9839.572395][T97227] Call Trace:
[ 9839.575559][T97227]  ? iomap_writepage_map+0x23a0/0x23a0
[ 9839.580900][T97227]  ? clear_page_dirty_for_io+0x990/0x990
[ 9839.586420][T97227]  ? rcu_read_lock_sched_held+0x9c/0xd0
[ 9839.591850][T97227]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 9839.597021][T97227]  ? find_held_lock+0x33/0x1c0
[ 9839.601670][T97227]  ? xfs_vm_writepages+0xc2/0x130
[ 9839.606575][T97227]  ? lock_downgrade+0x700/0x700
[ 9839.611305][T97227]  ? rcu_read_unlock+0x40/0x40
[ 9839.615949][T97227]  ? do_raw_spin_lock+0x121/0x290
[ 9839.620854][T97227]  ? rwlock_bug.part.1+0x90/0x90
[ 9839.625669][T97227]  iomap_writepages+0x3f/0xb0
iomap_writepages at fs/iomap/buffered-io.c:1576
[ 9839.630225][T97227]  xfs_vm_writepages+0xd7/0x130
[ 9839.634955][T97227]  ? xfs_vm_readahead+0x10/0x10
[ 9839.639686][T97227]  ? find_held_lock+0x33/0x1c0
[ 9839.644327][T97227]  do_writepages+0xcd/0x250
do_writepages at mm/page-writeback.c:2355
[ 9839.648707][T97227]  ? page_writeback_cpu_online+0x10/0x10
[ 9839.654224][T97227]  ? do_raw_spin_lock+0x121/0x290
[ 9839.659129][T97227]  ? rwlock_bug.part.1+0x90/0x90
[ 9839.663945][T97227]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 9839.669113][T97227]  __filemap_fdatawrite_range+0x250/0x310
__filemap_fdatawrite_range at mm/filemap.c:423
[ 9839.674717][T97227]  ? delete_from_page_cache_batch+0xaa0/0xaa0
[ 9839.680669][T97227]  ? rcu_read_lock_bh_held+0xb0/0xb0
[ 9839.685836][T97227]  ? rcu_read_lock_sched_held+0x9c/0xd0
[ 9839.691265][T97227]  file_write_and_wait_range+0x85/0xe0
file_write_and_wait_range at mm/filemap.c:761
[ 9839.696607][T97227]  xfs_file_fsync+0x192/0x710
fs_file_fsync at fs/xfs/xfs_file.c:105
[ 9839.701163][T97227]  ? xfs_file_read_iter+0x490/0x490
[ 9839.706242][T97227]  ? up_write+0x148/0x460
[ 9839.710446][T97227]  ? iomap_write_begin+0xde0/0xde0
[ 9839.715438][T97227]  xfs_file_buffered_aio_write+0x82a/0xa30
generic_write_sync at include/linux/fs.h:2727
(inlined by) xfs_file_buffered_aio_write at fs/xfs/xfs_file.c:684
[ 9839.721129][T97227]  ? xfs_file_aio_write_checks+0x620/0x620
[ 9839.726820][T97227]  ? lockdep_hardirqs_on_prepare+0x3d0/0x3d0
[ 9839.732691][T97227]  new_sync_write+0x3aa/0x610
[ 9839.737247][T97227]  ? new_sync_read+0x600/0x600
[ 9839.741888][T97227]  ? vfs_write+0x36c/0x5b0
[ 9839.746181][T97227]  ? rcu_read_lock_any_held+0xcd/0xf0
[ 9839.751433][T97227]  vfs_write+0x3e9/0x5b0

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Matthew Wilcox]

On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
> Today's linux-next starts to trigger this wondering if anyone has any clue.

I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
to try to get a clue about it.  Good to know it's not the THP patches
since they aren't in linux-next.

I don't understand how it can happen.  We have the page locked, and then we do:

                        if (PageWriteback(page)) {
                                if (wbc->sync_mode != WB_SYNC_NONE)
                                        wait_on_page_writeback(page);
                                else
                                        goto continue_unlock;
                        }

                        VM_BUG_ON_PAGE(PageWriteback(page), page);

Nobody should be able to put this page under writeback while we have it
locked ... right?  The page can be redirtied by the code that's supposed
to be writing it back, but I don't see how anyone can make PageWriteback
true while we're holding the page lock.

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[William Kucharski]

> On Oct 21, 2020, at 6:49 PM, Matthew Wilcox <willy@infradead.org> wrote:
> 
> On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
>> Today's linux-next starts to trigger this wondering if anyone has any clue.
> 
> I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
> to try to get a clue about it.  Good to know it's not the THP patches
> since they aren't in linux-next.
> 
> I don't understand how it can happen.  We have the page locked, and then we do:
> 
>                        if (PageWriteback(page)) {
>                                if (wbc->sync_mode != WB_SYNC_NONE)
>                                        wait_on_page_writeback(page);
>                                else
>                                        goto continue_unlock;
>                        }
> 
>                        VM_BUG_ON_PAGE(PageWriteback(page), page);
> 
> Nobody should be able to put this page under writeback while we have it
> locked ... right?  The page can be redirtied by the code that's supposed
> to be writing it back, but I don't see how anyone can make PageWriteback
> true while we're holding the page lock.

Looking at __test_set_page_writeback(), I see that it (and most other
callers to lock_page_memcg()) do the following:

  lock_page_memcg(page)

  /* do other stuff */

  ret = TestSetPageWriteback(page);

  /* do more stuff */

  unlock_page_memcg(page)

yet lock_page_memcg() does have a few cases where it can (silently)
return NULL to indicate an error.

Only test_clear_page_writeback() actually saves off the return value
(but it too never bothers to check whether it is NULL or not.)

Could it be one of those error conditions is occurring leading to no
lock actually being taken?

The conditions would be extremely rare, but it feels wrong not to check
somewhere:

	  struct page *head = compound_head(page); /* rmap on tail pages */

[ ... ]

          if (mem_cgroup_disabled())
              return NULL;
  again:
          memcg = head->mem_cgroup;
          if (unlikely(!memcg))
                  return NULL;

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Qian Cai]

On Thu, 2020-10-22 at 01:49 +0100, Matthew Wilcox wrote:
> On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
> > Today's linux-next starts to trigger this wondering if anyone has any clue.
> 
> I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
> to try to get a clue about it.  Good to know it's not the THP patches
> since they aren't in linux-next.
> 
> I don't understand how it can happen.  We have the page locked, and then we
> do:
> 
>                         if (PageWriteback(page)) {
>                                 if (wbc->sync_mode != WB_SYNC_NONE)
>                                         wait_on_page_writeback(page);
>                                 else
>                                         goto continue_unlock;
>                         }
> 
>                         VM_BUG_ON_PAGE(PageWriteback(page), page);
> 
> Nobody should be able to put this page under writeback while we have it
> locked ... right?  The page can be redirtied by the code that's supposed
> to be writing it back, but I don't see how anyone can make PageWriteback
> true while we're holding the page lock.

It happened again on today's linux-next:

[ 7613.579890][T55770] page:00000000a4b35e02 refcount:3 mapcount:0 mapping:00000000457ceb87 index:0x3e pfn:0x1cef4e
[ 7613.590594][T55770] aops:xfs_address_space_operations ino:805d85a dentry name:"doio.f1.55762"
[ 7613.599192][T55770] flags: 0xbfffc0000000bf(locked|waiters|referenced|uptodate|dirty|lru|active)
[ 7613.608596][T55770] raw: 00bfffc0000000bf ffffea0005027d48 ffff88810eaec030 ffff888231f3a6a8
[ 7613.617101][T55770] raw: 000000000000003e 0000000000000000 00000003ffffffff ffff888143724000
[ 7613.625590][T55770] page dumped because: VM_BUG_ON_PAGE(PageWriteback(page))
[ 7613.632695][T55770] page->mem_cgroup:ffff888143724000

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Matthew Wilcox]

On Thu, Oct 22, 2020 at 07:23:33AM -0600, William Kucharski wrote:
> 
> 
> > On Oct 21, 2020, at 6:49 PM, Matthew Wilcox <willy@infradead.org> wrote:
> > 
> > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
> >> Today's linux-next starts to trigger this wondering if anyone has any clue.
> > 
> > I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
> > to try to get a clue about it.  Good to know it's not the THP patches
> > since they aren't in linux-next.
> > 
> > I don't understand how it can happen.  We have the page locked, and then we do:
> > 
> >                        if (PageWriteback(page)) {
> >                                if (wbc->sync_mode != WB_SYNC_NONE)
> >                                        wait_on_page_writeback(page);
> >                                else
> >                                        goto continue_unlock;
> >                        }
> > 
> >                        VM_BUG_ON_PAGE(PageWriteback(page), page);
> > 
> > Nobody should be able to put this page under writeback while we have it
> > locked ... right?  The page can be redirtied by the code that's supposed
> > to be writing it back, but I don't see how anyone can make PageWriteback
> > true while we're holding the page lock.
> 
> Looking at __test_set_page_writeback(), I see that it (and most other
> callers to lock_page_memcg()) do the following:

lock_page_memcg() is, unfortunately, completely unrelated to lock_page().
I believe all callers of __test_set_page_writeback() have the page lock
held already, but I'm going to put in an assert to that effect.

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Matthew Wilcox]

On Thu, Oct 22, 2020 at 11:35:26AM -0400, Qian Cai wrote:
> On Thu, 2020-10-22 at 01:49 +0100, Matthew Wilcox wrote:
> > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
> > > Today's linux-next starts to trigger this wondering if anyone has any clue.
> > 
> > I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
> > to try to get a clue about it.  Good to know it's not the THP patches
> > since they aren't in linux-next.
> > 
> > I don't understand how it can happen.  We have the page locked, and then we
> > do:
> > 
> >                         if (PageWriteback(page)) {
> >                                 if (wbc->sync_mode != WB_SYNC_NONE)
> >                                         wait_on_page_writeback(page);
> >                                 else
> >                                         goto continue_unlock;
> >                         }
> > 
> >                         VM_BUG_ON_PAGE(PageWriteback(page), page);
> > 
> > Nobody should be able to put this page under writeback while we have it
> > locked ... right?  The page can be redirtied by the code that's supposed
> > to be writing it back, but I don't see how anyone can make PageWriteback
> > true while we're holding the page lock.
> 
> It happened again on today's linux-next:
> 
> [ 7613.579890][T55770] page:00000000a4b35e02 refcount:3 mapcount:0 mapping:00000000457ceb87 index:0x3e pfn:0x1cef4e
> [ 7613.590594][T55770] aops:xfs_address_space_operations ino:805d85a dentry name:"doio.f1.55762"
> [ 7613.599192][T55770] flags: 0xbfffc0000000bf(locked|waiters|referenced|uptodate|dirty|lru|active)
> [ 7613.608596][T55770] raw: 00bfffc0000000bf ffffea0005027d48 ffff88810eaec030 ffff888231f3a6a8
> [ 7613.617101][T55770] raw: 000000000000003e 0000000000000000 00000003ffffffff ffff888143724000
> [ 7613.625590][T55770] page dumped because: VM_BUG_ON_PAGE(PageWriteback(page))
> [ 7613.632695][T55770] page->mem_cgroup:ffff888143724000

Seems like it reproduces for you pretty quickly.  I have no luck ;-(

Can you add this?

+++ b/mm/page-writeback.c
@@ -2774,6 +2774,7 @@ int __test_set_page_writeback(struct page *page, bool keep_write)
        struct address_space *mapping = page_mapping(page);
        int ret, access_ret;
 
+       VM_BUG_ON_PAGE(!PageLocked(page), page);
        lock_page_memcg(page);
        if (mapping && mapping_use_writeback_tags(mapping)) {
                XA_STATE(xas, &mapping->i_pages, page_index(page));

This is the only place (afaict) that sets PageWriteback, so that will
tell us whether someone is setting Writeback without holding the lock,
or whether we're suffering from a spurious wakeup.

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Jan Kara]

On Thu 22-10-20 01:49:06, Matthew Wilcox wrote:
> On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
> > Today's linux-next starts to trigger this wondering if anyone has any clue.
> 
> I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
> to try to get a clue about it.  Good to know it's not the THP patches
> since they aren't in linux-next.
> 
> I don't understand how it can happen.  We have the page locked, and then we do:
> 
>                         if (PageWriteback(page)) {
>                                 if (wbc->sync_mode != WB_SYNC_NONE)
>                                         wait_on_page_writeback(page);
>                                 else
>                                         goto continue_unlock;
>                         }
> 
>                         VM_BUG_ON_PAGE(PageWriteback(page), page);
> 
> Nobody should be able to put this page under writeback while we have it
> locked ... right?  The page can be redirtied by the code that's supposed
> to be writing it back, but I don't see how anyone can make PageWriteback
> true while we're holding the page lock.

FWIW here's very similar report for ext4 [1] and I strongly suspect this
started happening after Linus' rewrite of the page bit waiting logic. Linus
thinks it's preexisting bug which just got exposed by his changes (which is
possible). I've been searching a culprit for some time but so far I failed.
It's good to know it isn't ext4 specific so we should be searching in the
generic code ;). So far I was concentrating more on ext4 bits...

								Honza

[1] https://lore.kernel.org/lkml/000000000000d3a33205add2f7b2@google.com/

-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Matthew Wilcox]

On Mon, Oct 26, 2020 at 10:49:48AM +0100, Jan Kara wrote:
> On Thu 22-10-20 01:49:06, Matthew Wilcox wrote:
> > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
> > > Today's linux-next starts to trigger this wondering if anyone has any clue.
> > 
> > I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
> > to try to get a clue about it.  Good to know it's not the THP patches
> > since they aren't in linux-next.
> > 
> > I don't understand how it can happen.  We have the page locked, and then we do:
> > 
> >                         if (PageWriteback(page)) {
> >                                 if (wbc->sync_mode != WB_SYNC_NONE)
> >                                         wait_on_page_writeback(page);
> >                                 else
> >                                         goto continue_unlock;
> >                         }
> > 
> >                         VM_BUG_ON_PAGE(PageWriteback(page), page);
> > 
> > Nobody should be able to put this page under writeback while we have it
> > locked ... right?  The page can be redirtied by the code that's supposed
> > to be writing it back, but I don't see how anyone can make PageWriteback
> > true while we're holding the page lock.
> 
> FWIW here's very similar report for ext4 [1] and I strongly suspect this
> started happening after Linus' rewrite of the page bit waiting logic. Linus
> thinks it's preexisting bug which just got exposed by his changes (which is
> possible). I've been searching a culprit for some time but so far I failed.
> It's good to know it isn't ext4 specific so we should be searching in the
> generic code ;). So far I was concentrating more on ext4 bits...
> 
> 								Honza
> 
> [1] https://lore.kernel.org/lkml/000000000000d3a33205add2f7b2@google.com/

Oh good, I was wondering if it was an XFS bug ;-)

I hope Qian gets it to reproduce soon with the assert because that will
tell us whether it's a spurious wakeup or someone calling SetPageWriteback
without holding the page lock.

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Jens Axboe]

On 10/26/20 7:13 AM, Matthew Wilcox wrote:
> On Mon, Oct 26, 2020 at 10:49:48AM +0100, Jan Kara wrote:
>> On Thu 22-10-20 01:49:06, Matthew Wilcox wrote:
>>> On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
>>>> Today's linux-next starts to trigger this wondering if anyone has any clue.
>>>
>>> I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
>>> to try to get a clue about it.  Good to know it's not the THP patches
>>> since they aren't in linux-next.
>>>
>>> I don't understand how it can happen.  We have the page locked, and then we do:
>>>
>>>                         if (PageWriteback(page)) {
>>>                                 if (wbc->sync_mode != WB_SYNC_NONE)
>>>                                         wait_on_page_writeback(page);
>>>                                 else
>>>                                         goto continue_unlock;
>>>                         }
>>>
>>>                         VM_BUG_ON_PAGE(PageWriteback(page), page);
>>>
>>> Nobody should be able to put this page under writeback while we have it
>>> locked ... right?  The page can be redirtied by the code that's supposed
>>> to be writing it back, but I don't see how anyone can make PageWriteback
>>> true while we're holding the page lock.
>>
>> FWIW here's very similar report for ext4 [1] and I strongly suspect this
>> started happening after Linus' rewrite of the page bit waiting logic. Linus
>> thinks it's preexisting bug which just got exposed by his changes (which is
>> possible). I've been searching a culprit for some time but so far I failed.
>> It's good to know it isn't ext4 specific so we should be searching in the
>> generic code ;). So far I was concentrating more on ext4 bits...
>>
>> 								Honza
>>
>> [1] https://lore.kernel.org/lkml/000000000000d3a33205add2f7b2@google.com/
> 
> Oh good, I was wondering if it was an XFS bug ;-)
> 
> I hope Qian gets it to reproduce soon with the assert because that will
> tell us whether it's a spurious wakeup or someone calling SetPageWriteback
> without holding the page lock.

I've tried to reproduce this as well, to no avail. Qian, could you perhaps
detail the setup? What kind of storage, kernel config, compiler, etc.

-- 
Jens Axboe

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Qian Cai]

On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote:
> I've tried to reproduce this as well, to no avail. Qian, could you perhaps
> detail the setup? What kind of storage, kernel config, compiler, etc.
> 

So far I have only been able to reproduce on this Intel platform:

HPE DL560 gen10
Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
131072 MB memory, 1000 GB disk space (smartpqi nvme)

It was running some CPU and memory hotplug, KVM and then LTP workloads
(syscalls, mm, and fs). Finally, it was always this LTP test case to trigger it:

# export LTPROOT; rwtest -N iogen01 -i 120s -s read,write -Da -Dv -n 2 500b:$TMPDIR/doio.f1.$$ 1000b:$TMPDIR/doio.f2.$$
https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/fs/doio/rwtest

gcc-8.3.1-5.1.el8.x86_64
.config: https://git.code.tencent.com/cail/linux-mm/blob/master/x86.config


== storage information == 
[   46.131150] smartpqi 0000:23:00.0: Online Firmware Activation enabled
[   46.138495] smartpqi 0000:23:00.0: Serial Management Protocol enabled
[   46.145701] smartpqi 0000:23:00.0: New Soft Reset Handshake enabled
[   46.152705] smartpqi 0000:23:00.0: RAID IU Timeout enabled
[   46.158934] smartpqi 0000:23:00.0: TMF IU Timeout enabled
[   46.168740] scsi host0: smartpqi
[   47.750425] nvme nvme1: 31/0/0 default/read/poll queues
[   47.826211] scsi 0:0:0:0: Direct-Access     ATA      MM1000GEFQV      HPG8 PQ: 0 ANSI: 6
[   47.841752] smartpqi 0000:23:00.0: added 0:0:0:0 0000000000000000 Direct-Access     ATA      MM1000GEFQV      AIO+ qd=32    
[   47.941249]  nvme1n1: p1
[   47.943078] scsi 0:0:1:0: Enclosure         HPE      Smart Adapter    2.62 PQ: 0 ANSI: 5
[   47.958506] smartpqi 0000:23:00.0: added 0:0:1:0 51402ec001f36448 Enclosure         HPE      Smart Adapter    AIO-
[   47.962844] nvme nvme0: 31/0/0 default/read/poll queues
[   48.015511] scsi 0:2:0:0: RAID              HPE      P408i-a SR Gen10 2.62 PQ: 0 ANSI: 5
[   48.029736] smartpqi 0000:23:00.0: added 0:2:0:0 0000000000000000 RAID              HPE      P408i-a SR Gen10 
[   48.042711] smartpqi 0000:4e:00.0: Microsemi Smart Family Controller found
[   48.149820]  nvme0n1: p1
[   48.956194] smartpqi 0000:4e:00.0: Online Firmware Activation enabled
[   48.963399] smartpqi 0000:4e:00.0: Serial Management Protocol enabled
[   48.970625] smartpqi 0000:4e:00.0: New Soft Reset Handshake enabled
[   48.977645] smartpqi 0000:4e:00.0: RAID IU Timeout enabled
[   48.983873] smartpqi 0000:4e:00.0: TMF IU Timeout enabled
[   48.994687] scsi host1: smartpqi
[   50.612955] scsi 1:0:0:0: Enclosure         HPE      Smart Adapter    2.62 PQ: 0 ANSI: 5
[   50.628219] smartpqi 0000:4e:00.0: added 1:0:0:0 51402ec01040ffc8 Enclosure         HPE      Smart Adapter    AIO-
[   50.681859] scsi 1:2:0:0: RAID              HPE      E208i-p SR Gen10 2.62 PQ: 0 ANSI: 5
[   50.695843] smartpqi 0000:4e:00.0: added 1:2:0:0 0000000000000000 RAID              HPE      E208i-p SR Gen10 
[   50.856683] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[   50.865195] sd 0:0:0:0: [sda] 4096-byte physical blocks
[   50.871354] sd 0:0:0:0: [sda] Write Protect is off
[   50.876956] sd 0:0:0:0: [sda] Mode Sense: 46 00 10 08
[   50.877299] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[   50.898824]  sda: sda1 sda2 sda3
[   50.943835] sd 0:0:0:0: [sda] Attached SCSI disk

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Qian Cai]

On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote:
> I've tried to reproduce this as well, to no avail. Qian, could you perhaps
> detail the setup? What kind of storage, kernel config, compiler, etc.

This should work:

https://gitlab.com/cailca/linux-mm/-/blob/master/x86.config

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Qian Cai]

On Thu, 2020-10-22 at 18:12 +0100, Matthew Wilcox wrote:
> On Thu, Oct 22, 2020 at 11:35:26AM -0400, Qian Cai wrote:
> > On Thu, 2020-10-22 at 01:49 +0100, Matthew Wilcox wrote:
> > > On Wed, Oct 21, 2020 at 08:30:18PM -0400, Qian Cai wrote:
> > > > Today's linux-next starts to trigger this wondering if anyone has any
> > > > clue.
> > > 
> > > I've seen that occasionally too.  I changed that BUG_ON to VM_BUG_ON_PAGE
> > > to try to get a clue about it.  Good to know it's not the THP patches
> > > since they aren't in linux-next.
> > > 
> > > I don't understand how it can happen.  We have the page locked, and then
> > > we
> > > do:
> > > 
> > >                         if (PageWriteback(page)) {
> > >                                 if (wbc->sync_mode != WB_SYNC_NONE)
> > >                                         wait_on_page_writeback(page);
> > >                                 else
> > >                                         goto continue_unlock;
> > >                         }
> > > 
> > >                         VM_BUG_ON_PAGE(PageWriteback(page), page);
> > > 
> > > Nobody should be able to put this page under writeback while we have it
> > > locked ... right?  The page can be redirtied by the code that's supposed
> > > to be writing it back, but I don't see how anyone can make PageWriteback
> > > true while we're holding the page lock.
> > 
> > It happened again on today's linux-next:
> > 
> > [ 7613.579890][T55770] page:00000000a4b35e02 refcount:3 mapcount:0
> > mapping:00000000457ceb87 index:0x3e pfn:0x1cef4e
> > [ 7613.590594][T55770] aops:xfs_address_space_operations ino:805d85a dentry
> > name:"doio.f1.55762"
> > [ 7613.599192][T55770] flags:
> > 0xbfffc0000000bf(locked|waiters|referenced|uptodate|dirty|lru|active)
> > [ 7613.608596][T55770] raw: 00bfffc0000000bf ffffea0005027d48
> > ffff88810eaec030 ffff888231f3a6a8
> > [ 7613.617101][T55770] raw: 000000000000003e 0000000000000000
> > 00000003ffffffff ffff888143724000
> > [ 7613.625590][T55770] page dumped because:
> > VM_BUG_ON_PAGE(PageWriteback(page))
> > [ 7613.632695][T55770] page->mem_cgroup:ffff888143724000
> 
> Seems like it reproduces for you pretty quickly.  I have no luck ;-(
> 
> Can you add this?

It turns out I had no luck for the last a few days. I'll keep running and report
back if it triggers again.

> 
> +++ b/mm/page-writeback.c
> @@ -2774,6 +2774,7 @@ int __test_set_page_writeback(struct page *page, bool
> keep_write)
>         struct address_space *mapping = page_mapping(page);
>         int ret, access_ret;
>  
> +       VM_BUG_ON_PAGE(!PageLocked(page), page);
>         lock_page_memcg(page);
>         if (mapping && mapping_use_writeback_tags(mapping)) {
>                 XA_STATE(xas, &mapping->i_pages, page_index(page));
> 
> This is the only place (afaict) that sets PageWriteback, so that will
> tell us whether someone is setting Writeback without holding the lock,
> or whether we're suffering from a spurious wakeup.
> 

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Jan Kara]

On Mon 26-10-20 10:26:26, Qian Cai wrote:
> On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote:
> > I've tried to reproduce this as well, to no avail. Qian, could you perhaps
> > detail the setup? What kind of storage, kernel config, compiler, etc.
> > 
> 
> So far I have only been able to reproduce on this Intel platform:
> 
> HPE DL560 gen10
> Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
> 131072 MB memory, 1000 GB disk space (smartpqi nvme)

Did you try running with the debug patch Matthew sent? Any results?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: kernel BUG at mm/page-writeback.c:2241 [ BUG_ON(PageWriteback(page); ]

[Qian Cai]

On Wed, 2020-11-04 at 16:16 +0100, Jan Kara wrote:
> On Mon 26-10-20 10:26:26, Qian Cai wrote:
> > On Mon, 2020-10-26 at 07:55 -0600, Jens Axboe wrote:
> > > I've tried to reproduce this as well, to no avail. Qian, could you perhaps
> > > detail the setup? What kind of storage, kernel config, compiler, etc.
> > > 
> > 
> > So far I have only been able to reproduce on this Intel platform:
> > 
> > HPE DL560 gen10
> > Intel(R) Xeon(R) Gold 6154 CPU @ 3.00GHz
> > 131072 MB memory, 1000 GB disk space (smartpqi nvme)
> 
> Did you try running with the debug patch Matthew sent? Any results?
Running every day, but no luck so far.