slab corruption in generic/388

slab corruption in generic/388

[Christoph Hellwig]

Recently I've a few of these warnings in generic/388 with a 4k
filesystem on x86_64, v5 file system with reflink and rmap
enabled but otherwise defaults.  If someones got and idea where
this might be from I'm curious, otherwise I'll add it to my TODO list.

[ 5062.894006] XFS (sdb): xfs_do_force_shutdown(0x1) called from line 236 of file fs/xfs/libxfs/xfs_defer.c.  Return address = 0xffffffff8156cb46
[ 5062.908923] XFS (sdb): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
[ 5062.986682] XFS (sdb): Unmounting Filesystem
[ 5063.008740] XFS (sdb): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
[ 5063.010063] XFS (sdb): EXPERIMENTAL reflink feature enabled. Use at your own risk!
[ 5063.011543] XFS (sdb): Mounting V5 Filesystem
[ 5063.299537] XFS (sdb): Starting recovery (logdev: internal)
[ 5063.305221] XFS (sdb): Ending recovery (logdev: internal)
[ 5063.355721] Slab corruption (Tainted: G        W      ): xfs_rui_item start=ffff88003908f708, len=672
[ 5063.357058] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b
[ 5063.357842] Last user: [<ffffffff815dfa2c>](xfs_rui_item_free+0x1c/0x30)
[ 5063.358712] 080: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk
[ 5063.359813] Single bit error detected. Probably bad RAM.
[ 5063.360512] Run memtest86+ or a similar memory test tool.
[ 5063.361216] Prev obj: start=ffff88003908f450, len=672
[ 5063.361876] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b
[ 5063.362836] Last user: [<ffffffff815dfa2c>](xfs_rui_item_free+0x1c/0x30)
[ 5063.363704] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[ 5063.364708] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[ 5063.365729] Next obj: start=ffff88003908f9c0, len=672
[ 5063.366397] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b
[ 5063.367101] Last user: [<ffffffff815dfa2c>](xfs_rui_item_free+0x1c/0x30)
[ 5063.367846] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[ 5063.368637] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[ 5065.460847] XFS (sdb): Unmounting Filesystem

Re: slab corruption in generic/388

[Darrick J. Wong]

On Wed, Sep 20, 2017 at 07:57:40AM -0700, Christoph Hellwig wrote:
> Recently I've a few of these warnings in generic/388 with a 4k
> filesystem on x86_64, v5 file system with reflink and rmap
> enabled but otherwise defaults.  If someones got and idea where
> this might be from I'm curious, otherwise I'll add it to my TODO list.
> 
> [ 5062.894006] XFS (sdb): xfs_do_force_shutdown(0x1) called from line 236 of file fs/xfs/libxfs/xfs_defer.c.  Return address = 0xffffffff8156cb46
> [ 5062.908923] XFS (sdb): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
> [ 5062.986682] XFS (sdb): Unmounting Filesystem
> [ 5063.008740] XFS (sdb): EXPERIMENTAL reverse mapping btree feature enabled. Use at your own risk!
> [ 5063.010063] XFS (sdb): EXPERIMENTAL reflink feature enabled. Use at your own risk!
> [ 5063.011543] XFS (sdb): Mounting V5 Filesystem
> [ 5063.299537] XFS (sdb): Starting recovery (logdev: internal)
> [ 5063.305221] XFS (sdb): Ending recovery (logdev: internal)
> [ 5063.355721] Slab corruption (Tainted: G        W      ): xfs_rui_item start=ffff88003908f708, len=672
> [ 5063.357058] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b
> [ 5063.357842] Last user: [<ffffffff815dfa2c>](xfs_rui_item_free+0x1c/0x30)
> [ 5063.358712] 080: 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  jkkkkkkkkkkkkkkk

Assuming this trace is from 4.14, something decremented rui_refcount
after the xfs_rui_log_item got freed.

--D

> [ 5063.359813] Single bit error detected. Probably bad RAM.
> [ 5063.360512] Run memtest86+ or a similar memory test tool.
> [ 5063.361216] Prev obj: start=ffff88003908f450, len=672
> [ 5063.361876] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b
> [ 5063.362836] Last user: [<ffffffff815dfa2c>](xfs_rui_item_free+0x1c/0x30)
> [ 5063.363704] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> [ 5063.364708] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> [ 5063.365729] Next obj: start=ffff88003908f9c0, len=672
> [ 5063.366397] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b
> [ 5063.367101] Last user: [<ffffffff815dfa2c>](xfs_rui_item_free+0x1c/0x30)
> [ 5063.367846] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> [ 5063.368637] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
> [ 5065.460847] XFS (sdb): Unmounting Filesystem
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: slab corruption in generic/388

[Chandan Rajendra]

On Wednesday, September 20, 2017 8:27:40 PM IST Christoph Hellwig wrote:
> Recently I've a few of these warnings in generic/388 with a 4k
> filesystem on x86_64, v5 file system with reflink and rmap
> enabled but otherwise defaults.  If someones got and idea where
> this might be from I'm curious, otherwise I'll add it to my TODO list.

I am able to recreate it on my ppc64 box as well. I will root cause the issue
and fix it.

=============================================================================
BUG xfs_rui_item (Tainted: G    B          ): Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xc000000105d15f0b-0xc000000105d15f0b. First byte 0x6a instead of 0x6b
INFO: Allocated in .kmem_zone_alloc+0xcc/0x180 age=64 cpu=0 pid=2189
        .__slab_alloc+0x24/0x50
        .kmem_cache_alloc+0x124/0x350
        .kmem_zone_alloc+0xcc/0x180
        .xfs_rui_init+0x3c/0xe0
        .xfs_rmap_update_create_intent+0x34/0x120
        .xfs_defer_intake_work+0x64/0x1d0
        .xfs_defer_finish+0xa4/0x5b0
        .xfs_iomap_write_allocate+0x1cc/0x420
        .xfs_map_blocks+0x2c4/0x490
        .xfs_do_writepage+0x408/0x830
        .write_cache_pages+0x270/0x5f0
        .xfs_vm_writepages+0xe8/0x140
        .do_writepages+0x44/0xd0
        .__writeback_single_inode+0x5c/0x550
        .writeback_sb_inodes+0x2b8/0x580
        .__writeback_inodes_wb+0xd4/0x140
INFO: Freed in .xfs_rui_item_free+0x28/0x60 age=64 cpu=0 pid=2189
        .kmem_cache_free+0x118/0x2a0
        .xfs_rui_item_free+0x28/0x60
        .xfs_trans_free_items+0xd4/0x160
        .__xfs_trans_commit+0xc8/0x350
        .xfs_trans_roll+0x44/0x80
        .xfs_defer_trans_roll+0x6c/0x240
        .xfs_defer_finish+0xb0/0x5b0
        .xfs_iomap_write_allocate+0x1cc/0x420
        .xfs_map_blocks+0x2c4/0x490
        .xfs_do_writepage+0x408/0x830
        .write_cache_pages+0x270/0x5f0
        .xfs_vm_writepages+0xe8/0x140
        .do_writepages+0x44/0xd0
        .__writeback_single_inode+0x5c/0x550
        .writeback_sb_inodes+0x2b8/0x580
        .__writeback_inodes_wb+0xd4/0x140
INFO: Slab 0xf000000000417440 objects=65 used=65 fp=0x          (null) flags=0x3ffff800000100
INFO: Object 0xc000000105d15e88 @offset=24200 fp=0xc000000105d19998

Redzone c000000105d15e80: bb bb bb bb bb bb bb bb                          ........
Object c000000105d15e88: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object c000000105d15e98: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object c000000105d15ea8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object c000000105d15eb8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk

-- 
chandan