Re: [PATCH 00/17] fs, btrfs refcount conversions

[Qu Wenruo <quwenruo@cn.fujitsu.com>]

[-- Attachment #1: Type: text/plain, Size: 6569 bytes --]



At 03/07/2017 03:41 PM, Reshetova, Elena wrote:
>> At 03/06/2017 05:43 PM, Reshetova, Elena wrote:
>>>
>>>> At 03/03/2017 04:55 PM, Elena Reshetova wrote:
>>>>> Now when new refcount_t type and API are finally merged
>>>>> (see include/linux/refcount.h), the following
>>>>> patches convert various refcounters in the btrfs filesystem from atomic_t
>>>>> to refcount_t. By doing this we prevent intentional or accidental
>>>>> underflows or overflows that can led to use-after-free vulnerabilities.
>>>>>
>>>>> The below patches are fully independent and can be cherry-picked separately.
>>>>> Since we convert all kernel subsystems in the same fashion, resulting
>>>>> in about 300 patches, we have to group them for sending at least in some
>>>>> fashion to be manageable. Please excuse the long cc list.
>>>>>
>>>>> These patches have been tested with xfstests by running btrfs-related tests.
>>>>> btrfs debug was enabled, warns on refcount errors, too. No output related to
>>>>> refcount errors produced. However, the following errors were during the run:
>>>>>  * tests btrfs/078, btrfs/114, btrfs/115, no errors anywhere in dmesg, but
>>>>>  process hangs. They all seem to be around qgroup, sometimes error visible
>>>>>  such as qgroup scan failed -4 before it blocks, but not always.
>>>>
>>>> How reproducible of the hang?
>>>
>>> Always in  my environment, but I would not much go into investigating why it
>> happens, if it works for you.
>>> My test environment is far from ideal: I am testing in VM with rather old
>> userspace and couple of additional changes in,
>>> so there are many things that can potentially go wrong. Anyway the strace for
>> 078 is in the attachment.
>>
>> Thanks for the strace.
>>
>> However no "-f" is passed to strace, so it doesn't contain much useful info.
>>
>>>
>>> If the patches pass all tests on your side, could you please take them in and
>> propagate further?
>>> I will continue with other kernel subsystems.
>>
>> The patchset itself looks like a common cleanup, while I did encounter
>> several cases (almost all scrub tests) causing kernel warning due to
>> underflow.
>
> Oh, could you please send me the warning outputs? I can hopefully analyze and fix them.

Attached. Which is the generated by running btrfs/070 test case.
And I canceled the case almost instantly, so output is not much, but 
still contains enough info.

Both refcount_inc() and refcount_sub_and_test() are causing warning.

So now I'm not sure which is the cause, btrfs or bad use of refcount?

Thanks,
Qu

>
> Best Regards,
> Elena.
>
>>
>> So I'm afraid the patchset will not be merged until we fix all the
>> underflows.
>>
>> But thanks for the patchset, it helps us to expose a lot of problem.
>>
>> Thanks,
>> Qu
>>
>>>
>>> Best Regards,
>>> Elena.
>>>
>>>
>>>>
>>>> I also see the -EINTR output, but that seems to be designed for
>>>> btrfs/11[45].
>>>>
>>>> btrfs/078 is unrelated to qgroup, and all these three test pass in my
>>>> test environment, which is v4.11-rc1 with your patches applied.
>>>>
>>>> I ran these 3 tests in a row with default and space_cache=v2 mount
>>>> options, and 5 times for each mount option, no hang at all.
>>>>
>>>> It would help much if more info can be provided, from blocked process
>>>> backtrace to test mount option to base commit.
>>>>
>>>> Thanks,
>>>> Qu
>>>>
>>>>>  * test btrfs/104 dmesg has additional error output:
>>>>>  BTRFS warning (device vdc): qgroup 258 reserved space underflow, have: 0,
>>>>>  to free: 4096
>>>>>  I tried looking at the code on what causes the failure, but could not figure
>>>>>  it out. It doesn't seem to be related to any refcount changes at least IMO.
>>>>>
>>>>> The above test failures are hard for me to understand and interpreted, but
>>>>> they don't seem to relate to refcount conversions.
>>>>>
>>>>> Elena Reshetova (17):
>>>>>   fs, btrfs: convert btrfs_bio.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert btrfs_transaction.use_count from atomic_t to
>>>>>     refcount_t
>>>>>   fs, btrfs: convert extent_map.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert btrfs_ordered_extent.refs from atomic_t to
>>>>>     refcount_t
>>>>>   fs, btrfs: convert btrfs_caching_control.count from atomic_t to
>>>>>     refcount_t
>>>>>   fs, btrfs: convert btrfs_delayed_ref_node.refs from atomic_t to
>>>>>     refcount_t
>>>>>   fs, btrfs: convert btrfs_delayed_node.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert btrfs_delayed_item.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert btrfs_root.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert extent_state.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert compressed_bio.pending_bios from atomic_t to
>>>>>     refcount_t
>>>>>   fs, btrfs: convert scrub_recover.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert scrub_page.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert scrub_block.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert scrub_parity.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert scrub_ctx.refs from atomic_t to refcount_t
>>>>>   fs, btrfs: convert btrfs_raid_bio.refs from atomic_t to refcount_t
>>>>>
>>>>>  fs/btrfs/backref.c           |  2 +-
>>>>>  fs/btrfs/compression.c       | 18 ++++++++---------
>>>>>  fs/btrfs/ctree.h             |  5 +++--
>>>>>  fs/btrfs/delayed-inode.c     | 46 ++++++++++++++++++++++----------------------
>>>>>  fs/btrfs/delayed-inode.h     |  5 +++--
>>>>>  fs/btrfs/delayed-ref.c       |  8 ++++----
>>>>>  fs/btrfs/delayed-ref.h       |  8 +++++---
>>>>>  fs/btrfs/disk-io.c           |  6 +++---
>>>>>  fs/btrfs/disk-io.h           |  4 ++--
>>>>>  fs/btrfs/extent-tree.c       | 20 +++++++++----------
>>>>>  fs/btrfs/extent_io.c         | 18 ++++++++---------
>>>>>  fs/btrfs/extent_io.h         |  3 ++-
>>>>>  fs/btrfs/extent_map.c        | 10 +++++-----
>>>>>  fs/btrfs/extent_map.h        |  3 ++-
>>>>>  fs/btrfs/ordered-data.c      | 20 +++++++++----------
>>>>>  fs/btrfs/ordered-data.h      |  2 +-
>>>>>  fs/btrfs/raid56.c            | 19 +++++++++---------
>>>>>  fs/btrfs/scrub.c             | 42 ++++++++++++++++++++--------------------
>>>>>  fs/btrfs/transaction.c       | 20 +++++++++----------
>>>>>  fs/btrfs/transaction.h       |  3 ++-
>>>>>  fs/btrfs/tree-log.c          |  2 +-
>>>>>  fs/btrfs/volumes.c           | 10 +++++-----
>>>>>  fs/btrfs/volumes.h           |  2 +-
>>>>>  include/trace/events/btrfs.h |  4 ++--
>>>>>  24 files changed, 143 insertions(+), 137 deletions(-)
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
>



[-- Attachment #2: output --]
[-- Type: text/plain, Size: 27170 bytes --]

[17805.205289] run fstests btrfs/070 at 2017-03-07 14:02:47
[17805.765749] BTRFS: device fsid ef047ab3-6c98-4d0c-b76c-f10631b58a51 devid 1 transid 2 /dev/vdb6
[17805.776084] BTRFS: device fsid ef047ab3-6c98-4d0c-b76c-f10631b58a51 devid 2 transid 3 /dev/vdb7
[17805.779794] BTRFS: device fsid ef047ab3-6c98-4d0c-b76c-f10631b58a51 devid 3 transid 3 /dev/vdb8
[17805.783487] BTRFS: device fsid ef047ab3-6c98-4d0c-b76c-f10631b58a51 devid 4 transid 3 /dev/vdb9
[17806.124656] BTRFS info (device vdb9): disk space caching is enabled
[17806.125164] BTRFS info (device vdb9): has skinny extents
[17806.125528] BTRFS info (device vdb9): flagging fs with big metadata feature
[17806.129091] BTRFS info (device vdb9): creating UUID tree
[17807.145227] BTRFS info (device vdb9): dev_replace from /dev/vdb7 (devid 2) to /dev/vdb10 started
[17808.335771] ------------[ cut here ]------------
[17808.336144] WARNING: CPU: 0 PID: 9448 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.336786] refcount_t: increment on 0; use-after-free.
[17808.337143] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.337753] CPU: 0 PID: 9448 Comm: btrfs Tainted: G        W  O    4.11.0-rc1+ #71
[17808.338134] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.338134] Call Trace:
[17808.338134]  dump_stack+0x68/0x93
[17808.338134]  __warn+0xcb/0xf0
[17808.338134]  warn_slowpath_fmt+0x4f/0x60
[17808.338134]  refcount_inc+0x27/0x30
[17808.338134]  scrub_pages+0x118/0x410 [btrfs]
[17808.338134]  scrub_stripe+0x851/0x1170 [btrfs]
[17808.338134]  ? trace_hardirqs_on+0xd/0x10
[17808.338134]  scrub_chunk+0x109/0x150 [btrfs]
[17808.338134]  scrub_enumerate_chunks+0x297/0x660 [btrfs]
[17808.338134]  btrfs_scrub_dev+0x1f6/0x670 [btrfs]
[17808.338134]  btrfs_dev_replace_start+0x2f1/0x4f0 [btrfs]
[17808.338134]  btrfs_dev_replace_by_ioctl+0x3e/0x70 [btrfs]
[17808.338134]  btrfs_ioctl+0x1e40/0x27f0 [btrfs]
[17808.338134]  ? do_sigaction+0x69/0x1b0
[17808.338134]  do_vfs_ioctl+0x94/0x710
[17808.338134]  ? do_sigaction+0x193/0x1b0
[17808.338134]  SyS_ioctl+0x79/0x90
[17808.338134]  entry_SYSCALL_64_fastpath+0x18/0xad
[17808.338134] RIP: 0033:0x7fe702de1787
[17808.338134] RSP: 002b:00007ffd3dbf0758 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[17808.338134] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fe702de1787
[17808.338134] RDX: 00007ffd3dbf0b90 RSI: 00000000ca289435 RDI: 0000000000000003
[17808.338134] RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000
[17808.338134] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000561120154050
[17808.338134] R13: 00007ffd3dbf2e6f R14: 0000000000000000 R15: 0000000000000001
[17808.347969] ---[ end trace 42fc4dedec8f6207 ]---
[17808.349374] ------------[ cut here ]------------
[17808.349716] WARNING: CPU: 0 PID: 9448 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.351339] refcount_t: increment on 0; use-after-free.
[17808.351667] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.352383] CPU: 0 PID: 9448 Comm: btrfs Tainted: G        W  O    4.11.0-rc1+ #71
[17808.353121] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.353371] Call Trace:
[17808.353371]  dump_stack+0x68/0x93
[17808.353371]  __warn+0xcb/0xf0
[17808.353371]  warn_slowpath_fmt+0x4f/0x60
[17808.353371]  refcount_inc+0x27/0x30
[17808.355021]  scrub_pages+0x118/0x410 [btrfs]
[17808.355021]  scrub_stripe+0x851/0x1170 [btrfs]
[17808.355021]  ? trace_hardirqs_on+0xd/0x10
[17808.355021]  scrub_chunk+0x109/0x150 [btrfs]
[17808.355021]  scrub_enumerate_chunks+0x297/0x660 [btrfs]
[17808.359021]  btrfs_scrub_dev+0x1f6/0x670 [btrfs]
[17808.363022]  btrfs_dev_replace_start+0x2f1/0x4f0 [btrfs]
[17808.363022]  btrfs_dev_replace_by_ioctl+0x3e/0x70 [btrfs]
[17808.363022]  btrfs_ioctl+0x1e40/0x27f0 [btrfs]
[17808.363022]  ? do_sigaction+0x69/0x1b0
[17808.363022]  do_vfs_ioctl+0x94/0x710
[17808.363022]  ? do_sigaction+0x193/0x1b0
[17808.363022]  SyS_ioctl+0x79/0x90
[17808.363022]  entry_SYSCALL_64_fastpath+0x18/0xad
[17808.363022] RIP: 0033:0x7fe702de1787
[17808.363022] RSP: 002b:00007ffd3dbf0758 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[17808.363022] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fe702de1787
[17808.367025] RDX: 00007ffd3dbf0b90 RSI: 00000000ca289435 RDI: 0000000000000003
[17808.367025] RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000
[17808.367025] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000561120154050
[17808.367025] R13: 00007ffd3dbf2e6f R14: 0000000000000000 R15: 0000000000000001
[17808.369634] ---[ end trace 42fc4dedec8f6208 ]---
[17808.375435] ------------[ cut here ]------------
[17808.381855] WARNING: CPU: 0 PID: 9448 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.385910] refcount_t: increment on 0; use-after-free.
[17808.387980] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.388588] CPU: 0 PID: 9448 Comm: btrfs Tainted: G        W  O    4.11.0-rc1+ #71
[17808.389166] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.389578] Call Trace:
[17808.389865]  dump_stack+0x68/0x93
[17808.389865]  __warn+0xcb/0xf0
[17808.389865]  warn_slowpath_fmt+0x4f/0x60
[17808.389865]  refcount_inc+0x27/0x30
[17808.389865]  scrub_pages+0x118/0x410 [btrfs]
[17808.389865]  scrub_stripe+0x851/0x1170 [btrfs]
[17808.389865]  ? trace_hardirqs_on+0xd/0x10
[17808.389865]  scrub_chunk+0x109/0x150 [btrfs]
[17808.389865]  scrub_enumerate_chunks+0x297/0x660 [btrfs]
[17808.389865]  btrfs_scrub_dev+0x1f6/0x670 [btrfs]
[17808.389865]  btrfs_dev_replace_start+0x2f1/0x4f0 [btrfs]
[17808.389865]  btrfs_dev_replace_by_ioctl+0x3e/0x70 [btrfs]
[17808.389865]  btrfs_ioctl+0x1e40/0x27f0 [btrfs]
[17808.389865]  ? do_sigaction+0x69/0x1b0
[17808.389865]  do_vfs_ioctl+0x94/0x710
[17808.389865]  ? do_sigaction+0x193/0x1b0
[17808.389865]  SyS_ioctl+0x79/0x90
[17808.389865]  entry_SYSCALL_64_fastpath+0x18/0xad
[17808.389865] RIP: 0033:0x7fe702de1787
[17808.389865] RSP: 002b:00007ffd3dbf0758 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[17808.389865] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fe702de1787
[17808.389865] RDX: 00007ffd3dbf0b90 RSI: 00000000ca289435 RDI: 0000000000000003
[17808.389865] RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000
[17808.389865] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000561120154050
[17808.389865] R13: 00007ffd3dbf2e6f R14: 0000000000000000 R15: 0000000000000001
[17808.401890] ---[ end trace 42fc4dedec8f6209 ]---
[17808.402282] ------------[ cut here ]------------
[17808.402623] WARNING: CPU: 0 PID: 9448 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.403358] refcount_t: increment on 0; use-after-free.
[17808.403701] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.404494] CPU: 0 PID: 9448 Comm: btrfs Tainted: G        W  O    4.11.0-rc1+ #71
[17808.405069] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.405455] Call Trace:
[17808.405455]  dump_stack+0x68/0x93
[17808.405455]  __warn+0xcb/0xf0
[17808.405455]  warn_slowpath_fmt+0x4f/0x60
[17808.405455]  refcount_inc+0x27/0x30
[17808.405455]  scrub_pages+0x118/0x410 [btrfs]
[17808.405455]  scrub_stripe+0x851/0x1170 [btrfs]
[17808.405455]  ? trace_hardirqs_on+0xd/0x10
[17808.405455]  scrub_chunk+0x109/0x150 [btrfs]
[17808.405455]  scrub_enumerate_chunks+0x297/0x660 [btrfs]
[17808.405455]  btrfs_scrub_dev+0x1f6/0x670 [btrfs]
[17808.405455]  btrfs_dev_replace_start+0x2f1/0x4f0 [btrfs]
[17808.405455]  btrfs_dev_replace_by_ioctl+0x3e/0x70 [btrfs]
[17808.405455]  btrfs_ioctl+0x1e40/0x27f0 [btrfs]
[17808.405455]  ? do_sigaction+0x69/0x1b0
[17808.405455]  do_vfs_ioctl+0x94/0x710
[17808.405455]  ? do_sigaction+0x193/0x1b0
[17808.405455]  SyS_ioctl+0x79/0x90
[17808.405455]  entry_SYSCALL_64_fastpath+0x18/0xad
[17808.405455] RIP: 0033:0x7fe702de1787
[17808.405455] RSP: 002b:00007ffd3dbf0758 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[17808.405455] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fe702de1787
[17808.405455] RDX: 00007ffd3dbf0b90 RSI: 00000000ca289435 RDI: 0000000000000003
[17808.405455] RBP: 0000000000000004 R08: 0000000000000000 R09: 0000000000000000
[17808.405455] R10: 00000000ffffffff R11: 0000000000000246 R12: 0000561120154050
[17808.405455] R13: 00007ffd3dbf2e6f R14: 0000000000000000 R15: 0000000000000001
[17808.415683] ---[ end trace 42fc4dedec8f620a ]---
[17808.416261] ------------[ cut here ]------------
[17808.417080] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.417824] refcount_t: increment on 0; use-after-free.
[17808.418425] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.419194] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.419932] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.420179] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.420179] Call Trace:
[17808.420179]  dump_stack+0x68/0x93
[17808.420179]  __warn+0xcb/0xf0
[17808.420179]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.420179]  warn_slowpath_fmt+0x4f/0x60
[17808.420179]  refcount_inc+0x27/0x30
[17808.420179]  scrub_write_page_to_dev_replace+0x1a1/0x2a0 [btrfs]
[17808.420179]  scrub_write_block_to_dev_replace+0x40/0x60 [btrfs]
[17808.420179]  scrub_bio_end_io_worker+0x36a/0x520 [btrfs]
[17808.420179]  ? debug_object_deactivate+0x56/0x130
[17808.420179]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.420179]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.420179]  process_one_work+0x2af/0x720
[17808.420179]  ? process_one_work+0x22b/0x720
[17808.420179]  worker_thread+0x4b/0x4f0
[17808.420179]  kthread+0x10f/0x150
[17808.420179]  ? process_one_work+0x720/0x720
[17808.420179]  ? kthread_create_on_node+0x40/0x40
[17808.420179]  ret_from_fork+0x2e/0x40
[17808.427447] ---[ end trace 42fc4dedec8f620b ]---
[17808.428357] ------------[ cut here ]------------
[17808.428828] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.429542] refcount_t: increment on 0; use-after-free.
[17808.430117] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.430766] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.431012] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.431012] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.431012] Call Trace:
[17808.431012]  dump_stack+0x68/0x93
[17808.431012]  __warn+0xcb/0xf0
[17808.431012]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.431012]  warn_slowpath_fmt+0x4f/0x60
[17808.431012]  refcount_inc+0x27/0x30
[17808.431012]  scrub_write_page_to_dev_replace+0x1a1/0x2a0 [btrfs]
[17808.431012]  scrub_write_block_to_dev_replace+0x40/0x60 [btrfs]
[17808.436040]  scrub_bio_end_io_worker+0x36a/0x520 [btrfs]
[17808.436040]  ? debug_object_deactivate+0x56/0x130
[17808.436040]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.436040]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.436040]  process_one_work+0x2af/0x720
[17808.436040]  ? process_one_work+0x22b/0x720
[17808.436040]  worker_thread+0x4b/0x4f0
[17808.436040]  kthread+0x10f/0x150
[17808.436040]  ? process_one_work+0x720/0x720
[17808.436040]  ? kthread_create_on_node+0x40/0x40
[17808.436040]  ret_from_fork+0x2e/0x40
[17808.440016] ---[ end trace 42fc4dedec8f620c ]---
[17808.440464] ------------[ cut here ]------------
[17808.440951] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.441632] refcount_t: increment on 0; use-after-free.
[17808.442151] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.442884] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.443033] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.443033] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.443033] Call Trace:
[17808.443033]  dump_stack+0x68/0x93
[17808.443033]  __warn+0xcb/0xf0
[17808.443033]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.443033]  warn_slowpath_fmt+0x4f/0x60
[17808.443033]  refcount_inc+0x27/0x30
[17808.443033]  scrub_write_page_to_dev_replace+0x1a1/0x2a0 [btrfs]
[17808.443033]  scrub_write_block_to_dev_replace+0x40/0x60 [btrfs]
[17808.443033]  scrub_bio_end_io_worker+0x36a/0x520 [btrfs]
[17808.443033]  ? debug_object_deactivate+0x56/0x130
[17808.443033]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.443033]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.443033]  process_one_work+0x2af/0x720
[17808.443033]  ? process_one_work+0x22b/0x720
[17808.443033]  worker_thread+0x4b/0x4f0
[17808.443033]  kthread+0x10f/0x150
[17808.443033]  ? process_one_work+0x720/0x720
[17808.443033]  ? kthread_create_on_node+0x40/0x40
[17808.443033]  ret_from_fork+0x2e/0x40
[17808.456507] ---[ end trace 42fc4dedec8f620d ]---
[17808.457696] ------------[ cut here ]------------
[17808.457998] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:114 refcount_inc+0x27/0x30
[17808.458681] refcount_t: increment on 0; use-after-free.
[17808.459075] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.459685] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.460006] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.460006] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.460006] Call Trace:
[17808.460006]  dump_stack+0x68/0x93
[17808.460006]  __warn+0xcb/0xf0
[17808.460006]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.460006]  warn_slowpath_fmt+0x4f/0x60
[17808.460006]  refcount_inc+0x27/0x30
[17808.460006]  scrub_write_page_to_dev_replace+0x1a1/0x2a0 [btrfs]
[17808.460006]  scrub_write_block_to_dev_replace+0x40/0x60 [btrfs]
[17808.460006]  scrub_bio_end_io_worker+0x36a/0x520 [btrfs]
[17808.460006]  ? debug_object_deactivate+0x56/0x130
[17808.460006]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.460006]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.460006]  process_one_work+0x2af/0x720
[17808.460006]  ? process_one_work+0x22b/0x720
[17808.460006]  worker_thread+0x4b/0x4f0
[17808.460006]  kthread+0x10f/0x150
[17808.460006]  ? process_one_work+0x720/0x720
[17808.460006]  ? kthread_create_on_node+0x40/0x40
[17808.460006]  ret_from_fork+0x2e/0x40
[17808.467673] ---[ end trace 42fc4dedec8f620e ]---
[17808.469434] ------------[ cut here ]------------
[17808.469913] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.470600] refcount_t: underflow; use-after-free.
[17808.471125] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.471836] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.472005] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.472005] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.472005] Call Trace:
[17808.472005]  dump_stack+0x68/0x93
[17808.472005]  __warn+0xcb/0xf0
[17808.472005]  warn_slowpath_fmt+0x4f/0x60
[17808.472005]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.472005]  refcount_sub_and_test+0x60/0x70
[17808.472005]  refcount_dec_and_test+0x11/0x20
[17808.472005]  scrub_page_put+0x16/0x40 [btrfs]
[17808.472005]  scrub_block_put+0x5a/0x80 [btrfs]
[17808.472005]  scrub_bio_end_io_worker+0x77/0x520 [btrfs]
[17808.472005]  ? debug_object_deactivate+0x56/0x130
[17808.472005]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.472005]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.472005]  process_one_work+0x2af/0x720
[17808.472005]  ? process_one_work+0x22b/0x720
[17808.472005]  worker_thread+0x4b/0x4f0
[17808.472005]  kthread+0x10f/0x150
[17808.472005]  ? process_one_work+0x720/0x720
[17808.472005]  ? kthread_create_on_node+0x40/0x40
[17808.472005]  ret_from_fork+0x2e/0x40
[17808.479703] ---[ end trace 42fc4dedec8f620f ]---
[17808.480326] ------------[ cut here ]------------
[17808.480875] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.481584] refcount_t: underflow; use-after-free.
[17808.481885] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.482802] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.483383] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.483680] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.483680] Call Trace:
[17808.483680]  dump_stack+0x68/0x93
[17808.483680]  __warn+0xcb/0xf0
[17808.483680]  warn_slowpath_fmt+0x4f/0x60
[17808.483680]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.483680]  refcount_sub_and_test+0x60/0x70
[17808.483680]  refcount_dec_and_test+0x11/0x20
[17808.483680]  scrub_page_put+0x16/0x40 [btrfs]
[17808.483680]  scrub_block_put+0x5a/0x80 [btrfs]
[17808.487038]  scrub_bio_end_io_worker+0x77/0x520 [btrfs]
[17808.487038]  ? debug_object_deactivate+0x56/0x130
[17808.487038]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.487038]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.487038]  process_one_work+0x2af/0x720
[17808.487038]  ? process_one_work+0x22b/0x720
[17808.487038]  worker_thread+0x4b/0x4f0
[17808.487038]  kthread+0x10f/0x150
[17808.487038]  ? process_one_work+0x720/0x720
[17808.487038]  ? kthread_create_on_node+0x40/0x40
[17808.487038]  ret_from_fork+0x2e/0x40
[17808.490757] ---[ end trace 42fc4dedec8f6210 ]---
[17808.491117] ------------[ cut here ]------------
[17808.491448] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.492062] refcount_t: underflow; use-after-free.
[17808.492393] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.493053] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.493648] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.494030] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.494030] Call Trace:
[17808.494030]  dump_stack+0x68/0x93
[17808.494030]  __warn+0xcb/0xf0
[17808.494030]  warn_slowpath_fmt+0x4f/0x60
[17808.494030]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.494030]  refcount_sub_and_test+0x60/0x70
[17808.494030]  refcount_dec_and_test+0x11/0x20
[17808.494030]  scrub_page_put+0x16/0x40 [btrfs]
[17808.494030]  scrub_block_put+0x5a/0x80 [btrfs]
[17808.494030]  scrub_bio_end_io_worker+0x77/0x520 [btrfs]
[17808.494030]  ? debug_object_deactivate+0x56/0x130
[17808.494030]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.494030]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.494030]  process_one_work+0x2af/0x720
[17808.494030]  ? process_one_work+0x22b/0x720
[17808.494030]  worker_thread+0x4b/0x4f0
[17808.494030]  kthread+0x10f/0x150
[17808.494030]  ? process_one_work+0x720/0x720
[17808.494030]  ? kthread_create_on_node+0x40/0x40
[17808.494030]  ret_from_fork+0x2e/0x40
[17808.501354] ---[ end trace 42fc4dedec8f6211 ]---
[17808.502677] ------------[ cut here ]------------
[17808.503040] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.503639] refcount_t: underflow; use-after-free.
[17808.503954] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.504592] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.505187] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.505569] Workqueue: btrfs-scrub btrfs_scrub_helper [btrfs]
[17808.505569] Call Trace:
[17808.505569]  dump_stack+0x68/0x93
[17808.505569]  __warn+0xcb/0xf0
[17808.505569]  warn_slowpath_fmt+0x4f/0x60
[17808.505569]  ? scrub_workers_put+0x80/0x80 [btrfs]
[17808.505569]  refcount_sub_and_test+0x60/0x70
[17808.505569]  refcount_dec_and_test+0x11/0x20
[17808.505569]  scrub_page_put+0x16/0x40 [btrfs]
[17808.505569]  scrub_block_put+0x5a/0x80 [btrfs]
[17808.505569]  scrub_bio_end_io_worker+0x77/0x520 [btrfs]
[17808.505569]  ? debug_object_deactivate+0x56/0x130
[17808.505569]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.505569]  btrfs_scrub_helper+0xe/0x10 [btrfs]
[17808.505569]  process_one_work+0x2af/0x720
[17808.505569]  ? process_one_work+0x22b/0x720
[17808.505569]  worker_thread+0x4b/0x4f0
[17808.505569]  kthread+0x10f/0x150
[17808.505569]  ? process_one_work+0x720/0x720
[17808.505569]  ? kthread_create_on_node+0x40/0x40
[17808.505569]  ret_from_fork+0x2e/0x40
[17808.512438] ---[ end trace 42fc4dedec8f6212 ]---
[17808.512962] ------------[ cut here ]------------
[17808.513288] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.513895] refcount_t: underflow; use-after-free.
[17808.514241] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.514865] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.515220] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.515220] Workqueue: btrfs-scrubwrc btrfs_scrubwrc_helper [btrfs]
[17808.515220] Call Trace:
[17808.515220]  dump_stack+0x68/0x93
[17808.515220]  __warn+0xcb/0xf0
[17808.515220]  warn_slowpath_fmt+0x4f/0x60
[17808.515220]  ? debug_object_deactivate+0x56/0x130
[17808.515220]  refcount_sub_and_test+0x60/0x70
[17808.515220]  refcount_dec_and_test+0x11/0x20
[17808.515220]  scrub_page_put+0x16/0x40 [btrfs]
[17808.515220]  scrub_wr_bio_end_io_worker+0x93/0xe0 [btrfs]
[17808.515220]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.515220]  btrfs_scrubwrc_helper+0xe/0x10 [btrfs]
[17808.515220]  process_one_work+0x2af/0x720
[17808.515220]  ? process_one_work+0x22b/0x720
[17808.515220]  worker_thread+0x4b/0x4f0
[17808.515220]  kthread+0x10f/0x150
[17808.515220]  ? process_one_work+0x720/0x720
[17808.515220]  ? kthread_create_on_node+0x40/0x40
[17808.515220]  ret_from_fork+0x2e/0x40
[17808.528894] ---[ end trace 42fc4dedec8f6213 ]---
[17808.529383] ------------[ cut here ]------------
[17808.529760] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.530390] refcount_t: underflow; use-after-free.
[17808.530718] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.531354] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.531961] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.532329] Workqueue: btrfs-scrubwrc btrfs_scrubwrc_helper [btrfs]
[17808.532329] Call Trace:
[17808.532329]  dump_stack+0x68/0x93
[17808.532329]  __warn+0xcb/0xf0
[17808.532329]  warn_slowpath_fmt+0x4f/0x60
[17808.532329]  ? debug_object_deactivate+0x56/0x130
[17808.532329]  refcount_sub_and_test+0x60/0x70
[17808.532329]  refcount_dec_and_test+0x11/0x20
[17808.532329]  scrub_page_put+0x16/0x40 [btrfs]
[17808.532329]  scrub_wr_bio_end_io_worker+0x93/0xe0 [btrfs]
[17808.532329]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.532329]  btrfs_scrubwrc_helper+0xe/0x10 [btrfs]
[17808.532329]  process_one_work+0x2af/0x720
[17808.532329]  ? process_one_work+0x22b/0x720
[17808.532329]  worker_thread+0x4b/0x4f0
[17808.532329]  kthread+0x10f/0x150
[17808.532329]  ? process_one_work+0x720/0x720
[17808.538039]  ? kthread_create_on_node+0x40/0x40
[17808.538039]  ret_from_fork+0x2e/0x40
[17808.538830] ---[ end trace 42fc4dedec8f6214 ]---
[17808.539179] ------------[ cut here ]------------
[17808.539520] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.540144] refcount_t: underflow; use-after-free.
[17808.540472] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.541110] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.541704] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.542087] Workqueue: btrfs-scrubwrc btrfs_scrubwrc_helper [btrfs]
[17808.542087] Call Trace:
[17808.542087]  dump_stack+0x68/0x93
[17808.542087]  __warn+0xcb/0xf0
[17808.542087]  warn_slowpath_fmt+0x4f/0x60
[17808.542087]  ? debug_object_deactivate+0x56/0x130
[17808.542087]  refcount_sub_and_test+0x60/0x70
[17808.542087]  refcount_dec_and_test+0x11/0x20
[17808.542087]  scrub_page_put+0x16/0x40 [btrfs]
[17808.542087]  scrub_wr_bio_end_io_worker+0x93/0xe0 [btrfs]
[17808.542087]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.542087]  btrfs_scrubwrc_helper+0xe/0x10 [btrfs]
[17808.542087]  process_one_work+0x2af/0x720
[17808.542087]  ? process_one_work+0x22b/0x720
[17808.542087]  worker_thread+0x4b/0x4f0
[17808.542087]  kthread+0x10f/0x150
[17808.542087]  ? process_one_work+0x720/0x720
[17808.542087]  ? kthread_create_on_node+0x40/0x40
[17808.542087]  ret_from_fork+0x2e/0x40
[17808.548374] ---[ end trace 42fc4dedec8f6215 ]---
[17808.548700] ------------[ cut here ]------------
[17808.549074] WARNING: CPU: 1 PID: 9508 at lib/refcount.c:128 refcount_sub_and_test+0x60/0x70
[17808.549702] refcount_t: underflow; use-after-free.
[17808.550101] Modules linked in: btrfs(O) ext4 jbd2 mbcache xor raid6_pq netconsole xfs [last unloaded: btrfs]
[17808.550726] CPU: 1 PID: 9508 Comm: kworker/u4:6 Tainted: G        W  O    4.11.0-rc1+ #71
[17808.551079] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-20161122_114906-anatol 04/01/2014
[17808.551079] Workqueue: btrfs-scrubwrc btrfs_scrubwrc_helper [btrfs]
[17808.551079] Call Trace:
[17808.551079]  dump_stack+0x68/0x93
[17808.551079]  __warn+0xcb/0xf0
[17808.551079]  warn_slowpath_fmt+0x4f/0x60
[17808.551079]  ? debug_object_deactivate+0x56/0x130
[17808.551079]  refcount_sub_and_test+0x60/0x70
[17808.551079]  refcount_dec_and_test+0x11/0x20
[17808.551079]  scrub_page_put+0x16/0x40 [btrfs]
[17808.551079]  scrub_wr_bio_end_io_worker+0x93/0xe0 [btrfs]
[17808.551079]  btrfs_scrubparity_helper+0xef/0x620 [btrfs]
[17808.551079]  btrfs_scrubwrc_helper+0xe/0x10 [btrfs]
[17808.551079]  process_one_work+0x2af/0x720
[17808.551079]  ? process_one_work+0x22b/0x720
[17808.551079]  worker_thread+0x4b/0x4f0
[17808.551079]  kthread+0x10f/0x150
[17808.551079]  ? process_one_work+0x720/0x720
[17808.551079]  ? kthread_create_on_node+0x40/0x40
[17808.551079]  ret_from_fork+0x2e/0x40
[17808.559351] ---[ end trace 42fc4dedec8f6216 ]---
[17809.477360] BTRFS info (device vdb9): dev_replace from /dev/vdb7 (devid 2) to /dev/vdb10 finished
[17810.229098] BTRFS info (device vdb9): dev_replace from /dev/vdb8 (devid 3) to /dev/vdb7 started
[17810.518980] BTRFS info (device vdb9): dev_replace from /dev/vdb8 (devid 3) to /dev/vdb7 finished