* A collection of btrfs lockup stack traces
@ 2019-03-20 3:39 Zygo Blaxell
2019-03-20 3:45 ` Zygo Blaxell
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Zygo Blaxell @ 2019-03-20 3:39 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 49293 bytes --]
I haven't been able to easily reproduce these in a test environment;
however, they have been happening several times a year on servers in
production.
Kernel: most recent observation on 4.14.105 + cherry-picked deadlock
and misc hang fixes:
btrfs: wakeup cleaner thread when adding delayed iput
Btrfs: fix deadlock when allocating tree block during leaf/node split
Btrfs: use nofs context when initializing security xattrs to avoid deadlock
Btrfs: fix deadlock with memory reclaim during scrub
Btrfs: fix deadlock between clone/dedupe and rename
Also observed on 4.20.13, and 4.14.0..4.14.105 (4.14.106 is currently
running, but hasn't locked up yet).
Filesystem mount flags: compress=zstd,ssd,flushoncommit,space_cache=v2.
Configuration is either -draid1/-mraid1 or -dsingle/-mraid1. I've
also reproduced a lockup without flushoncommit.
The machines that are locking up all run the same workload:
rsync receiving data continuously (gigabytes aren't enough,
I can barely reproduce this once a month with 2TB of rsync
traffic from 10 simulated clients)
bees doing continuous dedupe
snapshots daily and after each rsync
snapshot deletes as required to maintain free space
scrubs twice monthly plus after each crash
watchdog does a 'mkdir foo; rmdir foo' every few seconds.
If this takes longer than 50 minutes, collect a stack trace;
longer than 60 minutes, reboot the machine.
Observations:
In the past I've hit the known deadlocks between rsync
(rename) and dedupe, and between snapshot delete and scrub.
These deadlocks aren't those, and current kernels can still
deadlock with fixes for those (and some others) applied.
Deadlocks can occur when there are no snapshot deletes active
and scrub is not running. In a few cases there's been a snapshot
create active during a deadlock event.
The deadlocks occur in bursts, which correlate to periods
of rapid data expansion on the filesystem. e.g. after a big
snapshot delete frees a TB or two of space, there are several
lockups a few days (or even hours) apart. When the filesystem
gets near the free space threshold and a series of smaller
snapshot deletes occur, so that allocated space stays relatively
stable, then there can be months between lockups.
Other bugs in 4.14 kernels break lockdep soon after boot, so
there's only lockdep information in a few cases by the time the
btrfs deadlock happens.
There seem to be a few different identifiable behaviors, e.g.
in some cases the 50-minute timer expires (so a stack trace is
collected) but the 60-minute timer doesn't expire (so the machine
got itself out of the lockup somehow, and doesn't reboot).
Distractions:
In some cases increasing vm.dirty_ratio will unfreeze the
filesystem, in other cases it does nothing. Decreasing
vm.dirty_ratio (e.g. to 2) locks up the filesystem almost
immediately, but it can recover if vm.dirty_ratio is increased
again. I'm not sure if this is because there are two separate
bugs, or just a single bug with corner cases.
Here's a recent lockup example with lockdep output, from 4.14.105 with
the cherry-picked patches above:
[Fri Mar 15 21:05:59 2019] 1 lock held by khungtaskd/465:
[Fri Mar 15 21:05:59 2019] #0: (tasklist_lock){.+.+}, at: [<ffffffff950f0367>] debug_show_all_locks+0x37/0x190
[Fri Mar 15 21:05:59 2019] 3 locks held by btrfs-transacti/31131:
[Fri Mar 15 21:05:59 2019] #0: (&fs_info->transaction_kthread_mutex){+.+.}, at: [<ffffffff954352b4>] transaction_kthread+0x74/0x1d0
[Fri Mar 15 21:05:59 2019] #1: (&fs_info->delalloc_root_mutex){+.+.}, at: [<ffffffff9544f728>] btrfs_start_delalloc_roots+0x58/0x380
[Fri Mar 15 21:05:59 2019] #2: (&root->delalloc_mutex){+.+.}, at: [<ffffffff9544f302>] __start_delalloc_inodes+0x52/0x320
[Fri Mar 15 21:05:59 2019] 4 locks held by crawl_61509/20348:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff95479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 15 21:05:59 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff95479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 15 21:05:59 2019] #3: (sb_internal){.+.+}, at: [<ffffffff9543b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 15 21:05:59 2019] 2 locks held by crawl_294/20349:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff95479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 15 21:05:59 2019] 4 locks held by crawl_294/20350:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff95479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 15 21:05:59 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff95479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 15 21:05:59 2019] #3: (sb_internal){.+.+}, at: [<ffffffff9543b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 15 21:05:59 2019] 2 locks held by crawl_294/25732:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff95479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 15 21:05:59 2019] 4 locks held by crawl_61509/25733:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff95479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 15 21:05:59 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff95479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 15 21:05:59 2019] #3: (sb_internal){.+.+}, at: [<ffffffff9543b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 15 21:05:59 2019] 1 lock held by btrfs/14776:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] 1 lock held by btrfs/14777:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] 1 lock held by btrfs/14778:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] 1 lock held by btrfs/14779:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] 1 lock held by btrfs/14780:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:05:59 2019] 2 locks held by kworker/u8:5/5813:
[Fri Mar 15 21:05:59 2019] #0: ("%s-%s""btrfs", name){+.+.}, at: [<ffffffff950ae2be>] process_one_work+0x17e/0x650
[Fri Mar 15 21:05:59 2019] #1: ((&work->normal_work)){+.+.}, at: [<ffffffff950ae2be>] process_one_work+0x17e/0x650
[Fri Mar 15 21:05:59 2019] 4 locks held by rsync/26929:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4d90>] mnt_want_write+0x20/0x50
[Fri Mar 15 21:05:59 2019] #1: (&type->i_mutex_dir_key#6/1){+.+.}, at: [<ffffffff952bf698>] do_rmdir+0x168/0x210
[Fri Mar 15 21:05:59 2019] #2: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffff952bddcf>] vfs_rmdir+0x4f/0x140
[Fri Mar 15 21:05:59 2019] #3: (sb_internal){.+.+}, at: [<ffffffff9543b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 15 21:05:59 2019] 1 lock held by rsync/2977:
[Fri Mar 15 21:05:59 2019] #0: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffff952b9307>] lookup_slow+0xe7/0x210
[Fri Mar 15 21:05:59 2019] 3 locks held by mkdir/25975:
[Fri Mar 15 21:05:59 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff952d4d90>] mnt_want_write+0x20/0x50
[Fri Mar 15 21:05:59 2019] #1: (&type->i_mutex_dir_key#6/1){+.+.}, at: [<ffffffff952bf42f>] filename_create+0x6f/0x140
[Fri Mar 15 21:05:59 2019] #2: (sb_internal){.+.+}, at: [<ffffffff9543b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 15 21:05:59 2019] =============================================
[Fri Mar 15 21:53:36 2019] sysrq: SysRq : Show Blocked State
[Fri Mar 15 21:53:36 2019] task PC stack pid father
[Fri Mar 15 21:53:36 2019] btrfs-transacti D 0 31131 2 0x80000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] schedule_timeout+0x20f/0x590
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? mark_held_locks+0x6a/0x90
[Fri Mar 15 21:53:36 2019] ? _raw_spin_unlock_irq+0x29/0x40
[Fri Mar 15 21:53:36 2019] ? wait_for_common+0x131/0x1f0
[Fri Mar 15 21:53:36 2019] wait_for_common+0x131/0x1f0
[Fri Mar 15 21:53:36 2019] ? wake_up_q+0x70/0x70
[Fri Mar 15 21:53:36 2019] __start_delalloc_inodes+0x22f/0x320
[Fri Mar 15 21:53:36 2019] btrfs_start_delalloc_roots+0x1cd/0x380
[Fri Mar 15 21:53:36 2019] btrfs_commit_transaction+0x807/0xa20
[Fri Mar 15 21:53:36 2019] ? start_transaction+0x89/0x4d0
[Fri Mar 15 21:53:36 2019] transaction_kthread+0x194/0x1d0
[Fri Mar 15 21:53:36 2019] kthread+0x10d/0x140
[Fri Mar 15 21:53:36 2019] ? btrfs_cleanup_transaction+0x650/0x650
[Fri Mar 15 21:53:36 2019] ? kthread_create_on_node+0x40/0x40
[Fri Mar 15 21:53:36 2019] ret_from_fork+0x27/0x50
[Fri Mar 15 21:53:36 2019] hash_prefetch D 0 31272 31220 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690043607
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd686ffbb80 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00000000000e09c1 RCX: 00007fd690043607
[Fri Mar 15 21:53:36 2019] RDX: 00000000000e09c1 RSI: 00007fd574002590 RDI: 0000000000000003
[Fri Mar 15 21:53:36 2019] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007fd686ffbf40
[Fri Mar 15 21:53:36 2019] R10: 0000000000000180 R11: 0000000000000293 R12: 00007fd574002590
[Fri Mar 15 21:53:36 2019] R13: 0000000000000003 R14: 00007fd57400b0f0 R15: 00007fd686ffc120
[Fri Mar 15 21:53:36 2019] crawl_writeback D 0 31275 31220 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? rwsem_down_write_failed+0x134/0x2b0
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] rwsem_down_write_failed+0x139/0x2b0
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 15 21:53:36 2019] call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 15 21:53:36 2019] down_write_nested+0x87/0xb0
[Fri Mar 15 21:53:36 2019] do_unlinkat+0x113/0x300
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690044ed7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd685ff9f58 EFLAGS: 00000246 ORIG_RAX: 0000000000000107
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007fd685ff9f80 RCX: 00007fd690044ed7
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000000 RSI: 00007fd56c0008f0 RDI: 0000000000000003
[Fri Mar 15 21:53:36 2019] RBP: 000055ccdac39f80 R08: 0000000000000000 R09: 00007fd685ffa030
[Fri Mar 15 21:53:36 2019] R10: 000055ccda9a4c80 R11: 0000000000000246 R12: 00007fd685ffa090
[Fri Mar 15 21:53:36 2019] R13: 00007fd685ffa400 R14: 00007fd685ffa3d0 R15: 00007fd685ffa350
[Fri Mar 15 21:53:36 2019] crawl_61509 D 0 20348 31220 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 15 21:53:36 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 15 21:53:36 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690048dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd68e760ac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007fd57049edb0 RCX: 00007fd690048dd7
[Fri Mar 15 21:53:36 2019] RDX: 00007fd57049edb0 RSI: 00000000c0189436 RDI: 0000000000000627
[Fri Mar 15 21:53:36 2019] RBP: 00007fd68e760e40 R08: 00007fd5705d9950 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007fd5706227e0 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 15 21:53:36 2019] R13: 0000000000000018 R14: 00007fd5705d9968 R15: 00007fd68e760e48
[Fri Mar 15 21:53:36 2019] crawl_294 D 0 20349 31220 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? rwsem_down_write_failed+0x134/0x2b0
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] rwsem_down_write_failed+0x139/0x2b0
[Fri Mar 15 21:53:36 2019] ? call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 15 21:53:36 2019] call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 15 21:53:36 2019] down_write_nested+0x87/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 15 21:53:36 2019] ? rcu_read_lock_sched_held+0x68/0x70
[Fri Mar 15 21:53:36 2019] ? rcu_sync_lockdep_assert+0x30/0x60
[Fri Mar 15 21:53:36 2019] ? __sb_start_write+0xcc/0x1b0
[Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:53:36 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690048dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd687ffc0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007fd568332780 RCX: 00007fd690048dd7
[Fri Mar 15 21:53:36 2019] RDX: 00007fd568332780 RSI: 00000000c0189436 RDI: 0000000000000067
[Fri Mar 15 21:53:36 2019] RBP: 00007fd687ffc420 R08: 00007fd56804c940 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007fd568022ed0 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 15 21:53:36 2019] R13: 0000000000000018 R14: 00007fd56804c958 R15: 00007fd687ffc428
[Fri Mar 15 21:53:36 2019] crawl_294 D 0 20350 31220 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 15 21:53:36 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 15 21:53:36 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690048dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd68cf5e0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007fd55c224c70 RCX: 00007fd690048dd7
[Fri Mar 15 21:53:36 2019] RDX: 00007fd55c224c70 RSI: 00000000c0189436 RDI: 0000000000000067
[Fri Mar 15 21:53:36 2019] RBP: 00007fd68cf5e420 R08: 00007fd55c272c80 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007fd55c5effb0 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 15 21:53:36 2019] R13: 0000000000000018 R14: 00007fd55c272c98 R15: 00007fd68cf5e428
[Fri Mar 15 21:53:36 2019] crawl_294 D 0 25732 31220 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? rwsem_down_write_failed+0x134/0x2b0
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] rwsem_down_write_failed+0x139/0x2b0
[Fri Mar 15 21:53:36 2019] ? call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 15 21:53:36 2019] call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 15 21:53:36 2019] down_write_nested+0x87/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 15 21:53:36 2019] ? rcu_read_lock_sched_held+0x68/0x70
[Fri Mar 15 21:53:36 2019] ? rcu_sync_lockdep_assert+0x30/0x60
[Fri Mar 15 21:53:36 2019] ? __sb_start_write+0xcc/0x1b0
[Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:53:36 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690048dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd68ef620a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007fd5542c9590 RCX: 00007fd690048dd7
[Fri Mar 15 21:53:36 2019] RDX: 00007fd5542c9590 RSI: 00000000c0189436 RDI: 0000000000000067
[Fri Mar 15 21:53:36 2019] RBP: 00007fd68ef62420 R08: 00007fd5541acf90 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007fd5541b57a0 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 15 21:53:36 2019] R13: 0000000000000018 R14: 00007fd5541acfa8 R15: 00007fd68ef62428
[Fri Mar 15 21:53:36 2019] crawl_61509 D 0 25733 31220 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 15 21:53:36 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 15 21:53:36 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690048dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd68d75f0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007fd56070c9b0 RCX: 00007fd690048dd7
[Fri Mar 15 21:53:36 2019] RDX: 00007fd56070c9b0 RSI: 00000000c0189436 RDI: 0000000000000427
[Fri Mar 15 21:53:36 2019] RBP: 00007fd68d75f420 R08: 00007fd560195fa0 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007fd56070cf80 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 15 21:53:36 2019] R13: 0000000000000018 R14: 00007fd560195fb8 R15: 00007fd68d75f428
[Fri Mar 15 21:53:36 2019] btrfs D 0 14776 14768 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_for_commit+0x3f/0x70
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 15 21:53:36 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 15 21:53:36 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 15 21:53:36 2019] ? wait_woken+0x70/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007f33719fed58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000561c1b5de4f0 RCX: 00007f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RDX: 0000561c1b5de4f0 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 15 21:53:36 2019] RBP: 0000000000000000 R08: 00007f33719ff700 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007f33719ff700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 15 21:53:36 2019] R13: 00007ffe6a708eaf R14: 0000000000000000 R15: 00007f3372a7c040
[Fri Mar 15 21:53:36 2019] btrfs D 0 14777 14768 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_for_commit+0x3f/0x70
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 15 21:53:36 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 15 21:53:36 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 15 21:53:36 2019] ? wait_woken+0x70/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007f33711fdd58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000561c1b5de970 RCX: 00007f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RDX: 0000561c1b5de970 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 15 21:53:36 2019] RBP: 0000000000000000 R08: 00007f33711fe700 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007f33711fe700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 15 21:53:36 2019] R13: 00007ffe6a708eaf R14: 0000000000000000 R15: 00007f3372a7c040
[Fri Mar 15 21:53:36 2019] btrfs D 0 14778 14768 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_for_commit+0x3f/0x70
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 15 21:53:36 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 15 21:53:36 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 15 21:53:36 2019] ? wait_woken+0x70/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007f33709fcd58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000561c1b5dedf0 RCX: 00007f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RDX: 0000561c1b5dedf0 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 15 21:53:36 2019] RBP: 0000000000000000 R08: 00007f33709fd700 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007f33709fd700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 15 21:53:36 2019] R13: 00007ffe6a708eaf R14: 0000000000000000 R15: 00007f3372a7c040
[Fri Mar 15 21:53:36 2019] btrfs D 0 14779 14768 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_for_commit+0x3f/0x70
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 15 21:53:36 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 15 21:53:36 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 15 21:53:36 2019] ? wait_woken+0x70/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007f33701fbd58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000561c1b5df270 RCX: 00007f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RDX: 0000561c1b5df270 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 15 21:53:36 2019] RBP: 0000000000000000 R08: 00007f33701fc700 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007f33701fc700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 15 21:53:36 2019] R13: 00007ffe6a708eaf R14: 0000000000000000 R15: 00007f3372a7c040
[Fri Mar 15 21:53:36 2019] btrfs D 0 14780 14768 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_for_commit+0x3f/0x70
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 15 21:53:36 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 15 21:53:36 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 15 21:53:36 2019] ? wait_woken+0x70/0xa0
[Fri Mar 15 21:53:36 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
[Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007f336f9fad58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000561c1b5df6f0 RCX: 00007f3371ae0dd7
[Fri Mar 15 21:53:36 2019] RDX: 0000561c1b5df6f0 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 15 21:53:36 2019] RBP: 0000000000000000 R08: 00007f336f9fb700 R09: 0000000000000000
[Fri Mar 15 21:53:36 2019] R10: 00007f336f9fb700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 15 21:53:36 2019] R13: 00007ffe6a708eaf R14: 0000000000000000 R15: 00007f3372a7c040
[Fri Mar 15 21:53:36 2019] kworker/u8:5 D 0 5813 2 0x80000000
[Fri Mar 15 21:53:36 2019] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] io_schedule+0x12/0x40
[Fri Mar 15 21:53:36 2019] __lock_page+0x110/0x140
[Fri Mar 15 21:53:36 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Fri Mar 15 21:53:36 2019] extent_write_cache_pages.constprop.27+0x2d9/0x3b0
[Fri Mar 15 21:53:36 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 15 21:53:36 2019] extent_writepages+0x4e/0x70
[Fri Mar 15 21:53:36 2019] ? btrfs_direct_IO+0x3e0/0x3e0
[Fri Mar 15 21:53:36 2019] do_writepages+0x23/0x80
[Fri Mar 15 21:53:36 2019] __filemap_fdatawrite_range+0xaa/0xe0
[Fri Mar 15 21:53:36 2019] btrfs_run_delalloc_work+0x17/0x60
[Fri Mar 15 21:53:36 2019] btrfs_scrubparity_helper+0x8f/0x640
[Fri Mar 15 21:53:36 2019] process_one_work+0x204/0x650
[Fri Mar 15 21:53:36 2019] worker_thread+0x4d/0x3b0
[Fri Mar 15 21:53:36 2019] kthread+0x10d/0x140
[Fri Mar 15 21:53:36 2019] ? process_one_work+0x650/0x650
[Fri Mar 15 21:53:36 2019] ? kthread_create_on_node+0x40/0x40
[Fri Mar 15 21:53:36 2019] ret_from_fork+0x27/0x50
[Fri Mar 15 21:53:36 2019] rsync D 0 26929 26812 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_start_transaction_fallback_global_rsv+0x34/0x200
[Fri Mar 15 21:53:36 2019] btrfs_rmdir+0x67/0x180
[Fri Mar 15 21:53:36 2019] vfs_rmdir+0xbb/0x140
[Fri Mar 15 21:53:36 2019] do_rmdir+0x1fe/0x210
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7ff34b804f07
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffe20b69618 EFLAGS: 00000246 ORIG_RAX: 0000000000000054
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007ff34b804f07
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007ffe20b69680
[Fri Mar 15 21:53:36 2019] RBP: 00000000000041ed R08: 0000000000000000 R09: 0000000000008000
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe20b69680
[Fri Mar 15 21:53:36 2019] R13: 0000000000004000 R14: 0000000000000000 R15: 00000000000041ed
[Fri Mar 15 21:53:36 2019] mkdir D 0 25975 25972 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_mkdir+0x50/0x1d0
[Fri Mar 15 21:53:36 2019] vfs_mkdir+0x134/0x1c0
[Fri Mar 15 21:53:36 2019] SyS_mkdirat+0xb3/0xf0
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fa5def83447
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffcd2af2258 EFLAGS: 00000206 ORIG_RAX: 0000000000000053
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007ffcd2af2b28 RCX: 00007fa5def83447
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007ffcd2af2b4d
[Fri Mar 15 21:53:36 2019] RBP: 00007ffcd2af23d0 R08: 00000000000001ff R09: 000055775e473e10
[Fri Mar 15 21:53:36 2019] R10: 0000000000000376 R11: 0000000000000206 R12: 000055775e473e50
[Fri Mar 15 21:53:36 2019] R13: 00000000000001ff R14: 00007ffcd2af2570 R15: 0000000000000001
[Fri Mar 15 21:53:36 2019] rsync D 0 13191 13160 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f10ce1e34b0
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffcc3812488 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f10ce1e34b0
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffcc38146f0
[Fri Mar 15 21:53:36 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f10ce235950
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcc3814707
[Fri Mar 15 21:53:36 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f10ce271320
[Fri Mar 15 21:53:36 2019] rsync D 0 530 529 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f64876134b0
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffe8bd0d108 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f64876134b0
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffe8bd0f370
[Fri Mar 15 21:53:36 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f6487665950
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe8bd0f387
[Fri Mar 15 21:53:36 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f64876a1320
[Fri Mar 15 21:53:36 2019] rsync D 0 5301 5152 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f8f498434b0
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffc794a2168 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f8f498434b0
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc794a43d0
[Fri Mar 15 21:53:36 2019] RBP: 000000000003a2f8 R08: 000000000000fff8 R09: 00007f8f49895f10
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc794a43ed
[Fri Mar 15 21:53:36 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f8f498d1320
[Fri Mar 15 21:53:36 2019] rsync D 0 7098 7002 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fddb705b4b0
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffcd2922968 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fddb705b4b0
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffcd2924bd0
[Fri Mar 15 21:53:36 2019] RBP: 000000000003a2f8 R08: 000000000000ffff R09: 00007fddb70adb30
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcd2924c10
[Fri Mar 15 21:53:36 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fddb70e9320
[Fri Mar 15 21:53:36 2019] rsync D 0 18905 18820 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_start_transaction_fallback_global_rsv+0x34/0x200
[Fri Mar 15 21:53:36 2019] btrfs_unlink+0x30/0xd0
[Fri Mar 15 21:53:36 2019] vfs_unlink+0xf8/0x190
[Fri Mar 15 21:53:36 2019] do_unlinkat+0x287/0x300
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f61535dcea7
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffe50a14ae8 EFLAGS: 00000246 ORIG_RAX: 0000000000000057
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f61535dcea7
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000000 RSI: 00000000000081a4 RDI: 00007ffe50a15c60
[Fri Mar 15 21:53:36 2019] RBP: 00000000000081a4 R08: 000000000000fccf R09: 00007f615362def0
[Fri Mar 15 21:53:36 2019] R10: 00007f615137300a R11: 0000000000000246 R12: 00007ffe50a15c60
[Fri Mar 15 21:53:36 2019] R13: 0000000000008000 R14: 0000000000000000 R15: 00000000000081a4
[Fri Mar 15 21:53:36 2019] rsync D 0 26494 26493 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fe53c5c34b0
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffd48eca238 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe53c5c34b0
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffd48ecc4a0
[Fri Mar 15 21:53:36 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007fe53c615950
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd48ecc4b7
[Fri Mar 15 21:53:36 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fe53c651320
[Fri Mar 15 21:53:36 2019] rsync D 0 12384 12322 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f57d716b4b0
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffd8a2a00a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f57d716b4b0
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffd8a2a2310
[Fri Mar 15 21:53:36 2019] RBP: 000000000003a2f8 R08: 0000000000000000 R09: 00007f57d71be100
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd8a2a233b
[Fri Mar 15 21:53:36 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f57d71f9320
[Fri Mar 15 21:53:36 2019] rsync D 0 21369 21368 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 15 21:53:36 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 15 21:53:36 2019] start_transaction+0x29b/0x4d0
[Fri Mar 15 21:53:36 2019] btrfs_create+0x47/0x1d0
[Fri Mar 15 21:53:36 2019] lookup_open+0x404/0x760
[Fri Mar 15 21:53:36 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 15 21:53:36 2019] path_openat+0x4e9/0xcc0
[Fri Mar 15 21:53:36 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 15 21:53:36 2019] do_filp_open+0x8a/0xf0
[Fri Mar 15 21:53:36 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 15 21:53:36 2019] ? do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_sys_open+0x130/0x200
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fae262bb4b0
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007fff14b786b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fae262bb4b0
[Fri Mar 15 21:53:36 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007fff14b7a920
[Fri Mar 15 21:53:36 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007fae2630d950
[Fri Mar 15 21:53:36 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff14b7a937
[Fri Mar 15 21:53:36 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fae26349320
[Fri Mar 15 21:53:36 2019] watchdog-mdtest D 0 30096 25972 0x00000000
[Fri Mar 15 21:53:36 2019] Call Trace:
[Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
[Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
[Fri Mar 15 21:53:36 2019] io_schedule+0x12/0x40
[Fri Mar 15 21:53:36 2019] wait_on_page_bit+0x10d/0x140
[Fri Mar 15 21:53:36 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Fri Mar 15 21:53:36 2019] read_extent_buffer_pages+0x215/0x2d0
[Fri Mar 15 21:53:36 2019] ? free_root_pointers+0x60/0x60
[Fri Mar 15 21:53:36 2019] btree_read_extent_buffer_pages+0x86/0x100
[Fri Mar 15 21:53:36 2019] read_tree_block+0x2d/0x50
[Fri Mar 15 21:53:36 2019] read_block_for_search.isra.11+0x117/0x310
[Fri Mar 15 21:53:36 2019] btrfs_search_slot+0x3e7/0x9c0
[Fri Mar 15 21:53:36 2019] btrfs_lookup_dir_item+0x69/0xb0
[Fri Mar 15 21:53:36 2019] btrfs_lookup_dentry+0xa0/0x530
[Fri Mar 15 21:53:36 2019] ? lookup_slow+0xe7/0x210
[Fri Mar 15 21:53:36 2019] btrfs_lookup+0xe/0x30
[Fri Mar 15 21:53:36 2019] lookup_slow+0x131/0x210
[Fri Mar 15 21:53:36 2019] walk_component+0x1b2/0x2d0
[Fri Mar 15 21:53:36 2019] path_lookupat+0x64/0x1f0
[Fri Mar 15 21:53:36 2019] filename_lookup+0xa9/0x170
[Fri Mar 15 21:53:36 2019] ? getname_flags+0x4a/0x1e0
[Fri Mar 15 21:53:36 2019] ? __check_object_size+0xa7/0x190
[Fri Mar 15 21:53:36 2019] ? strncpy_from_user+0x48/0x160
[Fri Mar 15 21:53:36 2019] ? _raw_spin_unlock+0x24/0x30
[Fri Mar 15 21:53:36 2019] ? vfs_statx+0x63/0xb0
[Fri Mar 15 21:53:36 2019] vfs_statx+0x63/0xb0
[Fri Mar 15 21:53:36 2019] SyS_newstat+0x2b/0x50
[Fri Mar 15 21:53:36 2019] ? up_read+0x1c/0x40
[Fri Mar 15 21:53:36 2019] ? trace_hardirqs_off_thunk+0x1a/0x1c
[Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
[Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 15 21:53:36 2019] RIP: 0033:0x7f5ec1373055
[Fri Mar 15 21:53:36 2019] RSP: 002b:00007ffd8121ace8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 0000000000000024 RCX: 00007f5ec1373055
[Fri Mar 15 21:53:36 2019] RDX: 00007ffd8121acf0 RSI: 00007ffd8121acf0 RDI: 0000000001a55c48
[Fri Mar 15 21:53:36 2019] RBP: 0000000001a55c48 R08: 0000000000710710 R09: 00007f5ec13c57b0
[Fri Mar 15 21:53:36 2019] R10: 00007f5ec13c2430 R11: 0000000000000246 R12: 0000000001a55c48
[Fri Mar 15 21:53:36 2019] R13: 00007ffd8121ae00 R14: 00007ffd8121ae00 R15: 0000000001a546e8
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: A collection of btrfs lockup stack traces
2019-03-20 3:39 A collection of btrfs lockup stack traces Zygo Blaxell
@ 2019-03-20 3:45 ` Zygo Blaxell
2019-03-20 3:49 ` A collection of btrfs lockup stack traces (4.20.13) Zygo Blaxell
2019-03-27 2:37 ` A collection of btrfs lockup stack traces (4.14.106, no __sb_start_write) Zygo Blaxell
2 siblings, 0 replies; 6+ messages in thread
From: Zygo Blaxell @ 2019-03-20 3:45 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 49282 bytes --]
On Tue, Mar 19, 2019 at 11:39:59PM -0400, Zygo Blaxell wrote:
> I haven't been able to easily reproduce these in a test environment;
> however, they have been happening several times a year on servers in
> production.
>
> Kernel: most recent observation on 4.14.105 + cherry-picked deadlock
> and misc hang fixes:
>
> btrfs: wakeup cleaner thread when adding delayed iput
> Btrfs: fix deadlock when allocating tree block during leaf/node split
> Btrfs: use nofs context when initializing security xattrs to avoid deadlock
> Btrfs: fix deadlock with memory reclaim during scrub
> Btrfs: fix deadlock between clone/dedupe and rename
>
> Also observed on 4.20.13, and 4.14.0..4.14.105 (4.14.106 is currently
> running, but hasn't locked up yet).
>
> Filesystem mount flags: compress=zstd,ssd,flushoncommit,space_cache=v2.
> Configuration is either -draid1/-mraid1 or -dsingle/-mraid1. I've
> also reproduced a lockup without flushoncommit.
>
> The machines that are locking up all run the same workload:
>
> rsync receiving data continuously (gigabytes aren't enough,
> I can barely reproduce this once a month with 2TB of rsync
> traffic from 10 simulated clients)
>
> bees doing continuous dedupe
>
> snapshots daily and after each rsync
>
> snapshot deletes as required to maintain free space
>
> scrubs twice monthly plus after each crash
>
> watchdog does a 'mkdir foo; rmdir foo' every few seconds.
> If this takes longer than 50 minutes, collect a stack trace;
> longer than 60 minutes, reboot the machine.
>
> Observations:
>
> In the past I've hit the known deadlocks between rsync
> (rename) and dedupe, and between snapshot delete and scrub.
> These deadlocks aren't those, and current kernels can still
> deadlock with fixes for those (and some others) applied.
>
> Deadlocks can occur when there are no snapshot deletes active
> and scrub is not running. In a few cases there's been a snapshot
> create active during a deadlock event.
>
> The deadlocks occur in bursts, which correlate to periods
> of rapid data expansion on the filesystem. e.g. after a big
> snapshot delete frees a TB or two of space, there are several
> lockups a few days (or even hours) apart. When the filesystem
> gets near the free space threshold and a series of smaller
> snapshot deletes occur, so that allocated space stays relatively
> stable, then there can be months between lockups.
>
> Other bugs in 4.14 kernels break lockdep soon after boot, so
> there's only lockdep information in a few cases by the time the
> btrfs deadlock happens.
>
> There seem to be a few different identifiable behaviors, e.g.
> in some cases the 50-minute timer expires (so a stack trace is
> collected) but the 60-minute timer doesn't expire (so the machine
> got itself out of the lockup somehow, and doesn't reboot).
>
> Distractions:
>
> In some cases increasing vm.dirty_ratio will unfreeze the
> filesystem, in other cases it does nothing. Decreasing
> vm.dirty_ratio (e.g. to 2) locks up the filesystem almost
> immediately, but it can recover if vm.dirty_ratio is increased
> again. I'm not sure if this is because there are two separate
> bugs, or just a single bug with corner cases.
>
> Here's a recent lockup example with lockdep output, from 4.14.105 with
> the cherry-picked patches above:
And another:
[Fri Mar 8 03:50:32 2019] 1 lock held by khungtaskd/465:
[Fri Mar 8 03:50:32 2019] #0: (tasklist_lock){.+.+}, at: [<ffffffff870f0367>] debug_show_all_locks+0x37/0x190
[Fri Mar 8 03:50:32 2019] 3 locks held by btrfs-transacti/27078:
[Fri Mar 8 03:50:32 2019] #0: (&fs_info->transaction_kthread_mutex){+.+.}, at: [<ffffffff874352b4>] transaction_kthread+0x74/0x1d0
[Fri Mar 8 03:50:32 2019] #1: (&fs_info->delalloc_root_mutex){+.+.}, at: [<ffffffff8744f728>] btrfs_start_delalloc_roots+0x58/0x380
[Fri Mar 8 03:50:32 2019] #2: (&root->delalloc_mutex){+.+.}, at: [<ffffffff8744f302>] __start_delalloc_inodes+0x52/0x320
[Fri Mar 8 03:50:32 2019] 4 locks held by crawl_290/13174:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff87479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 8 03:50:32 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff87479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 8 03:50:32 2019] #3: (sb_internal){.+.+}, at: [<ffffffff8743b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 8 03:50:32 2019] 4 locks held by crawl_302/15836:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff87479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 8 03:50:32 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff87479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 8 03:50:32 2019] #3: (sb_internal){.+.+}, at: [<ffffffff8743b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 8 03:50:32 2019] 4 locks held by crawl_275/15837:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff87479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 8 03:50:32 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff87479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 8 03:50:32 2019] #3: (sb_internal){.+.+}, at: [<ffffffff8743b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 8 03:50:32 2019] 4 locks held by crawl_299/15838:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff87479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 8 03:50:32 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff87479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 8 03:50:32 2019] #3: (sb_internal){.+.+}, at: [<ffffffff8743b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 8 03:50:32 2019] 4 locks held by crawl_290/15839:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffff87479bde>] btrfs_dedupe_file_range+0x8e/0x660
[Fri Mar 8 03:50:32 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffff87479bef>] btrfs_dedupe_file_range+0x9f/0x660
[Fri Mar 8 03:50:32 2019] #3: (sb_internal){.+.+}, at: [<ffffffff8743b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 8 03:50:32 2019] 1 lock held by btrfs/9795:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] 1 lock held by btrfs/9796:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] 1 lock held by btrfs/9797:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] 1 lock held by btrfs/9798:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] 1 lock held by btrfs/9799:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4e1b>] mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 03:50:32 2019] 3 locks held by rsync/24322:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872aa7cc>] do_sys_ftruncate.constprop.1+0xdc/0x180
[Fri Mar 8 03:50:32 2019] #1: (&sb->s_type->i_mutex_key#14){+.+.}, at: [<ffffffff872aa3ff>] do_truncate+0x4f/0x90
[Fri Mar 8 03:50:32 2019] #2: (sb_internal){.+.+}, at: [<ffffffff8743b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 8 03:50:32 2019] 2 locks held by kworker/u8:20/20623:
[Fri Mar 8 03:50:32 2019] #0: ("%s-%s""btrfs", name){+.+.}, at: [<ffffffff870ae2be>] process_one_work+0x17e/0x650
[Fri Mar 8 03:50:32 2019] #1: ((&work->normal_work)){+.+.}, at: [<ffffffff870ae2be>] process_one_work+0x17e/0x650
[Fri Mar 8 03:50:32 2019] 3 locks held by mkdir/17020:
[Fri Mar 8 03:50:32 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffff872d4d90>] mnt_want_write+0x20/0x50
[Fri Mar 8 03:50:32 2019] #1: (&type->i_mutex_dir_key#6/1){+.+.}, at: [<ffffffff872bf42f>] filename_create+0x6f/0x140
[Fri Mar 8 03:50:32 2019] #2: (sb_internal){.+.+}, at: [<ffffffff8743b5d6>] start_transaction+0x2b6/0x4d0
[Fri Mar 8 03:50:32 2019] =============================================
[Fri Mar 8 04:38:21 2019] task PC stack pid father
[Fri Mar 8 04:38:21 2019] btrfs-transacti D 0 27078 2 0x80000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] schedule_timeout+0x20f/0x590
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? mark_held_locks+0x6a/0x90
[Fri Mar 8 04:38:21 2019] ? _raw_spin_unlock_irq+0x29/0x40
[Fri Mar 8 04:38:21 2019] ? wait_for_common+0x131/0x1f0
[Fri Mar 8 04:38:21 2019] wait_for_common+0x131/0x1f0
[Fri Mar 8 04:38:21 2019] ? wake_up_q+0x70/0x70
[Fri Mar 8 04:38:21 2019] __start_delalloc_inodes+0x22f/0x320
[Fri Mar 8 04:38:21 2019] btrfs_start_delalloc_roots+0x1cd/0x380
[Fri Mar 8 04:38:21 2019] btrfs_commit_transaction+0x807/0xa20
[Fri Mar 8 04:38:21 2019] ? start_transaction+0x89/0x4d0
[Fri Mar 8 04:38:21 2019] transaction_kthread+0x194/0x1d0
[Fri Mar 8 04:38:21 2019] kthread+0x10d/0x140
[Fri Mar 8 04:38:21 2019] ? btrfs_cleanup_transaction+0x650/0x650
[Fri Mar 8 04:38:21 2019] ? kthread_create_on_node+0x40/0x40
[Fri Mar 8 04:38:21 2019] ret_from_fork+0x27/0x50
[Fri Mar 8 04:38:21 2019] hash_prefetch D 0 27397 27345 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? rwsem_down_write_failed+0x134/0x2b0
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] rwsem_down_write_failed+0x139/0x2b0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 8 04:38:21 2019] call_rwsem_down_write_failed+0x13/0x20
[Fri Mar 8 04:38:21 2019] down_write_nested+0x87/0xb0
[Fri Mar 8 04:38:21 2019] do_unlinkat+0x113/0x300
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fdcc7534ed7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fdbc2c4ce18 EFLAGS: 00000246 ORIG_RAX: 0000000000000107
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00007fdbc2c4ce40 RCX: 00007fdcc7534ed7
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000000 RSI: 00007fdbb8002590 RDI: 0000000000000003
[Fri Mar 8 04:38:21 2019] RBP: 00005570f731f1a8 R08: 0000000000000000 R09: 00007fdbc2c4cef0
[Fri Mar 8 04:38:21 2019] R10: 00005570f6d65c80 R11: 0000000000000246 R12: 00007fdbc2c4cf50
[Fri Mar 8 04:38:21 2019] R13: 00007fdbc2c4d170 R14: 00007fdbb800af60 R15: 00007fdbc2c4d120
[Fri Mar 8 04:38:21 2019] crawl_writeback D 0 27400 27345 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fdcc7533607
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fdbc1c4acc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00000000000e09c1 RCX: 00007fdcc7533607
[Fri Mar 8 04:38:21 2019] RDX: 00000000000e09c1 RSI: 00007fdba40008f0 RDI: 0000000000000003
[Fri Mar 8 04:38:21 2019] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007fdbc1c4b080
[Fri Mar 8 04:38:21 2019] R10: 0000000000000180 R11: 0000000000000293 R12: 00007fdba40008f0
[Fri Mar 8 04:38:21 2019] R13: 0000000000000003 R14: 00007fdbc1c4b3d0 R15: 00007fdbc1c4b350
[Fri Mar 8 04:38:21 2019] crawl_290 D 0 13174 27345 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 8 04:38:21 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 8 04:38:21 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fdcc6451ac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00007fdb9c5e4010 RCX: 00007fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RDX: 00007fdb9c5e4010 RSI: 00000000c0189436 RDI: 0000000000000465
[Fri Mar 8 04:38:21 2019] RBP: 00007fdcc6451e40 R08: 00007fdb9c1e8510 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007fdb9c044b80 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 8 04:38:21 2019] R13: 0000000000000018 R14: 00007fdb9c1e8528 R15: 00007fdcc6451e48
[Fri Mar 8 04:38:21 2019] crawl_302 D 0 15836 27345 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 8 04:38:21 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 8 04:38:21 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fdcc4c4f0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00007fdb98125c40 RCX: 00007fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RDX: 00007fdb98125c40 RSI: 00000000c0189436 RDI: 0000000000000027
[Fri Mar 8 04:38:21 2019] RBP: 00007fdcc4c4f420 R08: 00007fdb9807ee10 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007fdb98090b70 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 8 04:38:21 2019] R13: 0000000000000018 R14: 00007fdb9807ee28 R15: 00007fdcc4c4f428
[Fri Mar 8 04:38:21 2019] crawl_275 D 0 15837 27345 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 8 04:38:21 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 8 04:38:21 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fdcc3c4d0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00007fdba02014f0 RCX: 00007fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RDX: 00007fdba02014f0 RSI: 00000000c0189436 RDI: 0000000000000029
[Fri Mar 8 04:38:21 2019] RBP: 00007fdcc3c4d420 R08: 00007fdba0010cc0 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007fdba0088130 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 8 04:38:21 2019] R13: 0000000000000018 R14: 00007fdba0010cd8 R15: 00007fdcc3c4d428
[Fri Mar 8 04:38:21 2019] crawl_299 D 0 15838 27345 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 8 04:38:21 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 8 04:38:21 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fdcc444e0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00007fdb942937c0 RCX: 00007fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RDX: 00007fdb942937c0 RSI: 00000000c0189436 RDI: 00000000000004a0
[Fri Mar 8 04:38:21 2019] RBP: 00007fdcc444e420 R08: 00007fdb94094dd0 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007fdb94049c10 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 8 04:38:21 2019] R13: 0000000000000018 R14: 00007fdb94094de8 R15: 00007fdcc444e428
[Fri Mar 8 04:38:21 2019] crawl_290 D 0 15839 27345 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_clone+0x5a3/0xcc0
[Fri Mar 8 04:38:21 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Fri Mar 8 04:38:21 2019] vfs_dedupe_file_range+0x22e/0x280
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x24d/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fdcc5c510a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00007fdb8c4a6060 RCX: 00007fdcc7538dd7
[Fri Mar 8 04:38:21 2019] RDX: 00007fdb8c4a6060 RSI: 00000000c0189436 RDI: 00000000000004a1
[Fri Mar 8 04:38:21 2019] RBP: 00007fdcc5c51420 R08: 00007fdb8c060170 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007fdb8c07dd40 R11: 0000000000000246 R12: 0000000000000020
[Fri Mar 8 04:38:21 2019] R13: 0000000000000018 R14: 00007fdb8c060188 R15: 00007fdcc5c51428
[Fri Mar 8 04:38:21 2019] btrfs D 0 9795 9787 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_for_commit+0x3f/0x70
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 8 04:38:21 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 8 04:38:21 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 8 04:38:21 2019] ? wait_woken+0x70/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 8 04:38:21 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 04:38:21 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007f0ab8b6ed58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00005601803ae4f0 RCX: 00007f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RDX: 00005601803ae4f0 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 8 04:38:21 2019] RBP: 0000000000000000 R08: 00007f0ab8b6f700 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007f0ab8b6f700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 8 04:38:21 2019] R13: 00007ffd74e35cdf R14: 0000000000000000 R15: 00007f0ab9bec040
[Fri Mar 8 04:38:21 2019] btrfs D 0 9796 9787 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_for_commit+0x3f/0x70
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 8 04:38:21 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 8 04:38:21 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 8 04:38:21 2019] ? wait_woken+0x70/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 8 04:38:21 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 04:38:21 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007f0ab836dd58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00005601803ae970 RCX: 00007f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RDX: 00005601803ae970 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 8 04:38:21 2019] RBP: 0000000000000000 R08: 00007f0ab836e700 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007f0ab836e700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 8 04:38:21 2019] R13: 00007ffd74e35cdf R14: 0000000000000000 R15: 00007f0ab9bec040
[Fri Mar 8 04:38:21 2019] btrfs D 0 9797 9787 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_for_commit+0x3f/0x70
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 8 04:38:21 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 8 04:38:21 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 8 04:38:21 2019] ? wait_woken+0x70/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 8 04:38:21 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 04:38:21 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007f0ab7b6cd58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00005601803aedf0 RCX: 00007f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RDX: 00005601803aedf0 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 8 04:38:21 2019] RBP: 0000000000000000 R08: 00007f0ab7b6d700 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007f0ab7b6d700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 8 04:38:21 2019] R13: 00007ffd74e35cdf R14: 0000000000000000 R15: 00007f0ab9bec040
[Fri Mar 8 04:38:21 2019] btrfs D 0 9798 9787 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_for_commit+0x3f/0x70
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 8 04:38:21 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 8 04:38:21 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 8 04:38:21 2019] ? wait_woken+0x70/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 8 04:38:21 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 04:38:21 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007f0ab736bd58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00005601803af270 RCX: 00007f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RDX: 00005601803af270 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 8 04:38:21 2019] RBP: 0000000000000000 R08: 00007f0ab736c700 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007f0ab736c700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 8 04:38:21 2019] R13: 00007ffd74e35cdf R14: 0000000000000000 R15: 00007f0ab9bec040
[Fri Mar 8 04:38:21 2019] btrfs D 0 9799 9787 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_for_commit+0x3f/0x70
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_wait_for_commit+0xf2/0x130
[Fri Mar 8 04:38:21 2019] btrfs_inc_block_group_ro+0x52/0x170
[Fri Mar 8 04:38:21 2019] scrub_enumerate_chunks+0x191/0x610
[Fri Mar 8 04:38:21 2019] ? wait_woken+0x70/0xa0
[Fri Mar 8 04:38:21 2019] btrfs_scrub_dev+0x1fa/0x650
[Fri Mar 8 04:38:21 2019] ? mnt_want_write_file+0x3b/0xb0
[Fri Mar 8 04:38:21 2019] btrfs_ioctl+0x1342/0x26a0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] do_vfs_ioctl+0x92/0x6b0
[Fri Mar 8 04:38:21 2019] ? __fget+0x11f/0x210
[Fri Mar 8 04:38:21 2019] SyS_ioctl+0x74/0x80
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007f0ab6b6ad58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00005601803af6f0 RCX: 00007f0ab8c50dd7
[Fri Mar 8 04:38:21 2019] RDX: 00005601803af6f0 RSI: 00000000c400941b RDI: 0000000000000003
[Fri Mar 8 04:38:21 2019] RBP: 0000000000000000 R08: 00007f0ab6b6b700 R09: 0000000000000000
[Fri Mar 8 04:38:21 2019] R10: 00007f0ab6b6b700 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 8 04:38:21 2019] R13: 00007ffd74e35cdf R14: 0000000000000000 R15: 00007f0ab9bec040
[Fri Mar 8 04:38:21 2019] rsync D 0 12329 12200 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f37f89b34b0
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffdd3371078 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f37f89b34b0
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffdd33732e0
[Fri Mar 8 04:38:21 2019] RBP: 000000000003a2f8 R08: 000000000000001e R09: 00007f37f8a06120
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffdd33732f6
[Fri Mar 8 04:38:21 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f37f8a41320
[Fri Mar 8 04:38:21 2019] rsync D 0 15392 15306 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7ff710ce34b0
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffc267318a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff710ce34b0
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc26733b10
[Fri Mar 8 04:38:21 2019] RBP: 000000000003a2f8 R08: 0000000000000000 R09: 00007ff710d35b30
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc26733b3f
[Fri Mar 8 04:38:21 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007ff710d71320
[Fri Mar 8 04:38:21 2019] rsync D 0 24322 12266 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_setattr+0x1e6/0x3b0
[Fri Mar 8 04:38:21 2019] notify_change+0x2ed/0x450
[Fri Mar 8 04:38:21 2019] do_truncate+0x5e/0x90
[Fri Mar 8 04:38:21 2019] ? rcu_read_lock_sched_held+0x68/0x70
[Fri Mar 8 04:38:21 2019] do_sys_ftruncate.constprop.1+0x120/0x180
[Fri Mar 8 04:38:21 2019] ? trace_hardirqs_off_thunk+0x1a/0x1c
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f2994512c87
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fff6ffc5be8 EFLAGS: 00000246 ORIG_RAX: 000000000000004d
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2994512c87
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000000 RSI: 0000000004648f62 RDI: 0000000000000004
[Fri Mar 8 04:38:21 2019] RBP: 0000000000000004 R08: 0000000000000000 R09: 00007f299455d7b8
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000004648f62
[Fri Mar 8 04:38:21 2019] R13: 00000000000008a1 R14: 0000000000001f78 R15: 0000000000000005
[Fri Mar 8 04:38:21 2019] rsync D 0 25113 12114 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] __btrfs_setxattr+0x61/0x110
[Fri Mar 8 04:38:21 2019] __vfs_setxattr+0x66/0x80
[Fri Mar 8 04:38:21 2019] __vfs_setxattr_noperm+0x6d/0x1b0
[Fri Mar 8 04:38:21 2019] vfs_setxattr+0xa0/0xb0
[Fri Mar 8 04:38:21 2019] setxattr+0x131/0x1a0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] ? rcu_read_lock_sched_held+0x68/0x70
[Fri Mar 8 04:38:21 2019] ? rcu_sync_lockdep_assert+0x30/0x60
[Fri Mar 8 04:38:21 2019] ? __sb_start_write+0xcc/0x1b0
[Fri Mar 8 04:38:21 2019] ? mnt_want_write+0x20/0x50
[Fri Mar 8 04:38:21 2019] path_setxattr+0x94/0xc0
[Fri Mar 8 04:38:21 2019] SyS_lsetxattr+0xd/0x10
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f1934b64229
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffc0ca30b38 EFLAGS: 00000246 ORIG_RAX: 00000000000000bd
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1934b64229
[Fri Mar 8 04:38:21 2019] RDX: 00007ffc0ca30c70 RSI: 000055cb645696dd RDI: 00007ffc0ca33050
[Fri Mar 8 04:38:21 2019] RBP: 00007ffc0ca30c70 R08: 0000000000000000 R09: 00000000000003ea
[Fri Mar 8 04:38:21 2019] R10: 0000000000000014 R11: 0000000000000246 R12: 0000000000000000
[Fri Mar 8 04:38:21 2019] R13: 00007ffc0ca33050 R14: 00000000000081a4 R15: 00007f19355af7ec
[Fri Mar 8 04:38:21 2019] rsync D 0 14184 14076 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_start_transaction_fallback_global_rsv+0x34/0x200
[Fri Mar 8 04:38:21 2019] btrfs_unlink+0x30/0xd0
[Fri Mar 8 04:38:21 2019] vfs_unlink+0xf8/0x190
[Fri Mar 8 04:38:21 2019] do_unlinkat+0x287/0x300
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fc9a562cea7
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffd3cbd9a18 EFLAGS: 00000246 ORIG_RAX: 0000000000000057
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fc9a562cea7
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000000 RSI: 00000000000081a4 RDI: 00007ffd3cbdaad0
[Fri Mar 8 04:38:21 2019] RBP: 00000000000081a4 R08: 000000000000fff0 R09: 00007fc9a567d770
[Fri Mar 8 04:38:21 2019] R10: 632e766c6f736572 R11: 0000000000000246 R12: 00007ffd3cbdaad0
[Fri Mar 8 04:38:21 2019] R13: 0000000000008000 R14: 0000000000000000 R15: 00000000000081a4
[Fri Mar 8 04:38:21 2019] kworker/u8:20 D 0 20623 2 0x80000000
[Fri Mar 8 04:38:21 2019] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] io_schedule+0x12/0x40
[Fri Mar 8 04:38:21 2019] __lock_page+0x110/0x140
[Fri Mar 8 04:38:21 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Fri Mar 8 04:38:21 2019] extent_write_cache_pages.constprop.27+0x2d9/0x3b0
[Fri Mar 8 04:38:21 2019] ? lock_acquire+0xbc/0x200
[Fri Mar 8 04:38:21 2019] extent_writepages+0x4e/0x70
[Fri Mar 8 04:38:21 2019] ? btrfs_direct_IO+0x3e0/0x3e0
[Fri Mar 8 04:38:21 2019] do_writepages+0x23/0x80
[Fri Mar 8 04:38:21 2019] __filemap_fdatawrite_range+0xaa/0xe0
[Fri Mar 8 04:38:21 2019] btrfs_run_delalloc_work+0x17/0x60
[Fri Mar 8 04:38:21 2019] btrfs_scrubparity_helper+0x8f/0x640
[Fri Mar 8 04:38:21 2019] process_one_work+0x204/0x650
[Fri Mar 8 04:38:21 2019] worker_thread+0x4d/0x3b0
[Fri Mar 8 04:38:21 2019] kthread+0x10d/0x140
[Fri Mar 8 04:38:21 2019] ? process_one_work+0x650/0x650
[Fri Mar 8 04:38:21 2019] ? kthread_create_on_node+0x40/0x40
[Fri Mar 8 04:38:21 2019] ret_from_fork+0x27/0x50
[Fri Mar 8 04:38:21 2019] mkdir D 0 17020 17013 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_mkdir+0x50/0x1d0
[Fri Mar 8 04:38:21 2019] vfs_mkdir+0x134/0x1c0
[Fri Mar 8 04:38:21 2019] SyS_mkdirat+0xb3/0xf0
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fed308fb447
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffcfc5df778 EFLAGS: 00000206 ORIG_RAX: 0000000000000053
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 00007ffcfc5e1b28 RCX: 00007fed308fb447
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007ffcfc5e1b4d
[Fri Mar 8 04:38:21 2019] RBP: 00007ffcfc5df8f0 R08: 00000000000001ff R09: 000056093bb17e10
[Fri Mar 8 04:38:21 2019] R10: 0000000000000376 R11: 0000000000000206 R12: 000056093bb17e50
[Fri Mar 8 04:38:21 2019] R13: 00000000000001ff R14: 00007ffcfc5dfa90 R15: 0000000000000001
[Fri Mar 8 04:38:21 2019] rsync D 0 2327 2261 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f084cebb4b0
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffea5c6f238 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f084cebb4b0
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffea5c714a0
[Fri Mar 8 04:38:21 2019] RBP: 000000000003a2f8 R08: 000000000000f000 R09: 00007f084cf0e300
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffea5c714e6
[Fri Mar 8 04:38:21 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f084cf49320
[Fri Mar 8 04:38:21 2019] rsync D 0 6002 6001 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f8e8d0b34b0
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffeaa16f288 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f8e8d0b34b0
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffeaa1714f0
[Fri Mar 8 04:38:21 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f8e8d105950
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffeaa171507
[Fri Mar 8 04:38:21 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f8e8d141320
[Fri Mar 8 04:38:21 2019] rsync D 0 24698 24688 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f062ff5b4b0
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffce5979488 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f062ff5b4b0
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffce597b6f0
[Fri Mar 8 04:38:21 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f062ffad950
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffce597b707
[Fri Mar 8 04:38:21 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f062ffe9320
[Fri Mar 8 04:38:21 2019] rsync D 0 10903 10902 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7fe4adc8b4b0
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007fff53e747a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe4adc8b4b0
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007fff53e76a10
[Fri Mar 8 04:38:21 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007fe4adcddf10
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff53e76a23
[Fri Mar 8 04:38:21 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fe4add19320
[Fri Mar 8 04:38:21 2019] rsync D 0 14079 13983 0x00000000
[Fri Mar 8 04:38:21 2019] Call Trace:
[Fri Mar 8 04:38:21 2019] ? __schedule+0x429/0xbb0
[Fri Mar 8 04:38:21 2019] ? prepare_to_wait_event+0x97/0x190
[Fri Mar 8 04:38:21 2019] schedule+0x39/0x90
[Fri Mar 8 04:38:21 2019] wait_current_trans+0xa3/0xf0
[Fri Mar 8 04:38:21 2019] ? wait_woken+0xa0/0xa0
[Fri Mar 8 04:38:21 2019] start_transaction+0x29b/0x4d0
[Fri Mar 8 04:38:21 2019] btrfs_create+0x47/0x1d0
[Fri Mar 8 04:38:21 2019] lookup_open+0x404/0x760
[Fri Mar 8 04:38:21 2019] ? __wake_up_common_lock+0x53/0x90
[Fri Mar 8 04:38:21 2019] path_openat+0x4e9/0xcc0
[Fri Mar 8 04:38:21 2019] ? filemap_map_pages+0x312/0x520
[Fri Mar 8 04:38:21 2019] do_filp_open+0x8a/0xf0
[Fri Mar 8 04:38:21 2019] ? __alloc_fd+0xee/0x200
[Fri Mar 8 04:38:21 2019] ? do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_sys_open+0x130/0x200
[Fri Mar 8 04:38:21 2019] do_syscall_64+0x76/0x180
[Fri Mar 8 04:38:21 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Fri Mar 8 04:38:21 2019] RIP: 0033:0x7f4bb133b4b0
[Fri Mar 8 04:38:21 2019] RSP: 002b:00007ffe9a1d4a48 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Fri Mar 8 04:38:21 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4bb133b4b0
[Fri Mar 8 04:38:21 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffe9a1d6cb0
[Fri Mar 8 04:38:21 2019] RBP: 000000000003a2f8 R08: 000000000000fff8 R09: 00007f4bb138df10
[Fri Mar 8 04:38:21 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe9a1d6ccd
[Fri Mar 8 04:38:21 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f4bb13c9320
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: A collection of btrfs lockup stack traces (4.20.13)
2019-03-20 3:39 A collection of btrfs lockup stack traces Zygo Blaxell
2019-03-20 3:45 ` Zygo Blaxell
@ 2019-03-20 3:49 ` Zygo Blaxell
2020-02-04 3:34 ` A collection of btrfs lockup stack traces (4.14..4.20.17, but not later kernels) Zygo Blaxell
2019-03-27 2:37 ` A collection of btrfs lockup stack traces (4.14.106, no __sb_start_write) Zygo Blaxell
2 siblings, 1 reply; 6+ messages in thread
From: Zygo Blaxell @ 2019-03-20 3:49 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 20935 bytes --]
On Tue, Mar 19, 2019 at 11:39:59PM -0400, Zygo Blaxell wrote:
> I haven't been able to easily reproduce these in a test environment;
> however, they have been happening several times a year on servers in
> production.
>
> Kernel: most recent observation on 4.14.105 + cherry-picked deadlock
> and misc hang fixes:
>
> btrfs: wakeup cleaner thread when adding delayed iput
> Btrfs: fix deadlock when allocating tree block during leaf/node split
> Btrfs: use nofs context when initializing security xattrs to avoid deadlock
> Btrfs: fix deadlock with memory reclaim during scrub
> Btrfs: fix deadlock between clone/dedupe and rename
>
> Also observed on 4.20.13, and 4.14.0..4.14.105 (4.14.106 is currently
> running, but hasn't locked up yet).
>
> Filesystem mount flags: compress=zstd,ssd,flushoncommit,space_cache=v2.
> Configuration is either -draid1/-mraid1 or -dsingle/-mraid1. I've
> also reproduced a lockup without flushoncommit.
>
> The machines that are locking up all run the same workload:
>
> rsync receiving data continuously (gigabytes aren't enough,
> I can barely reproduce this once a month with 2TB of rsync
> traffic from 10 simulated clients)
>
> bees doing continuous dedupe
>
> snapshots daily and after each rsync
>
> snapshot deletes as required to maintain free space
>
> scrubs twice monthly plus after each crash
>
> watchdog does a 'mkdir foo; rmdir foo' every few seconds.
> If this takes longer than 50 minutes, collect a stack trace;
> longer than 60 minutes, reboot the machine.
>
> Observations:
>
> In the past I've hit the known deadlocks between rsync
> (rename) and dedupe, and between snapshot delete and scrub.
> These deadlocks aren't those, and current kernels can still
> deadlock with fixes for those (and some others) applied.
>
> Deadlocks can occur when there are no snapshot deletes active
> and scrub is not running. In a few cases there's been a snapshot
> create active during a deadlock event.
>
> The deadlocks occur in bursts, which correlate to periods
> of rapid data expansion on the filesystem. e.g. after a big
> snapshot delete frees a TB or two of space, there are several
> lockups a few days (or even hours) apart. When the filesystem
> gets near the free space threshold and a series of smaller
> snapshot deletes occur, so that allocated space stays relatively
> stable, then there can be months between lockups.
>
> Other bugs in 4.14 kernels break lockdep soon after boot, so
> there's only lockdep information in a few cases by the time the
> btrfs deadlock happens.
>
> There seem to be a few different identifiable behaviors, e.g.
> in some cases the 50-minute timer expires (so a stack trace is
> collected) but the 60-minute timer doesn't expire (so the machine
> got itself out of the lockup somehow, and doesn't reboot).
>
> Distractions:
>
> In some cases increasing vm.dirty_ratio will unfreeze the
> filesystem, in other cases it does nothing. Decreasing
> vm.dirty_ratio (e.g. to 2) locks up the filesystem almost
> immediately, but it can recover if vm.dirty_ratio is increased
> again. I'm not sure if this is because there are two separate
> bugs, or just a single bug with corner cases.
>
Here's one on 4.20.13. Alas, no lockdep output in this one. I note
that there is both rename and dedupe in here, but they don't seem to be
deadlocked with each other (that case would have lock_two_nondirectories
in the stack, and that isn't present here).
[573870.943159] sysrq: SysRq : Show Blocked State
[573870.948005] task PC stack pid father
[573870.954068] btrfs-transacti D 0 4481 2 0x80000000
[573870.960142] Call Trace:
[573870.963075] ? __schedule+0x3cf/0xb60
[573870.967334] schedule+0x39/0x90
[573870.971170] btrfs_commit_transaction+0x2a7/0x990
[573870.976658] ? wait_woken+0xa0/0xa0
[573870.980761] transaction_kthread+0x149/0x180
[573870.985725] kthread+0x106/0x140
[573870.989529] ? btrfs_cleanup_transaction+0x620/0x620
[573870.995281] ? kthread_park+0x90/0x90
[573870.999734] ret_from_fork+0x3a/0x50
[573871.003985] crawl_5 D 0 4547 4537 0x00000000
[573871.010913] Call Trace:
[573871.014051] ? __schedule+0x3cf/0xb60
[573871.018543] schedule+0x39/0x90
[573871.022735] wait_current_trans+0xc3/0x100
[573871.027895] ? wait_woken+0xa0/0xa0
[573871.032218] start_transaction+0x13f/0x460
[573871.035587] btrfs_clone+0x5cd/0xcf0
[573871.037992] ? do_raw_spin_lock+0xa0/0xb0
[573871.041436] btrfs_extent_same_range+0x380/0x440
[573871.045232] btrfs_remap_file_range+0x2d8/0x300
[573871.049143] vfs_dedupe_file_range_one+0x10c/0x160
[573871.053176] vfs_dedupe_file_range+0x14c/0x1e0
[573871.056805] do_vfs_ioctl+0x278/0x6f0
[573871.059844] ? __fget+0x11f/0x210
[573871.062627] ksys_ioctl+0x70/0x80
[573871.064115] __x64_sys_ioctl+0x16/0x20
[573871.065819] do_syscall_64+0x60/0x190
[573871.068437] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.071756] RIP: 0033:0x7fd5eda85017
[573871.072514] Code: Bad RIP value.
[573871.074543] RSP: 002b:00007fd5ed19e998 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[573871.079062] RAX: ffffffffffffffda RBX: 00007fd5880c2070 RCX: 00007fd5eda85017
[573871.084890] RDX: 00007fd5880c2070 RSI: 00000000c0189436 RDI: 0000000000000249
[573871.090847] RBP: 00007fd5ed19ed20 R08: 00007fd5885a9f20 R09: 0000000000000000
[573871.096633] R10: 00007fd5ed19ef40 R11: 0000000000000246 R12: 0000000000000001
[573871.102273] R13: 0000000000000018 R14: 00007fd5885a9f38 R15: 00007fd5ed19ed28
[573871.107906] hash_writeback D 0 4551 4537 0x00000000
[573871.112175] Call Trace:
[573871.114219] ? __schedule+0x3cf/0xb60
[573871.117138] ? _raw_spin_unlock_irqrestore+0x32/0x60
[573871.121025] schedule+0x39/0x90
[573871.123576] schedule_timeout+0x1de/0x4e0
[573871.126728] ? collect_expired_timers+0xb0/0xb0
[573871.130113] io_schedule_timeout+0x19/0x40
[573871.133246] balance_dirty_pages+0x354/0xeb0
[573871.136407] ? lock_acquire+0xbc/0x1c0
[573871.139268] ? kvm_sched_clock_read+0x14/0x30
[573871.142567] balance_dirty_pages_ratelimited+0x5e0/0xae0
[573871.146543] btrfs_buffered_write.isra.4+0x40a/0x7b0
[573871.150250] btrfs_file_write_iter+0x224/0x5c3
[573871.153632] __vfs_write+0x113/0x1a0
[573871.156478] vfs_write+0xc5/0x1b0
[573871.159245] ksys_pwrite64+0x74/0x90
[573871.162194] do_syscall_64+0x60/0x190
[573871.165247] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.169292] RIP: 0033:0x7fd5ee5f0983
[573871.172240] Code: Bad RIP value.
[573871.174953] RSP: 002b:00007fd5e6ffc390 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
[573871.180664] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007fd5ee5f0983
[573871.186068] RDX: 0000000000020000 RSI: 00007fd59c000d80 RDI: 0000000000000005
[573871.191469] RBP: 000000001f7e0000 R08: 00007fd59c000d80 R09: 0000000000020010
[573871.196794] R10: 000000001f7e0000 R11: 0000000000000293 R12: 00007fd5e6ffc5c0
[573871.202069] R13: 0000000000020000 R14: 00007fd5bf7e0000 R15: 00007fd59c020d80
[573871.207279] hash_prefetch D 0 4552 4537 0x00000000
[573871.211333] Call Trace:
[573871.213305] ? __schedule+0x3cf/0xb60
[573871.216119] ? mark_held_locks+0x6a/0x90
[573871.219125] ? rwsem_down_write_failed+0x165/0x300
[573871.222752] schedule+0x39/0x90
[573871.225294] rwsem_down_write_failed+0x16a/0x300
[573871.228933] ? call_rwsem_down_write_failed+0x13/0x20
[573871.232925] call_rwsem_down_write_failed+0x13/0x20
[573871.236742] down_write_nested+0x87/0xb0
[573871.239851] do_unlinkat+0x130/0x310
[573871.242747] do_syscall_64+0x60/0x190
[573871.245683] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.249591] RIP: 0033:0x7fd5eda81117
[573871.252454] Code: Bad RIP value.
[573871.255045] RSP: 002b:00007fd5e67fae18 EFLAGS: 00000246 ORIG_RAX: 0000000000000107
[573871.261002] RAX: ffffffffffffffda RBX: 00007fd5e67fae40 RCX: 00007fd5eda81117
[573871.266855] RDX: 0000000000000000 RSI: 00007fd598003d70 RDI: 0000000000000003
[573871.272742] RBP: 0000563346f72f78 R08: 0000000000000000 R09: 00007fd5e67faef0
[573871.274377] R10: 000056334558ec60 R11: 0000000000000246 R12: 00007fd5e67faf50
[573871.280664] R13: 00007fd5e67fb170 R14: 00007fd59800b4b0 R15: 00007fd5e67fb120
[573871.286681] crawl_writeback D 0 4554 4537 0x00000000
[573871.291274] Call Trace:
[573871.293418] ? __schedule+0x3cf/0xb60
[573871.296478] schedule+0x39/0x90
[573871.299132] wait_current_trans+0xc3/0x100
[573871.302523] ? wait_woken+0xa0/0xa0
[573871.305539] start_transaction+0x13f/0x460
[573871.308956] btrfs_create+0x58/0x1f0
[573871.311985] lookup_open+0x451/0x770
[573871.315006] path_openat+0x435/0xbb0
[573871.318000] do_filp_open+0x99/0x110
[573871.321097] ? __alloc_fd+0xee/0x200
[573871.324301] ? do_raw_spin_unlock+0x49/0xc0
[573871.328074] ? do_sys_open+0x14a/0x210
[573871.331430] do_sys_open+0x14a/0x210
[573871.334633] do_syscall_64+0x60/0x190
[573871.337905] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.342230] RIP: 0033:0x7fd5eda7f847
[573871.345333] Code: Bad RIP value.
[573871.348248] RSP: 002b:00007fd5e499fcc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[573871.354668] RAX: ffffffffffffffda RBX: 00000000000e09c1 RCX: 00007fd5eda7f847
[573871.360572] RDX: 00000000000e09c1 RSI: 00007fd594000b20 RDI: 0000000000000003
[573871.366456] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007fd5e49a0080
[573871.372160] R10: 0000000000000180 R11: 0000000000000293 R12: 00007fd594000b20
[573871.377834] R13: 0000000000000003 R14: 00007fd5e49a03d0 R15: 00007fd5e49a0350
[573871.383707] kworker/u8:0 D 0 27279 2 0x80000000
[573871.388290] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[573871.393359] Call Trace:
[573871.395574] ? __schedule+0x3cf/0xb60
[573871.398743] schedule+0x39/0x90
[573871.401438] wait_current_trans+0xc3/0x100
[573871.404926] ? wait_woken+0xa0/0xa0
[573871.407881] start_transaction+0x21a/0x460
[573871.411309] btrfs_finish_ordered_io+0x492/0x7e0
[573871.415121] normal_work_helper+0x95/0x530
[573871.418555] process_one_work+0x223/0x5d0
[573871.421998] worker_thread+0x4f/0x3b0
[573871.425198] kthread+0x106/0x140
[573871.428121] ? process_one_work+0x5d0/0x5d0
[573871.431825] ? kthread_park+0x90/0x90
[573871.435066] ret_from_fork+0x3a/0x50
[573871.438181] kworker/u8:14 D 0 15161 2 0x80000000
[573871.442785] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[573871.447777] Call Trace:
[573871.449916] ? __schedule+0x3cf/0xb60
[573871.452971] schedule+0x39/0x90
[573871.455716] wait_current_trans+0xc3/0x100
[573871.459181] ? wait_woken+0xa0/0xa0
[573871.462174] start_transaction+0x21a/0x460
[573871.465633] btrfs_finish_ordered_io+0x492/0x7e0
[573871.469430] normal_work_helper+0x95/0x530
[573871.472784] process_one_work+0x223/0x5d0
[573871.476078] worker_thread+0x4f/0x3b0
[573871.479116] kthread+0x106/0x140
[573871.481819] ? process_one_work+0x5d0/0x5d0
[573871.485236] ? kthread_park+0x90/0x90
[573871.488292] ret_from_fork+0x3a/0x50
[573871.491453] kworker/u8:17 D 0 23801 2 0x80000000
[573871.496239] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper
[573871.501882] Call Trace:
[573871.504102] ? __schedule+0x3cf/0xb60
[573871.507305] schedule+0x39/0x90
[573871.510091] btrfs_start_ordered_extent+0x10a/0x1b0
[573871.514178] ? wait_woken+0xa0/0xa0
[573871.517248] btrfs_run_ordered_extent_work+0x21/0x30
[573871.519956] normal_work_helper+0x95/0x530
[573871.521156] process_one_work+0x223/0x5d0
[573871.525102] worker_thread+0x4f/0x3b0
[573871.527891] kthread+0x106/0x140
[573871.531303] ? process_one_work+0x5d0/0x5d0
[573871.534825] ? kthread_park+0x90/0x90
[573871.537987] ret_from_fork+0x3a/0x50
[573871.541073] kworker/u8:10 D 0 4421 2 0x80000000
[573871.546477] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[573871.552581] Call Trace:
[573871.554839] ? __schedule+0x3cf/0xb60
[573871.558046] schedule+0x39/0x90
[573871.560804] wait_current_trans+0xc3/0x100
[573871.564327] ? wait_woken+0xa0/0xa0
[573871.568464] start_transaction+0x21a/0x460
[573871.572087] btrfs_finish_ordered_io+0x492/0x7e0
[573871.576026] normal_work_helper+0x95/0x530
[573871.580231] process_one_work+0x223/0x5d0
[573871.583564] worker_thread+0x4f/0x3b0
[573871.586649] kthread+0x106/0x140
[573871.589343] ? process_one_work+0x5d0/0x5d0
[573871.592762] ? kthread_park+0x90/0x90
[573871.595837] ret_from_fork+0x3a/0x50
[573871.598912] kworker/u8:1 D 0 17773 2 0x80000000
[573871.603488] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[573871.608535] Call Trace:
[573871.610722] ? __schedule+0x3cf/0xb60
[573871.613924] schedule+0x39/0x90
[573871.616706] wait_current_trans+0xc3/0x100
[573871.620241] ? wait_woken+0xa0/0xa0
[573871.623248] start_transaction+0x21a/0x460
[573871.626734] btrfs_finish_ordered_io+0x492/0x7e0
[573871.630627] normal_work_helper+0x95/0x530
[573871.634232] process_one_work+0x223/0x5d0
[573871.637772] worker_thread+0x4f/0x3b0
[573871.641029] kthread+0x106/0x140
[573871.643825] ? process_one_work+0x5d0/0x5d0
[573871.647422] ? kthread_park+0x90/0x90
[573871.648344] ret_from_fork+0x3a/0x50
[573871.649126] rsync D 0 3068 3054 0x00000000
[573871.650256] Call Trace:
[573871.650789] ? __schedule+0x3cf/0xb60
[573871.651589] schedule+0x39/0x90
[573871.652283] wait_current_trans+0xc3/0x100
[573871.653191] ? wait_woken+0xa0/0xa0
[573871.654236] start_transaction+0x21a/0x460
[573871.655154] btrfs_evict_inode+0x47d/0x520
[573871.656043] ? dput+0x25/0x2e0
[573871.656726] evict+0xbf/0x1c0
[573871.657394] __dentry_kill+0xca/0x1b0
[573871.658198] dentry_kill+0x50/0x1d0
[573871.658929] ? dput+0x25/0x2e0
[573871.659588] dput+0x286/0x2e0
[573871.660224] do_renameat2+0x3fc/0x570
[573871.661000] __x64_sys_rename+0x1c/0x20
[573871.661806] do_syscall_64+0x60/0x190
[573871.662741] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.663971] RIP: 0033:0x7f539eaa4d47
[573871.664867] Code: Bad RIP value.
[573871.665663] RSP: 002b:00007ffc71486c68 EFLAGS: 00000246 ORIG_RAX: 0000000000000052
[573871.667489] RAX: ffffffffffffffda RBX: 00007ffc71488e30 RCX: 00007f539eaa4d47
[573871.669223] RDX: 0000000000000000 RSI: 00007ffc71486e30 RDI: 00007ffc71488e30
[573871.671126] RBP: 00007ffc71486e30 R08: 0000000000000000 R09: 0000000000000001
[573871.672985] R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000000000
[573871.674899] R13: 0000000000000004 R14: 0000000000008180 R15: 0000000000000001
[573871.676784] rsync D 0 9157 9148 0x00000000
[573871.678162] Call Trace:
[573871.678789] ? __schedule+0x3cf/0xb60
[573871.679835] schedule+0x39/0x90
[573871.680703] wait_current_trans+0xc3/0x100
[573871.681829] ? wait_woken+0xa0/0xa0
[573871.682690] start_transaction+0x21a/0x460
[573871.683789] btrfs_dirty_inode+0x42/0xd0
[573871.684748] btrfs_setattr+0xab/0x590
[573871.685693] ? current_time+0x4d/0x90
[573871.686610] notify_change+0x2f0/0x450
[573871.687603] chown_common+0x1ac/0x1d0
[573871.688485] ? rcu_read_lock_sched_held+0x68/0x70
[573871.689743] do_fchownat+0xbc/0xf0
[573871.690650] __x64_sys_lchown+0x21/0x30
[573871.691594] do_syscall_64+0x60/0x190
[573871.692460] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.693756] RIP: 0033:0x7f8cc9ac29b7
[573871.694648] Code: Bad RIP value.
[573871.695452] RSP: 002b:00007ffd2bdbd5a8 EFLAGS: 00000246 ORIG_RAX: 000000000000005e
[573871.697377] RAX: ffffffffffffffda RBX: 00000000000041c0 RCX: 00007f8cc9ac29b7
[573871.699268] RDX: 00000000000003e8 RSI: 00000000000003e8 RDI: 00007ffd2bdbfa10
[573871.701174] RBP: 00007ffd2bdbfa10 R08: 0000000000000000 R09: 0000000000000000
[573871.702972] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd2bdbd8e0
[573871.704864] R13: 00000000000003e8 R14: 0000558019715758 R15: 0000000000000001
[573871.706837] rsync D 0 10148 10138 0x00000000
[573871.708286] Call Trace:
[573871.708921] ? __schedule+0x3cf/0xb60
[573871.710017] schedule+0x39/0x90
[573871.710865] wait_current_trans+0xc3/0x100
[573871.711956] ? wait_woken+0xa0/0xa0
[573871.712942] start_transaction+0x13f/0x460
[573871.714018] btrfs_link+0xb1/0x280
[573871.714899] vfs_link+0x23e/0x310
[573871.715746] do_linkat+0x296/0x340
[573871.716742] __x64_sys_link+0x1c/0x20
[573871.717664] do_syscall_64+0x60/0x190
[573871.718691] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.720020] RIP: 0033:0x7f0ad9ccbfc7
[573871.720964] Code: Bad RIP value.
[573871.721808] RSP: 002b:00007ffe17e262a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000056
[573871.723825] RAX: ffffffffffffffda RBX: 0000561a33fcd098 RCX: 00007f0ad9ccbfc7
[573871.725767] RDX: 0000000000000000 RSI: 00007ffe17e262f0 RDI: 0000561a303452c0
[573871.727529] RBP: 0000561a303452c0 R08: 0000000000000000 R09: 00007ffe17e2a710
[573871.729252] R10: 0000000000000064 R11: 0000000000000246 R12: 00007ffe17e262f0
[573871.731148] R13: 0000000000000000 R14: 0000561a303452c0 R15: 0000561a33fcd098
[573871.732946] kworker/u8:5 D 0 1064 2 0x80000000
[573871.734371] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[573871.735956] Call Trace:
[573871.736625] ? __schedule+0x3cf/0xb60
[573871.737644] schedule+0x39/0x90
[573871.738424] wait_current_trans+0xc3/0x100
[573871.739520] ? wait_woken+0xa0/0xa0
[573871.740416] start_transaction+0x21a/0x460
[573871.741506] btrfs_finish_ordered_io+0x492/0x7e0
[573871.742726] normal_work_helper+0x95/0x530
[573871.743747] process_one_work+0x223/0x5d0
[573871.744868] worker_thread+0x4f/0x3b0
[573871.745812] kthread+0x106/0x140
[573871.746603] ? process_one_work+0x5d0/0x5d0
[573871.747753] ? kthread_park+0x90/0x90
[573871.748751] ret_from_fork+0x3a/0x50
[573871.749713] kworker/u8:9 D 0 6027 2 0x80000000
[573871.751135] Workqueue: btrfs-endio-write btrfs_endio_write_helper
[573871.752691] Call Trace:
[573871.753233] ? __schedule+0x3cf/0xb60
[573871.754177] schedule+0x39/0x90
[573871.754957] wait_current_trans+0xc3/0x100
[573871.755864] ? wait_woken+0xa0/0xa0
[573871.756777] start_transaction+0x21a/0x460
[573871.757843] btrfs_finish_ordered_io+0x492/0x7e0
[573871.759189] normal_work_helper+0x95/0x530
[573871.760342] process_one_work+0x223/0x5d0
[573871.761526] worker_thread+0x4f/0x3b0
[573871.762559] kthread+0x106/0x140
[573871.763463] ? process_one_work+0x5d0/0x5d0
[573871.764592] ? kthread_park+0x90/0x90
[573871.765657] ret_from_fork+0x3a/0x50
[573871.766674] btrfs D 0 19926 4537 0x00000000
[573871.768208] Call Trace:
[573871.768956] ? __schedule+0x3cf/0xb60
[573871.770052] schedule+0x39/0x90
[573871.770905] schedule_timeout+0x228/0x4e0
[573871.771996] ? mark_held_locks+0x6a/0x90
[573871.772961] ? _raw_spin_unlock_irq+0x29/0x40
[573871.774157] wait_for_common+0x155/0x220
[573871.775262] ? wake_up_q+0x70/0x70
[573871.776133] btrfs_wait_ordered_extents+0x27c/0x400
[573871.777477] btrfs_mksubvol+0x3cd/0x5e0
[573871.778450] ? rcu_read_lock_sched_held+0x68/0x70
[573871.779637] ? rcu_sync_lockdep_assert+0x30/0x60
[573871.780829] ? __sb_start_write+0xcc/0x1b0
[573871.781831] ? mnt_want_write_file+0x24/0x50
[573871.783016] btrfs_ioctl_snap_create_transid+0x169/0x1a0
[573871.784449] btrfs_ioctl_snap_create_v2+0x11d/0x170
[573871.785716] btrfs_ioctl+0x1351/0x2cb0
[573871.786692] ? kvm_sched_clock_read+0x14/0x30
[573871.787778] ? __handle_mm_fault+0x1067/0x1810
[573871.788844] ? do_raw_spin_unlock+0x49/0xc0
[573871.789842] ? _raw_spin_unlock+0x24/0x30
[573871.790882] ? __handle_mm_fault+0x1067/0x1810
[573871.791917] ? do_vfs_ioctl+0xa2/0x6f0
[573871.792901] ? btrfs_ioctl_get_supported_features+0x30/0x30
[573871.794404] do_vfs_ioctl+0xa2/0x6f0
[573871.795263] ? __do_page_fault+0x1fa/0x5a0
[573871.796289] ? entry_SYSCALL_64_after_hwframe+0x3e/0xbe
[573871.797557] ksys_ioctl+0x70/0x80
[573871.798431] __x64_sys_ioctl+0x16/0x20
[573871.799384] do_syscall_64+0x60/0x190
[573871.800254] entry_SYSCALL_64_after_hwframe+0x49/0xbe
[573871.801407] RIP: 0033:0x7f4657246017
[573871.802177] Code: Bad RIP value.
[573871.802975] RSP: 002b:00007fffb9cdb928 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
[573871.804897] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4657246017
[573871.806659] RDX: 00007fffb9cdb968 RSI: 0000000050009417 RDI: 0000000000000003
[573871.808372] RBP: 00000000013e1010 R08: 0000000000000008 R09: 0000000000000048
[573871.810129] R10: 0000000000000541 R11: 0000000000000206 R12: 00000000013e1040
[573871.811974] R13: 00000000013e1040 R14: 0000000000000004 R15: 0000000000000003
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: A collection of btrfs lockup stack traces (4.14.106, no __sb_start_write)
2019-03-20 3:39 A collection of btrfs lockup stack traces Zygo Blaxell
2019-03-20 3:45 ` Zygo Blaxell
2019-03-20 3:49 ` A collection of btrfs lockup stack traces (4.20.13) Zygo Blaxell
@ 2019-03-27 2:37 ` Zygo Blaxell
2019-03-27 20:04 ` A collection of btrfs lockup stack traces (4.14.107, kcompactd and two processes running __lock_page) Zygo Blaxell
2 siblings, 1 reply; 6+ messages in thread
From: Zygo Blaxell @ 2019-03-27 2:37 UTC (permalink / raw)
To: linux-btrfs
On Tue, Mar 19, 2019 at 11:39:59PM -0400, Zygo Blaxell wrote:
> I haven't been able to easily reproduce these in a test environment;
> however, they have been happening several times a year on servers in
> production.
>
> Kernel: most recent observation on 4.14.105 + cherry-picked deadlock
> and misc hang fixes:
>
> btrfs: wakeup cleaner thread when adding delayed iput
> Btrfs: fix deadlock when allocating tree block during leaf/node split
> Btrfs: use nofs context when initializing security xattrs to avoid deadlock
> Btrfs: fix deadlock with memory reclaim during scrub
> Btrfs: fix deadlock between clone/dedupe and rename
>
> Also observed on 4.20.13, and 4.14.0..4.14.105 (4.14.106 is currently
> running, but hasn't locked up yet).
>
> Filesystem mount flags: compress=zstd,ssd,flushoncommit,space_cache=v2.
> Configuration is either -draid1/-mraid1 or -dsingle/-mraid1. I've
> also reproduced a lockup without flushoncommit.
>
> The machines that are locking up all run the same workload:
>
> rsync receiving data continuously (gigabytes aren't enough,
> I can barely reproduce this once a month with 2TB of rsync
> traffic from 10 simulated clients)
>
> bees doing continuous dedupe
>
> snapshots daily and after each rsync
>
> snapshot deletes as required to maintain free space
>
> scrubs twice monthly plus after each crash
>
> watchdog does a 'mkdir foo; rmdir foo' every few seconds.
> If this takes longer than 50 minutes, collect a stack trace;
> longer than 60 minutes, reboot the machine.
>
[...]
> Here's a recent lockup example with lockdep output, from 4.14.105 with
> the cherry-picked patches above:
The above had __sb_start_write in the call stack, which occurs in 7 out of 15
call traces on one machine:
> [Fri Mar 15 21:53:36 2019] crawl_294 D 0 20349 31220 0x00000000
> [Fri Mar 15 21:53:36 2019] Call Trace:
> [Fri Mar 15 21:53:36 2019] ? __schedule+0x429/0xbb0
> [Fri Mar 15 21:53:36 2019] ? rwsem_down_write_failed+0x134/0x2b0
> [Fri Mar 15 21:53:36 2019] schedule+0x39/0x90
> [Fri Mar 15 21:53:36 2019] rwsem_down_write_failed+0x139/0x2b0
> [Fri Mar 15 21:53:36 2019] ? call_rwsem_down_write_failed+0x13/0x20
> [Fri Mar 15 21:53:36 2019] call_rwsem_down_write_failed+0x13/0x20
> [Fri Mar 15 21:53:36 2019] down_write_nested+0x87/0xb0
> [Fri Mar 15 21:53:36 2019] btrfs_dedupe_file_range+0x8e/0x660
> [Fri Mar 15 21:53:36 2019] ? rcu_read_lock_sched_held+0x68/0x70
> [Fri Mar 15 21:53:36 2019] ? rcu_sync_lockdep_assert+0x30/0x60
> [Fri Mar 15 21:53:36 2019] ? __sb_start_write+0xcc/0x1b0
> [Fri Mar 15 21:53:36 2019] ? mnt_want_write_file+0x3b/0xb0
> [Fri Mar 15 21:53:36 2019] vfs_dedupe_file_range+0x22e/0x280
> [Fri Mar 15 21:53:36 2019] do_vfs_ioctl+0x24d/0x6b0
> [Fri Mar 15 21:53:36 2019] ? __fget+0x11f/0x210
> [Fri Mar 15 21:53:36 2019] SyS_ioctl+0x74/0x80
> [Fri Mar 15 21:53:36 2019] do_syscall_64+0x76/0x180
> [Fri Mar 15 21:53:36 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
> [Fri Mar 15 21:53:36 2019] RIP: 0033:0x7fd690048dd7
> [Fri Mar 15 21:53:36 2019] RSP: 002b:00007fd687ffc0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [Fri Mar 15 21:53:36 2019] RAX: ffffffffffffffda RBX: 00007fd568332780 RCX: 00007fd690048dd7
> [Fri Mar 15 21:53:36 2019] RDX: 00007fd568332780 RSI: 00000000c0189436 RDI: 0000000000000067
> [Fri Mar 15 21:53:36 2019] RBP: 00007fd687ffc420 R08: 00007fd56804c940 R09: 0000000000000000
> [Fri Mar 15 21:53:36 2019] R10: 00007fd568022ed0 R11: 0000000000000246 R12: 0000000000000020
> [Fri Mar 15 21:53:36 2019] R13: 0000000000000018 R14: 00007fd56804c958 R15: 00007fd687ffc428
Here's an example of a deadlock on 4.14.106 that doesn't call
__sb_start_write:
[Sat Mar 23 06:13:07 2019] sysrq: SysRq : Show Blocked State
[Sat Mar 23 06:13:07 2019] task PC stack pid father
[Sat Mar 23 06:13:07 2019] btrfs-transacti D 0 28857 2 0x80000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] schedule_timeout+0x20f/0x590
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] ? wait_for_common+0x3c/0x1f0
[Sat Mar 23 06:13:07 2019] ? wait_for_common+0x131/0x1f0
[Sat Mar 23 06:13:07 2019] wait_for_common+0x131/0x1f0
[Sat Mar 23 06:13:07 2019] ? wake_up_q+0x70/0x70
[Sat Mar 23 06:13:07 2019] __start_delalloc_inodes+0x22f/0x320
[Sat Mar 23 06:13:07 2019] btrfs_start_delalloc_roots+0x1cd/0x380
[Sat Mar 23 06:13:07 2019] btrfs_commit_transaction+0x807/0xa20
[Sat Mar 23 06:13:07 2019] ? start_transaction+0x89/0x4d0
[Sat Mar 23 06:13:07 2019] transaction_kthread+0x194/0x1d0
[Sat Mar 23 06:13:07 2019] kthread+0x10d/0x140
[Sat Mar 23 06:13:07 2019] ? btrfs_cleanup_transaction+0x650/0x650
[Sat Mar 23 06:13:07 2019] ? kthread_create_on_node+0x40/0x40
[Sat Mar 23 06:13:07 2019] ret_from_fork+0x27/0x50
[Sat Mar 23 06:13:07 2019] crawl_writeback D 0 10240 9418 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fbb538ab607
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fbb497f8cc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 00000000000e09c1 RCX: 00007fbb538ab607
[Sat Mar 23 06:13:07 2019] RDX: 00000000000e09c1 RSI: 00007fba300008f0 RDI: 0000000000000003
[Sat Mar 23 06:13:07 2019] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007fbb497f9080
[Sat Mar 23 06:13:07 2019] R10: 0000000000000180 R11: 0000000000000293 R12: 00007fba300008f0
[Sat Mar 23 06:13:07 2019] R13: 0000000000000003 R14: 00007fbb497f93d0 R15: 00007fbb497f9350
[Sat Mar 23 06:13:07 2019] crawl_61509 D 0 27096 9418 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_clone+0x5a3/0xcc0
[Sat Mar 23 06:13:07 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Sat Mar 23 06:13:07 2019] vfs_dedupe_file_range+0x22e/0x280
[Sat Mar 23 06:13:07 2019] do_vfs_ioctl+0x24d/0x6b0
[Sat Mar 23 06:13:07 2019] ? __fget+0x11f/0x210
[Sat Mar 23 06:13:07 2019] SyS_ioctl+0x74/0x80
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fbb51fc90a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 00007fba18b67e40 RCX: 00007fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RDX: 00007fba18b67e40 RSI: 00000000c0189436 RDI: 0000000000000752
[Sat Mar 23 06:13:07 2019] RBP: 00007fbb51fc9420 R08: 00007fba18405480 R09: 0000000000000000
[Sat Mar 23 06:13:07 2019] R10: 00007fba19e4e0e0 R11: 0000000000000246 R12: 0000000000000020
[Sat Mar 23 06:13:07 2019] R13: 0000000000000018 R14: 00007fba18405498 R15: 00007fbb51fc9428
[Sat Mar 23 06:13:07 2019] sysrq: SysRq : Show Locks Held
[Sat Mar 23 06:13:07 2019] crawl_61509 D
[Sat Mar 23 06:13:07 2019] INFO: lockdep is turned off.
[Sat Mar 23 06:13:07 2019] 0 2421 9418 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_clone+0x5a3/0xcc0
[Sat Mar 23 06:13:07 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Sat Mar 23 06:13:07 2019] vfs_dedupe_file_range+0x22e/0x280
[Sat Mar 23 06:13:07 2019] do_vfs_ioctl+0x24d/0x6b0
[Sat Mar 23 06:13:07 2019] ? __fget+0x11f/0x210
[Sat Mar 23 06:13:07 2019] SyS_ioctl+0x74/0x80
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fbb527c9ac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 00007fba2e672370 RCX: 00007fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RDX: 00007fba2e672370 RSI: 00000000c0189436 RDI: 000000000000074d
[Sat Mar 23 06:13:07 2019] RBP: 00007fbb527c9e40 R08: 00007fba2dcb3970 R09: 0000000000000000
[Sat Mar 23 06:13:07 2019] R10: 00007fba2d008300 R11: 0000000000000246 R12: 0000000000000020
[Sat Mar 23 06:13:07 2019] R13: 0000000000000018 R14: 00007fba2dcb3988 R15: 00007fbb527c9e48
[Sat Mar 23 06:13:07 2019] crawl_61509 D 0 17948 9418 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_clone+0x5a3/0xcc0
[Sat Mar 23 06:13:07 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Sat Mar 23 06:13:07 2019] vfs_dedupe_file_range+0x22e/0x280
[Sat Mar 23 06:13:07 2019] do_vfs_ioctl+0x24d/0x6b0
[Sat Mar 23 06:13:07 2019] ? __fget+0x11f/0x210
[Sat Mar 23 06:13:07 2019] SyS_ioctl+0x74/0x80
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fbb50fc6ac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 00007fba34402eb0 RCX: 00007fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RDX: 00007fba34402eb0 RSI: 00000000c0189436 RDI: 00000000000006fb
[Sat Mar 23 06:13:07 2019] RBP: 00007fbb50fc6e40 R08: 00007fba3428f630 R09: 0000000000000000
[Sat Mar 23 06:13:07 2019] R10: 00007fba361733c0 R11: 0000000000000246 R12: 0000000000000020
[Sat Mar 23 06:13:07 2019] R13: 0000000000000018 R14: 00007fba3428f648 R15: 00007fbb50fc6e48
[Sat Mar 23 06:13:07 2019] crawl_61509 D 0 17949 9418 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_clone+0x5a3/0xcc0
[Sat Mar 23 06:13:07 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Sat Mar 23 06:13:07 2019] vfs_dedupe_file_range+0x22e/0x280
[Sat Mar 23 06:13:07 2019] do_vfs_ioctl+0x24d/0x6b0
[Sat Mar 23 06:13:07 2019] ? __fget+0x11f/0x210
[Sat Mar 23 06:13:07 2019] SyS_ioctl+0x74/0x80
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fbb4b7fb0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 00007fba1e1f8870 RCX: 00007fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RDX: 00007fba1e1f8870 RSI: 00000000c0189436 RDI: 0000000000000754
[Sat Mar 23 06:13:07 2019] RBP: 00007fbb4b7fb420 R08: 00007fba1e25cc60 R09: 0000000000000000
[Sat Mar 23 06:13:07 2019] R10: 00007fba1dbcb0c0 R11: 0000000000000246 R12: 0000000000000020
[Sat Mar 23 06:13:07 2019] R13: 0000000000000018 R14: 00007fba1e25cc78 R15: 00007fbb4b7fb428
[Sat Mar 23 06:13:07 2019] crawl_61509 D 0 18445 9418 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_clone+0x5a3/0xcc0
[Sat Mar 23 06:13:07 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Sat Mar 23 06:13:07 2019] vfs_dedupe_file_range+0x22e/0x280
[Sat Mar 23 06:13:07 2019] do_vfs_ioctl+0x24d/0x6b0
[Sat Mar 23 06:13:07 2019] ? __fget+0x11f/0x210
[Sat Mar 23 06:13:07 2019] SyS_ioctl+0x74/0x80
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fbb4bffc0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 00007fba200fb8c0 RCX: 00007fbb538b0dd7
[Sat Mar 23 06:13:07 2019] RDX: 00007fba200fb8c0 RSI: 00000000c0189436 RDI: 0000000000000753
[Sat Mar 23 06:13:07 2019] RBP: 00007fbb4bffc420 R08: 00007fba21a8c680 R09: 0000000000000000
[Sat Mar 23 06:13:07 2019] R10: 00007fba20736910 R11: 0000000000000246 R12: 0000000000000020
[Sat Mar 23 06:13:07 2019] R13: 0000000000000018 R14: 00007fba21a8c698 R15: 00007fbb4bffc428
[Sat Mar 23 06:13:07 2019] rsync D 0 1718 8245 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_setattr+0x1e6/0x3b0
[Sat Mar 23 06:13:07 2019] notify_change+0x2ed/0x450
[Sat Mar 23 06:13:07 2019] do_truncate+0x5e/0x90
[Sat Mar 23 06:13:07 2019] do_sys_ftruncate.constprop.1+0x120/0x180
[Sat Mar 23 06:13:07 2019] ? trace_hardirqs_off_thunk+0x1a/0x1c
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7f105ec2ac87
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fff425b6b38 EFLAGS: 00000246 ORIG_RAX: 000000000000004d
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f105ec2ac87
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000000 RSI: 0000000000f04e27 RDI: 0000000000000004
[Sat Mar 23 06:13:07 2019] RBP: 0000000000000004 R08: 0000000000000000 R09: 00007f105ec757b8
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000f04e27
[Sat Mar 23 06:13:07 2019] R13: 0000000000005bbf R14: 0000000000000f78 R15: 0000000000000005
[Sat Mar 23 06:13:07 2019] kworker/u8:8 D 0 22026 2 0x80000000
[Sat Mar 23 06:13:07 2019] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] io_schedule+0x12/0x40
[Sat Mar 23 06:13:07 2019] __lock_page+0x110/0x140
[Sat Mar 23 06:13:07 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Sat Mar 23 06:13:07 2019] extent_write_cache_pages.constprop.27+0x2d9/0x3b0
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] extent_writepages+0x4e/0x70
[Sat Mar 23 06:13:07 2019] ? btrfs_direct_IO+0x3e0/0x3e0
[Sat Mar 23 06:13:07 2019] do_writepages+0x23/0x80
[Sat Mar 23 06:13:07 2019] __filemap_fdatawrite_range+0xaa/0xe0
[Sat Mar 23 06:13:07 2019] btrfs_run_delalloc_work+0x17/0x60
[Sat Mar 23 06:13:07 2019] btrfs_scrubparity_helper+0x8f/0x640
[Sat Mar 23 06:13:07 2019] process_one_work+0x204/0x650
[Sat Mar 23 06:13:07 2019] worker_thread+0x4d/0x3b0
[Sat Mar 23 06:13:07 2019] kthread+0x10d/0x140
[Sat Mar 23 06:13:07 2019] ? process_one_work+0x650/0x650
[Sat Mar 23 06:13:07 2019] ? kthread_create_on_node+0x40/0x40
[Sat Mar 23 06:13:07 2019] ret_from_fork+0x27/0x50
[Sat Mar 23 06:13:07 2019] rsync D 0 10508 10507 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? __might_fault+0x3e/0x90
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7f35b016b4b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffd7c166a28 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f35b016b4b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffd7c168c90
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f35b01bdf10
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd7c168ca3
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f35b01f9320
[Sat Mar 23 06:13:07 2019] mkdir D 0 11463 11460 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_mkdir+0x50/0x1d0
[Sat Mar 23 06:13:07 2019] vfs_mkdir+0x134/0x1c0
[Sat Mar 23 06:13:07 2019] SyS_mkdirat+0xb3/0xf0
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7f09ce6ab447
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffc0d353648 EFLAGS: 00000206 ORIG_RAX: 0000000000000053
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 00007ffc0d354b28 RCX: 00007f09ce6ab447
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007ffc0d354b4d
[Sat Mar 23 06:13:07 2019] RBP: 00007ffc0d3537c0 R08: 00000000000001ff R09: 000055d022f36e10
[Sat Mar 23 06:13:07 2019] R10: 0000000000000376 R11: 0000000000000206 R12: 000055d022f36e50
[Sat Mar 23 06:13:07 2019] R13: 00000000000001ff R14: 00007ffc0d353960 R15: 0000000000000001
[Sat Mar 23 06:13:07 2019] rsync D 0 17989 17923 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fa5521834b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007fff48155588 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa5521834b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007fff481577f0
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000ffff R09: 00007fa5521d5d40
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff48157805
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fa552211320
[Sat Mar 23 06:13:07 2019] rsync D 0 24892 24891 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7f6a4ed8b4b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffdf6712108 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f6a4ed8b4b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffdf6714370
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f6a4eddd950
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffdf6714387
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f6a4ee19320
[Sat Mar 23 06:13:07 2019] rsync D 0 12215 12151 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7f8110fb34b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffcce477ec8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f8110fb34b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffcce47a130
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000fff8 R09: 00007f8111005f10
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffcce47a14d
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f8111041320
[Sat Mar 23 06:13:07 2019] rsync D 0 27127 27028 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7f65a9c1b4b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffea8320a18 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f65a9c1b4b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffea8322c80
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000ffff R09: 00007f65a9c6e300
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffea8322cc6
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f65a9ca9320
[Sat Mar 23 06:13:07 2019] rsync D 0 3891 3890 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fa6d9f8b4b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffff60095b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa6d9f8b4b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffff600b820
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000ffff R09: 00007fa6d9fde520
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffff600b833
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fa6da019320
[Sat Mar 23 06:13:07 2019] rsync D 0 26758 26728 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fe51da8b4b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffd1ad7aca8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fe51da8b4b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffd1ad7cf10
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007fe51dadd950
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd1ad7cf27
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fe51db19320
[Sat Mar 23 06:13:07 2019] rsync D 0 12327 12326 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:07 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:07 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:07 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:07 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:07 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:07 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:07 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:07 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:07 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:07 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:07 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:07 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:07 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:07 2019] RIP: 0033:0x7fdd14d1b4b0
[Sat Mar 23 06:13:07 2019] RSP: 002b:00007ffc83bc5938 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:07 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fdd14d1b4b0
[Sat Mar 23 06:13:07 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc83bc7ba0
[Sat Mar 23 06:13:07 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007fdd14d6d950
[Sat Mar 23 06:13:07 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc83bc7bb7
[Sat Mar 23 06:13:07 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fdd14da9320
[Sat Mar 23 06:13:07 2019] rsync D 0 16793 16762 0x00000000
[Sat Mar 23 06:13:07 2019] Call Trace:
[Sat Mar 23 06:13:07 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:07 2019] ? prepare_to_wait_event+0x97/0x190
[Sat Mar 23 06:13:07 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:07 2019] wait_current_trans+0xa3/0xf0
[Sat Mar 23 06:13:08 2019] ? wait_woken+0xa0/0xa0
[Sat Mar 23 06:13:08 2019] start_transaction+0x29b/0x4d0
[Sat Mar 23 06:13:08 2019] btrfs_create+0x47/0x1d0
[Sat Mar 23 06:13:08 2019] lookup_open+0x404/0x760
[Sat Mar 23 06:13:08 2019] path_openat+0x4e9/0xcc0
[Sat Mar 23 06:13:08 2019] ? filemap_map_pages+0x312/0x520
[Sat Mar 23 06:13:08 2019] ? lock_acquire+0xbc/0x200
[Sat Mar 23 06:13:08 2019] do_filp_open+0x8a/0xf0
[Sat Mar 23 06:13:08 2019] ? __alloc_fd+0x34/0x200
[Sat Mar 23 06:13:08 2019] ? expand_files+0x3a/0x2a0
[Sat Mar 23 06:13:08 2019] ? do_sys_open+0x130/0x200
[Sat Mar 23 06:13:08 2019] do_sys_open+0x130/0x200
[Sat Mar 23 06:13:08 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:08 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:08 2019] RIP: 0033:0x7f89056234b0
[Sat Mar 23 06:13:08 2019] RSP: 002b:00007ffc38a789a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Sat Mar 23 06:13:08 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f89056234b0
[Sat Mar 23 06:13:08 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc38a7ac10
[Sat Mar 23 06:13:08 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f8905675950
[Sat Mar 23 06:13:08 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc38a7ac27
[Sat Mar 23 06:13:08 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f89056b1320
[Sat Mar 23 06:13:08 2019] watchdog-mdtest D 0 20886 11460 0x00000000
[Sat Mar 23 06:13:08 2019] Call Trace:
[Sat Mar 23 06:13:08 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:08 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:08 2019] io_schedule+0x12/0x40
[Sat Mar 23 06:13:08 2019] wait_on_page_bit+0x10d/0x140
[Sat Mar 23 06:13:08 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Sat Mar 23 06:13:08 2019] read_extent_buffer_pages+0x215/0x2d0
[Sat Mar 23 06:13:08 2019] ? free_root_pointers+0x60/0x60
[Sat Mar 23 06:13:08 2019] btree_read_extent_buffer_pages+0x86/0x100
[Sat Mar 23 06:13:08 2019] read_tree_block+0x2d/0x50
[Sat Mar 23 06:13:08 2019] read_block_for_search.isra.11+0x117/0x310
[Sat Mar 23 06:13:08 2019] ? _raw_read_lock+0x3b/0x70
[Sat Mar 23 06:13:08 2019] btrfs_search_slot+0x3e7/0x9c0
[Sat Mar 23 06:13:08 2019] btrfs_lookup_dir_item+0x69/0xb0
[Sat Mar 23 06:13:08 2019] btrfs_lookup_dentry+0xa0/0x530
[Sat Mar 23 06:13:08 2019] btrfs_lookup+0xe/0x30
[Sat Mar 23 06:13:08 2019] lookup_slow+0x131/0x210
[Sat Mar 23 06:13:08 2019] walk_component+0x1b2/0x2d0
[Sat Mar 23 06:13:08 2019] path_lookupat+0x64/0x1f0
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_on_caller+0x12/0x1c0
[Sat Mar 23 06:13:08 2019] filename_lookup+0xa9/0x170
[Sat Mar 23 06:13:08 2019] ? __check_object_size+0xa7/0x190
[Sat Mar 23 06:13:08 2019] ? strncpy_from_user+0x48/0x160
[Sat Mar 23 06:13:08 2019] ? _raw_spin_unlock+0x24/0x30
[Sat Mar 23 06:13:08 2019] ? vfs_statx+0x63/0xb0
[Sat Mar 23 06:13:08 2019] vfs_statx+0x63/0xb0
[Sat Mar 23 06:13:08 2019] SyS_newstat+0x2b/0x50
[Sat Mar 23 06:13:08 2019] ? up_read+0x1c/0x40
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_off_caller+0x1b/0xe0
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_off_thunk+0x1a/0x1c
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_on_caller+0x12/0x1c0
[Sat Mar 23 06:13:08 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:08 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:08 2019] RIP: 0033:0x7fe3ffd7b055
[Sat Mar 23 06:13:08 2019] RSP: 002b:00007ffea6aaeed8 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[Sat Mar 23 06:13:08 2019] RAX: ffffffffffffffda RBX: 0000000000000024 RCX: 00007fe3ffd7b055
[Sat Mar 23 06:13:08 2019] RDX: 00007ffea6aaeee0 RSI: 00007ffea6aaeee0 RDI: 00000000014e13c8
[Sat Mar 23 06:13:08 2019] RBP: 00000000014e13c8 R08: 0000000000710710 R09: 00007fe3ffdcdf20
[Sat Mar 23 06:13:08 2019] R10: 00007fe3ffdca430 R11: 0000000000000246 R12: 00000000014e13c8
[Sat Mar 23 06:13:08 2019] R13: 00007ffea6aaeff0 R14: 00007ffea6aaeff0 R15: 00000000014dfda8
[Sat Mar 23 06:13:08 2019] tee D 0 20887 11460 0x80000000
[Sat Mar 23 06:13:08 2019] Call Trace:
[Sat Mar 23 06:13:08 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:08 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:08 2019] io_schedule+0x12/0x40
[Sat Mar 23 06:13:08 2019] wait_on_page_bit_killable+0x114/0x170
[Sat Mar 23 06:13:08 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Sat Mar 23 06:13:08 2019] __lock_page_or_retry+0xa1/0xe0
[Sat Mar 23 06:13:08 2019] filemap_fault+0x244/0x830
[Sat Mar 23 06:13:08 2019] __do_fault+0x3e/0x170
[Sat Mar 23 06:13:08 2019] __handle_mm_fault+0x118e/0x18b0
[Sat Mar 23 06:13:08 2019] handle_mm_fault+0x15c/0x300
[Sat Mar 23 06:13:08 2019] __do_page_fault+0x1fa/0x520
[Sat Mar 23 06:13:08 2019] page_fault+0x25/0x50
[Sat Mar 23 06:13:08 2019] RIP: 0010:__clear_user+0x3d/0x60
[Sat Mar 23 06:13:08 2019] RSP: 0018:ffff9b32495a7d68 EFLAGS: 00010202
[Sat Mar 23 06:13:08 2019] RAX: 0000000000000000 RBX: 00000000000001b0 RCX: 00000000000001b0
[Sat Mar 23 06:13:08 2019] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 000055feabb1c280
[Sat Mar 23 06:13:08 2019] RBP: 000055feabb1c280 R08: 0000000000000001 R09: 0000000000000000
[Sat Mar 23 06:13:08 2019] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[Sat Mar 23 06:13:08 2019] R13: ffff9b32495a4000 R14: ffff8d636f181a00 R15: ffff8d6307e816c0
[Sat Mar 23 06:13:08 2019] ? __clear_user+0x1e/0x60
[Sat Mar 23 06:13:08 2019] load_elf_binary+0x101a/0x170d
[Sat Mar 23 06:13:08 2019] search_binary_handler+0xa0/0x1f0
[Sat Mar 23 06:13:08 2019] do_execveat_common.isra.13+0x689/0x9a0
[Sat Mar 23 06:13:08 2019] SyS_execve+0x2c/0x40
[Sat Mar 23 06:13:08 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:08 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:08 2019] RIP: 0033:0x7fe3ffd58647
[Sat Mar 23 06:13:08 2019] RSP: 002b:00007ffea6aaf038 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[Sat Mar 23 06:13:08 2019] RAX: ffffffffffffffda RBX: 00000000014dec28 RCX: 00007fe3ffd58647
[Sat Mar 23 06:13:08 2019] RDX: 00000000014cf608 RSI: 00000000014e1688 RDI: 00000000014dec28
[Sat Mar 23 06:13:08 2019] RBP: 0000000000000000 R08: 0000000000710710 R09: 0000000000000000
[Sat Mar 23 06:13:08 2019] R10: 000000000000059a R11: 0000000000000202 R12: 0000000000000000
[Sat Mar 23 06:13:08 2019] R13: 00000000014e1688 R14: 00000000014cf608 R15: 0000000000000040
[Sat Mar 23 06:13:08 2019] watchdog-mdtest D 0 20890 11460 0x00000000
[Sat Mar 23 06:13:08 2019] Call Trace:
[Sat Mar 23 06:13:08 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:08 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:08 2019] io_schedule+0x12/0x40
[Sat Mar 23 06:13:08 2019] wait_on_page_bit+0x10d/0x140
[Sat Mar 23 06:13:08 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Sat Mar 23 06:13:08 2019] read_extent_buffer_pages+0x215/0x2d0
[Sat Mar 23 06:13:08 2019] ? free_root_pointers+0x60/0x60
[Sat Mar 23 06:13:08 2019] btree_read_extent_buffer_pages+0x86/0x100
[Sat Mar 23 06:13:08 2019] read_tree_block+0x2d/0x50
[Sat Mar 23 06:13:08 2019] read_block_for_search.isra.11+0x117/0x310
[Sat Mar 23 06:13:08 2019] ? _raw_read_lock+0x3b/0x70
[Sat Mar 23 06:13:08 2019] btrfs_search_slot+0x3e7/0x9c0
[Sat Mar 23 06:13:08 2019] btrfs_lookup_dir_item+0x69/0xb0
[Sat Mar 23 06:13:08 2019] btrfs_lookup_dentry+0xa0/0x530
[Sat Mar 23 06:13:08 2019] btrfs_lookup+0xe/0x30
[Sat Mar 23 06:13:08 2019] lookup_slow+0x131/0x210
[Sat Mar 23 06:13:08 2019] walk_component+0x1b2/0x2d0
[Sat Mar 23 06:13:08 2019] path_lookupat+0x64/0x1f0
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_on_caller+0x12/0x1c0
[Sat Mar 23 06:13:08 2019] filename_lookup+0xa9/0x170
[Sat Mar 23 06:13:08 2019] ? __check_object_size+0xa7/0x190
[Sat Mar 23 06:13:08 2019] ? strncpy_from_user+0x48/0x160
[Sat Mar 23 06:13:08 2019] ? finish_task_switch+0x64/0x230
[Sat Mar 23 06:13:08 2019] ? vfs_statx+0x63/0xb0
[Sat Mar 23 06:13:08 2019] vfs_statx+0x63/0xb0
[Sat Mar 23 06:13:08 2019] ? finish_task_switch+0xa0/0x230
[Sat Mar 23 06:13:08 2019] ? finish_task_switch+0x64/0x230
[Sat Mar 23 06:13:08 2019] SyS_newstat+0x2b/0x50
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_off_caller+0x1b/0xe0
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_off_thunk+0x1a/0x1c
[Sat Mar 23 06:13:08 2019] ? trace_hardirqs_on_caller+0x12/0x1c0
[Sat Mar 23 06:13:08 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:08 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:08 2019] RIP: 0033:0x7fe3ffd7b055
[Sat Mar 23 06:13:08 2019] RSP: 002b:00007ffea6aaee08 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[Sat Mar 23 06:13:08 2019] RAX: ffffffffffffffda RBX: 0000000000000024 RCX: 00007fe3ffd7b055
[Sat Mar 23 06:13:08 2019] RDX: 00007ffea6aaee10 RSI: 00007ffea6aaee10 RDI: 00000000014cdea8
[Sat Mar 23 06:13:08 2019] RBP: 00000000014cdea8 R08: 0000000000710710 R09: 00007fe3ffdcd7b0
[Sat Mar 23 06:13:08 2019] R10: ffffffffffffffe0 R11: 0000000000000246 R12: 00000000014cdea8
[Sat Mar 23 06:13:08 2019] R13: 00007ffea6aaef20 R14: 00007ffea6aaef20 R15: 00000000014cde88
[Sat Mar 23 06:13:08 2019] tee D 0 20892 11460 0x80000000
[Sat Mar 23 06:13:08 2019] Call Trace:
[Sat Mar 23 06:13:08 2019] ? __schedule+0x429/0xbb0
[Sat Mar 23 06:13:08 2019] schedule+0x39/0x90
[Sat Mar 23 06:13:08 2019] io_schedule+0x12/0x40
[Sat Mar 23 06:13:08 2019] wait_on_page_bit_killable+0x114/0x170
[Sat Mar 23 06:13:08 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Sat Mar 23 06:13:08 2019] __lock_page_or_retry+0xa1/0xe0
[Sat Mar 23 06:13:08 2019] filemap_fault+0x244/0x830
[Sat Mar 23 06:13:08 2019] __do_fault+0x3e/0x170
[Sat Mar 23 06:13:08 2019] __handle_mm_fault+0x118e/0x18b0
[Sat Mar 23 06:13:08 2019] handle_mm_fault+0x15c/0x300
[Sat Mar 23 06:13:08 2019] __do_page_fault+0x1fa/0x520
[Sat Mar 23 06:13:08 2019] page_fault+0x25/0x50
[Sat Mar 23 06:13:08 2019] RIP: 0010:__clear_user+0x3d/0x60
[Sat Mar 23 06:13:08 2019] RSP: 0018:ffff9b32495d7d68 EFLAGS: 00010202
[Sat Mar 23 06:13:08 2019] RAX: 0000000000000000 RBX: 00000000000001b0 RCX: 00000000000001b0
[Sat Mar 23 06:13:08 2019] RDX: 0000000000000000 RSI: 0000000000000008 RDI: 000055f378ec1280
[Sat Mar 23 06:13:08 2019] RBP: 000055f378ec1280 R08: 0000000000000001 R09: 0000000000000000
[Sat Mar 23 06:13:08 2019] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[Sat Mar 23 06:13:08 2019] R13: ffff9b32495d4000 R14: ffff8d656a48ae00 R15: ffff8d63146576c0
[Sat Mar 23 06:13:08 2019] ? __clear_user+0x1e/0x60
[Sat Mar 23 06:13:08 2019] load_elf_binary+0x101a/0x170d
[Sat Mar 23 06:13:08 2019] search_binary_handler+0xa0/0x1f0
[Sat Mar 23 06:13:08 2019] do_execveat_common.isra.13+0x689/0x9a0
[Sat Mar 23 06:13:08 2019] SyS_execve+0x2c/0x40
[Sat Mar 23 06:13:08 2019] do_syscall_64+0x76/0x180
[Sat Mar 23 06:13:08 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Sat Mar 23 06:13:08 2019] RIP: 0033:0x7fe3ffd58647
[Sat Mar 23 06:13:08 2019] RSP: 002b:00007ffea6aaef68 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[Sat Mar 23 06:13:08 2019] RAX: ffffffffffffffda RBX: 00000000014dfda8 RCX: 00007fe3ffd58647
[Sat Mar 23 06:13:08 2019] RDX: 00000000014cf608 RSI: 00000000014d3608 RDI: 00000000014dfda8
[Sat Mar 23 06:13:08 2019] RBP: 0000000000000000 R08: 0000000000710710 R09: 0000000000000000
[Sat Mar 23 06:13:08 2019] R10: 000000000000059a R11: 0000000000000202 R12: 0000000000000000
[Sat Mar 23 06:13:08 2019] R13: 00000000014d3608 R14: 00000000014cf608 R15: 0000000000000040
This looks like it may be a distinct deadlock bug on 4.14. Statistically,
it's very unlikely that there's only one.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: A collection of btrfs lockup stack traces (4.14.107, kcompactd and two processes running __lock_page)
2019-03-27 2:37 ` A collection of btrfs lockup stack traces (4.14.106, no __sb_start_write) Zygo Blaxell
@ 2019-03-27 20:04 ` Zygo Blaxell
0 siblings, 0 replies; 6+ messages in thread
From: Zygo Blaxell @ 2019-03-27 20:04 UTC (permalink / raw)
To: linux-btrfs
On Tue, Mar 26, 2019 at 10:37:06PM -0400, Zygo Blaxell wrote:
> On Tue, Mar 19, 2019 at 11:39:59PM -0400, Zygo Blaxell wrote:
> > I haven't been able to easily reproduce these in a test environment;
> > however, they have been happening several times a year on servers in
> > production.
> >
> > Kernel: most recent observation on 4.14.105 + cherry-picked deadlock
> > and misc hang fixes:
> >
> > btrfs: wakeup cleaner thread when adding delayed iput
> > Btrfs: fix deadlock when allocating tree block during leaf/node split
> > Btrfs: use nofs context when initializing security xattrs to avoid deadlock
> > Btrfs: fix deadlock with memory reclaim during scrub
> > Btrfs: fix deadlock between clone/dedupe and rename
> >
> > Also observed on 4.20.13, and 4.14.0..4.14.105 (4.14.106 is currently
> > running, but hasn't locked up yet).
> >
> > Filesystem mount flags: compress=zstd,ssd,flushoncommit,space_cache=v2.
> > Configuration is either -draid1/-mraid1 or -dsingle/-mraid1. I've
> > also reproduced a lockup without flushoncommit.
> >
> > The machines that are locking up all run the same workload:
> >
> > rsync receiving data continuously (gigabytes aren't enough,
> > I can barely reproduce this once a month with 2TB of rsync
> > traffic from 10 simulated clients)
> >
> > bees doing continuous dedupe
> >
> > snapshots daily and after each rsync
> >
> > snapshot deletes as required to maintain free space
> >
> > scrubs twice monthly plus after each crash
> >
> > watchdog does a 'mkdir foo; rmdir foo' every few seconds.
> > If this takes longer than 50 minutes, collect a stack trace;
> > longer than 60 minutes, reboot the machine.
> >
> [...]
Here's 4.14.107 with kcompactd and two processes running __lock_page.
This happens in 6 of 16 stack traces--the others don't have kcompactd
nor do they have two threads executing __lock_page. It looks just
different enough from the others to possibly be a distinct bug.
[Tue Mar 26 16:43:30 2019] sysrq: SysRq : Show Locks Held
[Tue Mar 26 16:43:30 2019]
Showing all locks held in the system:
[Tue Mar 26 16:43:30 2019] 1 lock held by in:imklog/7096:
[Tue Mar 26 16:43:30 2019] #0: (&f->f_pos_lock){+.+.}, at: [<ffffffffaa2d2088>] __fdget_pos+0x48/0x60
[Tue Mar 26 16:43:30 2019] 3 locks held by btrfs-transacti/29027:
[Tue Mar 26 16:43:30 2019] #0: (&fs_info->transaction_kthread_mutex){+.+.}, at: [<ffffffffaa435584>] transaction_kthread+0x74/0x1d0
[Tue Mar 26 16:43:30 2019] #1: (&fs_info->delalloc_root_mutex){+.+.}, at: [<ffffffffaa44f9f8>] btrfs_start_delalloc_roots+0x58/0x380
[Tue Mar 26 16:43:30 2019] #2: (&root->delalloc_mutex){+.+.}, at: [<ffffffffaa44f5d2>] __start_delalloc_inodes+0x52/0x320
[Tue Mar 26 16:43:30 2019] 2 locks held by hash_writeback/29369:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2ae2f3>] vfs_write+0x163/0x1b0
[Tue Mar 26 16:43:30 2019] #1: (&sb->s_type->i_mutex_key#14){+.+.}, at: [<ffffffffaa4558a0>] btrfs_file_write_iter+0x90/0x5b0
[Tue Mar 26 16:43:30 2019] 3 locks held by crawl_writeback/29399:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by crawl_61509/25922:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4f5b>] mnt_want_write_file+0x3b/0xb0
[Tue Mar 26 16:43:30 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffffaa479f0e>] btrfs_dedupe_file_range+0x8e/0x660
[Tue Mar 26 16:43:30 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffffaa479f1f>] btrfs_dedupe_file_range+0x9f/0x660
[Tue Mar 26 16:43:30 2019] 4 locks held by crawl_61509/25923:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4f5b>] mnt_want_write_file+0x3b/0xb0
[Tue Mar 26 16:43:30 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffffaa479f0e>] btrfs_dedupe_file_range+0x8e/0x660
[Tue Mar 26 16:43:30 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffffaa479f1f>] btrfs_dedupe_file_range+0x9f/0x660
[Tue Mar 26 16:43:30 2019] #3: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 4 locks held by crawl_61509/26256:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4f5b>] mnt_want_write_file+0x3b/0xb0
[Tue Mar 26 16:43:30 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffffaa479f0e>] btrfs_dedupe_file_range+0x8e/0x660
[Tue Mar 26 16:43:30 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffffaa479f1f>] btrfs_dedupe_file_range+0x9f/0x660
[Tue Mar 26 16:43:30 2019] #3: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 4 locks held by crawl_61509/26257:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4f5b>] mnt_want_write_file+0x3b/0xb0
[Tue Mar 26 16:43:30 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffffaa479f0e>] btrfs_dedupe_file_range+0x8e/0x660
[Tue Mar 26 16:43:30 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffffaa479f1f>] btrfs_dedupe_file_range+0x9f/0x660
[Tue Mar 26 16:43:30 2019] #3: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 4 locks held by crawl_61509/26258:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4f5b>] mnt_want_write_file+0x3b/0xb0
[Tue Mar 26 16:43:30 2019] #1: (&sb->s_type->i_mutex_key#14/1){+.+.}, at: [<ffffffffaa479f0e>] btrfs_dedupe_file_range+0x8e/0x660
[Tue Mar 26 16:43:30 2019] #2: (&sb->s_type->i_mutex_key#14/2){+.+.}, at: [<ffffffffaa479f1f>] btrfs_dedupe_file_range+0x9f/0x660
[Tue Mar 26 16:43:30 2019] #3: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/22902:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2aa8ec>] do_sys_ftruncate.constprop.1+0xdc/0x180
[Tue Mar 26 16:43:30 2019] #1: (&sb->s_type->i_mutex_key#14){+.+.}, at: [<ffffffffaa2aa51f>] do_truncate+0x4f/0x90
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 1 lock held by rsync/12425:
[Tue Mar 26 16:43:30 2019] #0: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2b9447>] lookup_slow+0xe7/0x210
[Tue Mar 26 16:43:30 2019] 2 locks held by kworker/u8:14/13297:
[Tue Mar 26 16:43:30 2019] #0: ("%s-%s""btrfs", name){+.+.}, at: [<ffffffffaa0ae2ce>] process_one_work+0x17e/0x650
[Tue Mar 26 16:43:30 2019] #1: ((&work->normal_work)){+.+.}, at: [<ffffffffaa0ae2ce>] process_one_work+0x17e/0x650
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/9359:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 4 locks held by rsync/8086:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6/1){+.+.}, at: [<ffffffffaa2bf993>] do_unlinkat+0x113/0x300
[Tue Mar 26 16:43:30 2019] #2: (&sb->s_type->i_mutex_key#14){+.+.}, at: [<ffffffffaa2b9edc>] vfs_unlink+0x4c/0x190
[Tue Mar 26 16:43:30 2019] #3: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by kworker/u8:10/27109:
[Tue Mar 26 16:43:30 2019] #0: ("writeback"){+.+.}, at: [<ffffffffaa0ae2ce>] process_one_work+0x17e/0x650
[Tue Mar 26 16:43:30 2019] #1: ((&(&wb->dwork)->work)){+.+.}, at: [<ffffffffaa0ae2ce>] process_one_work+0x17e/0x650
[Tue Mar 26 16:43:30 2019] #2: (&type->s_umount_key#24){++++}, at: [<ffffffffaa2b1276>] trylock_super+0x16/0x50
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/1942:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/4476:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by mkdir/4816:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6/1){+.+.}, at: [<ffffffffaa2bf56f>] filename_create+0x6f/0x140
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/6356:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/6626:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/17256:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 1 lock held by rsync/22060:
[Tue Mar 26 16:43:30 2019] #0: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2b9447>] lookup_slow+0xe7/0x210
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/29615:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/7453:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/29610:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6/1){+.+.}, at: [<ffffffffaa2bf56f>] filename_create+0x6f/0x140
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/29638:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/10214:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/29837:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/31885:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/7092:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/10725:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/27579:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/27949:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6){++++}, at: [<ffffffffaa2be4ce>] path_openat+0x4ce/0xcc0
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by rsync/7504:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#8){.+.+}, at: [<ffffffffaa2d4ed0>] mnt_want_write+0x20/0x50
[Tue Mar 26 16:43:30 2019] #1: (&type->i_mutex_dir_key#6/1){+.+.}, at: [<ffffffffaa2bf56f>] filename_create+0x6f/0x140
[Tue Mar 26 16:43:30 2019] #2: (sb_internal){.+.+}, at: [<ffffffffaa43b8a6>] start_transaction+0x2b6/0x4d0
[Tue Mar 26 16:43:30 2019] 3 locks held by watchdog-mdtest/10434:
[Tue Mar 26 16:43:30 2019] #0: (sb_writers#6){.+.+}, at: [<ffffffffaa2ae2f3>] vfs_write+0x163/0x1b0
[Tue Mar 26 16:43:30 2019] #1: (rcu_read_lock){....}, at: [<ffffffffaa7442d5>] __handle_sysrq+0x5/0x210
[Tue Mar 26 16:43:30 2019] #2: (tasklist_lock){.+.+}, at: [<ffffffffaa0f0377>] debug_show_all_locks+0x37/0x190
[Tue Mar 26 16:43:30 2019] =============================================
[Tue Mar 26 16:43:30 2019] sysrq: SysRq : Show Blocked State
[Tue Mar 26 16:43:30 2019] task PC stack pid father
[Tue Mar 26 16:43:30 2019] kcompactd0 D 0 469 2 0x80000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] io_schedule+0x12/0x40
[Tue Mar 26 16:43:30 2019] __lock_page+0x110/0x140
[Tue Mar 26 16:43:30 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Tue Mar 26 16:43:30 2019] migrate_pages+0xc12/0xea0
[Tue Mar 26 16:43:30 2019] ? __compaction_suitable.part.14+0x80/0x80
[Tue Mar 26 16:43:30 2019] ? update_pageblock_skip+0xb0/0xb0
[Tue Mar 26 16:43:30 2019] compact_zone+0x238/0xc40
[Tue Mar 26 16:43:30 2019] ? rcu_read_lock_sched_held+0x68/0x70
[Tue Mar 26 16:43:30 2019] kcompactd_do_work+0x140/0x300
[Tue Mar 26 16:43:30 2019] ? kcompactd+0x8e/0x290
[Tue Mar 26 16:43:30 2019] kcompactd+0x8e/0x290
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] kthread+0x10d/0x140
[Tue Mar 26 16:43:30 2019] ? kcompactd_do_work+0x300/0x300
[Tue Mar 26 16:43:30 2019] ? kthread_create_on_node+0x40/0x40
[Tue Mar 26 16:43:30 2019] ret_from_fork+0x27/0x50
[Tue Mar 26 16:43:30 2019] btrfs-transacti D 0 29027 2 0x80000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] schedule_timeout+0x20f/0x590
[Tue Mar 26 16:43:30 2019] ? lock_acquire+0xbc/0x200
[Tue Mar 26 16:43:30 2019] ? mark_held_locks+0x6a/0x90
[Tue Mar 26 16:43:30 2019] ? _raw_spin_unlock_irq+0x29/0x40
[Tue Mar 26 16:43:30 2019] ? wait_for_common+0x131/0x1f0
[Tue Mar 26 16:43:30 2019] wait_for_common+0x131/0x1f0
[Tue Mar 26 16:43:30 2019] ? wake_up_q+0x70/0x70
[Tue Mar 26 16:43:30 2019] __start_delalloc_inodes+0x22f/0x320
[Tue Mar 26 16:43:30 2019] btrfs_start_delalloc_roots+0x1cd/0x380
[Tue Mar 26 16:43:30 2019] btrfs_commit_transaction+0x807/0xa20
[Tue Mar 26 16:43:30 2019] ? start_transaction+0x89/0x4d0
[Tue Mar 26 16:43:30 2019] transaction_kthread+0x194/0x1d0
[Tue Mar 26 16:43:30 2019] kthread+0x10d/0x140
[Tue Mar 26 16:43:30 2019] ? btrfs_cleanup_transaction+0x650/0x650
[Tue Mar 26 16:43:30 2019] ? kthread_create_on_node+0x40/0x40
[Tue Mar 26 16:43:30 2019] ret_from_fork+0x27/0x50
[Tue Mar 26 16:43:30 2019] hash_writeback D 0 29369 29318 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? _raw_spin_unlock_irqrestore+0x32/0x60
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] schedule_timeout+0x1d0/0x590
[Tue Mar 26 16:43:30 2019] ? run_timer_softirq+0x130/0x130
[Tue Mar 26 16:43:30 2019] ? io_schedule_timeout+0x19/0x40
[Tue Mar 26 16:43:30 2019] io_schedule_timeout+0x19/0x40
[Tue Mar 26 16:43:30 2019] balance_dirty_pages.isra.12+0x345/0xf80
[Tue Mar 26 16:43:30 2019] balance_dirty_pages_ratelimited+0x648/0xab0
[Tue Mar 26 16:43:30 2019] __btrfs_buffered_write.isra.4+0x425/0x730
[Tue Mar 26 16:43:30 2019] btrfs_file_write_iter+0x1ff/0x5b0
[Tue Mar 26 16:43:30 2019] __vfs_write+0xd7/0x160
[Tue Mar 26 16:43:30 2019] vfs_write+0xc5/0x1b0
[Tue Mar 26 16:43:30 2019] SyS_pwrite64+0x74/0x90
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f30fb140963
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007f30f1ffa380 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f30fb140963
[Tue Mar 26 16:43:30 2019] RDX: 0000000000020000 RSI: 00007f2fe4000d80 RDI: 0000000000000005
[Tue Mar 26 16:43:30 2019] RBP: 0000000089e80000 R08: 00007f2fe4000d80 R09: 0000000000000000
[Tue Mar 26 16:43:30 2019] R10: 0000000089e80000 R11: 0000000000000293 R12: 00007f30f1ffa5b0
[Tue Mar 26 16:43:30 2019] R13: 0000000000020000 R14: 00007f3075e80000 R15: 00007f2fe4020d80
[Tue Mar 26 16:43:30 2019] crawl_writeback D 0 29399 29318 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f30fa5c3607
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007f2fdfffdcc0 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00000000000e09c1 RCX: 00007f30fa5c3607
[Tue Mar 26 16:43:30 2019] RDX: 00000000000e09c1 RSI: 00007f2fd80008f0 RDI: 0000000000000003
[Tue Mar 26 16:43:30 2019] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007f2fdfffe080
[Tue Mar 26 16:43:30 2019] R10: 0000000000000180 R11: 0000000000000293 R12: 00007f2fd80008f0
[Tue Mar 26 16:43:30 2019] R13: 0000000000000003 R14: 00007f2fdfffe3d0 R15: 00007f2fdfffe350
[Tue Mar 26 16:43:30 2019] crawl_61509 D 0 25922 29318 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? rwsem_down_write_failed+0x134/0x2b0
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] rwsem_down_write_failed+0x139/0x2b0
[Tue Mar 26 16:43:30 2019] ? call_rwsem_down_write_failed+0x13/0x20
[Tue Mar 26 16:43:30 2019] call_rwsem_down_write_failed+0x13/0x20
[Tue Mar 26 16:43:30 2019] down_write_nested+0x87/0xb0
[Tue Mar 26 16:43:30 2019] btrfs_dedupe_file_range+0x9f/0x660
[Tue Mar 26 16:43:30 2019] ? rcu_read_lock_sched_held+0x68/0x70
[Tue Mar 26 16:43:30 2019] ? rcu_sync_lockdep_assert+0x30/0x60
[Tue Mar 26 16:43:30 2019] ? __sb_start_write+0xcc/0x1b0
[Tue Mar 26 16:43:30 2019] ? mnt_want_write_file+0x3b/0xb0
[Tue Mar 26 16:43:30 2019] vfs_dedupe_file_range+0x22e/0x280
[Tue Mar 26 16:43:30 2019] do_vfs_ioctl+0x24d/0x6b0
[Tue Mar 26 16:43:30 2019] ? __fget+0x11f/0x210
[Tue Mar 26 16:43:30 2019] SyS_ioctl+0x74/0x80
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007f30f8ce10a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00007f2fc449b570 RCX: 00007f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RDX: 00007f2fc449b570 RSI: 00000000c0189436 RDI: 0000000000000c70
[Tue Mar 26 16:43:30 2019] RBP: 00007f30f8ce1420 R08: 00007f2fc50482b0 R09: 0000000000000000
[Tue Mar 26 16:43:30 2019] R10: 00007f2fc554b720 R11: 0000000000000246 R12: 0000000000000020
[Tue Mar 26 16:43:30 2019] R13: 0000000000000018 R14: 00007f2fc50482c8 R15: 00007f30f8ce1428
[Tue Mar 26 16:43:30 2019] crawl_61509 D 0 25923 29318 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_clone+0x5a3/0xcc0
[Tue Mar 26 16:43:30 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Tue Mar 26 16:43:30 2019] vfs_dedupe_file_range+0x22e/0x280
[Tue Mar 26 16:43:30 2019] do_vfs_ioctl+0x24d/0x6b0
[Tue Mar 26 16:43:30 2019] ? __fget+0x11f/0x210
[Tue Mar 26 16:43:30 2019] SyS_ioctl+0x74/0x80
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007f30f27f8ac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00007f2fc9cb4290 RCX: 00007f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RDX: 00007f2fc9cb4290 RSI: 00000000c0189436 RDI: 0000000000000bf6
[Tue Mar 26 16:43:30 2019] RBP: 00007f30f27f8e40 R08: 00007f2fc9dbae10 R09: 0000000000000000
[Tue Mar 26 16:43:30 2019] R10: 00007f2fc861c100 R11: 0000000000000246 R12: 0000000000000020
[Tue Mar 26 16:43:30 2019] R13: 0000000000000018 R14: 00007f2fc9dbae28 R15: 00007f30f27f8e48
[Tue Mar 26 16:43:30 2019] crawl_61509 D 0 26256 29318 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_clone+0x5a3/0xcc0
[Tue Mar 26 16:43:30 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Tue Mar 26 16:43:30 2019] vfs_dedupe_file_range+0x22e/0x280
[Tue Mar 26 16:43:30 2019] do_vfs_ioctl+0x24d/0x6b0
[Tue Mar 26 16:43:30 2019] ? __fget+0x11f/0x210
[Tue Mar 26 16:43:30 2019] SyS_ioctl+0x74/0x80
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007f30f94e20a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00007f2fd5da86f0 RCX: 00007f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RDX: 00007f2fd5da86f0 RSI: 00000000c0189436 RDI: 0000000000000c6e
[Tue Mar 26 16:43:30 2019] RBP: 00007f30f94e2420 R08: 00007f2fd65725e0 R09: 0000000000000000
[Tue Mar 26 16:43:30 2019] R10: 00007f2fd41c3800 R11: 0000000000000246 R12: 0000000000000020
[Tue Mar 26 16:43:30 2019] R13: 0000000000000018 R14: 00007f2fd65725f8 R15: 00007f30f94e2428
[Tue Mar 26 16:43:30 2019] crawl_61509 D 0 26257 29318 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_clone+0x5a3/0xcc0
[Tue Mar 26 16:43:30 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Tue Mar 26 16:43:30 2019] vfs_dedupe_file_range+0x22e/0x280
[Tue Mar 26 16:43:30 2019] do_vfs_ioctl+0x24d/0x6b0
[Tue Mar 26 16:43:30 2019] ? __fget+0x11f/0x210
[Tue Mar 26 16:43:30 2019] SyS_ioctl+0x74/0x80
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007f30f2ffa0a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00007f2fcc8cdc00 RCX: 00007f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RDX: 00007f2fcc8cdc00 RSI: 00000000c0189436 RDI: 0000000000000c6f
[Tue Mar 26 16:43:30 2019] RBP: 00007f30f2ffa420 R08: 00007f2fcc79f1c0 R09: 0000000000000000
[Tue Mar 26 16:43:30 2019] R10: 00007f2fcc496010 R11: 0000000000000246 R12: 0000000000000020
[Tue Mar 26 16:43:30 2019] R13: 0000000000000018 R14: 00007f2fcc79f1d8 R15: 00007f30f2ffa428
[Tue Mar 26 16:43:30 2019] crawl_61509 D 0 26258 29318 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_clone+0x5a3/0xcc0
[Tue Mar 26 16:43:30 2019] btrfs_dedupe_file_range+0x5e3/0x660
[Tue Mar 26 16:43:30 2019] vfs_dedupe_file_range+0x22e/0x280
[Tue Mar 26 16:43:30 2019] do_vfs_ioctl+0x24d/0x6b0
[Tue Mar 26 16:43:30 2019] ? __fget+0x11f/0x210
[Tue Mar 26 16:43:30 2019] SyS_ioctl+0x74/0x80
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007f30f37faac8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00007f2fc1cd04a0 RCX: 00007f30fa5c8dd7
[Tue Mar 26 16:43:30 2019] RDX: 00007f2fc1cd04a0 RSI: 00000000c0189436 RDI: 0000000000000a3b
[Tue Mar 26 16:43:30 2019] RBP: 00007f30f37fae40 R08: 00007f2fc04365e0 R09: 0000000000000000
[Tue Mar 26 16:43:30 2019] R10: 00007f2fc00751a0 R11: 0000000000000246 R12: 0000000000000020
[Tue Mar 26 16:43:30 2019] R13: 0000000000000018 R14: 00007f2fc04365f8 R15: 00007f30f37fae48
[Tue Mar 26 16:43:30 2019] rsync D 0 22902 9371 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_setattr+0x1e6/0x3b0
[Tue Mar 26 16:43:30 2019] notify_change+0x2ed/0x450
[Tue Mar 26 16:43:30 2019] do_truncate+0x5e/0x90
[Tue Mar 26 16:43:30 2019] ? rcu_read_lock_sched_held+0x68/0x70
[Tue Mar 26 16:43:30 2019] do_sys_ftruncate.constprop.1+0x120/0x180
[Tue Mar 26 16:43:30 2019] ? trace_hardirqs_off_thunk+0x1a/0x1c
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f9a8fd2ac87
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007fff7df0a3c8 EFLAGS: 00000246 ORIG_RAX: 000000000000004d
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9a8fd2ac87
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000000 RSI: 00000000009f80ef RDI: 0000000000000004
[Tue Mar 26 16:43:30 2019] RBP: 0000000000000004 R08: 0000000000000000 R09: 00007f9a8fd757b8
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000009f80ef
[Tue Mar 26 16:43:30 2019] R13: 0000000000006dc3 R14: 0000000000000b28 R15: 0000000000000005
[Tue Mar 26 16:43:30 2019] rsync D 0 12425 12309 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] io_schedule+0x12/0x40
[Tue Mar 26 16:43:30 2019] wait_on_page_bit+0x10d/0x140
[Tue Mar 26 16:43:30 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Tue Mar 26 16:43:30 2019] read_extent_buffer_pages+0x215/0x2d0
[Tue Mar 26 16:43:30 2019] ? free_root_pointers+0x60/0x60
[Tue Mar 26 16:43:30 2019] btree_read_extent_buffer_pages+0x86/0x100
[Tue Mar 26 16:43:30 2019] read_tree_block+0x2d/0x50
[Tue Mar 26 16:43:30 2019] read_block_for_search.isra.11+0x117/0x310
[Tue Mar 26 16:43:30 2019] btrfs_search_slot+0x3e7/0x9c0
[Tue Mar 26 16:43:30 2019] btrfs_lookup_inode+0x2a/0x90
[Tue Mar 26 16:43:30 2019] btrfs_iget+0xea/0x6e0
[Tue Mar 26 16:43:30 2019] btrfs_lookup_dentry+0x3ed/0x530
[Tue Mar 26 16:43:30 2019] btrfs_lookup+0xe/0x30
[Tue Mar 26 16:43:30 2019] lookup_slow+0x131/0x210
[Tue Mar 26 16:43:30 2019] walk_component+0x1b2/0x2d0
[Tue Mar 26 16:43:30 2019] path_lookupat+0x64/0x1f0
[Tue Mar 26 16:43:30 2019] filename_lookup+0xa9/0x170
[Tue Mar 26 16:43:30 2019] ? getname_flags+0x4a/0x1e0
[Tue Mar 26 16:43:30 2019] ? __check_object_size+0xa7/0x190
[Tue Mar 26 16:43:30 2019] ? strncpy_from_user+0x48/0x160
[Tue Mar 26 16:43:30 2019] ? lockref_put_or_lock+0x9/0x30
[Tue Mar 26 16:43:30 2019] ? vfs_statx+0x63/0xb0
[Tue Mar 26 16:43:30 2019] vfs_statx+0x63/0xb0
[Tue Mar 26 16:43:30 2019] ? mntput_no_expire+0x5/0x410
[Tue Mar 26 16:43:30 2019] SyS_newlstat+0x2b/0x50
[Tue Mar 26 16:43:30 2019] ? trace_hardirqs_off_thunk+0x1a/0x1c
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7ffaaf50b0f5
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffc8acb9fb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000006
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ffaaf50b0f5
[Tue Mar 26 16:43:30 2019] RDX: 00007ffc8acba0f0 RSI: 00007ffc8acba0f0 RDI: 00007ffc8acba180
[Tue Mar 26 16:43:30 2019] RBP: 0000000000000000 R08: 00007ffc8acba1e3 R09: 0000000000000000
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc8acba180
[Tue Mar 26 16:43:30 2019] R13: 00007ffc8acba0f0 R14: 0000000000000004 R15: 0000000000000002
[Tue Mar 26 16:43:30 2019] kworker/u8:14 D 0 13297 2 0x80000000
[Tue Mar 26 16:43:30 2019] Workqueue: btrfs-flush_delalloc btrfs_flush_delalloc_helper
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] io_schedule+0x12/0x40
[Tue Mar 26 16:43:30 2019] __lock_page+0x110/0x140
[Tue Mar 26 16:43:30 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Tue Mar 26 16:43:30 2019] extent_write_cache_pages.constprop.27+0x2d9/0x3b0
[Tue Mar 26 16:43:30 2019] ? lock_acquire+0xbc/0x200
[Tue Mar 26 16:43:30 2019] extent_writepages+0x4e/0x70
[Tue Mar 26 16:43:30 2019] ? btrfs_direct_IO+0x3e0/0x3e0
[Tue Mar 26 16:43:30 2019] do_writepages+0x23/0x80
[Tue Mar 26 16:43:30 2019] __filemap_fdatawrite_range+0xaa/0xe0
[Tue Mar 26 16:43:30 2019] btrfs_run_delalloc_work+0x17/0x60
[Tue Mar 26 16:43:30 2019] normal_work_helper+0x89/0x640
[Tue Mar 26 16:43:30 2019] process_one_work+0x204/0x650
[Tue Mar 26 16:43:30 2019] worker_thread+0x4d/0x3b0
[Tue Mar 26 16:43:30 2019] kthread+0x10d/0x140
[Tue Mar 26 16:43:30 2019] ? process_one_work+0x650/0x650
[Tue Mar 26 16:43:30 2019] ? kthread_create_on_node+0x40/0x40
[Tue Mar 26 16:43:30 2019] ret_from_fork+0x27/0x50
[Tue Mar 26 16:43:30 2019] rsync D 0 9359 9358 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f544c09b4b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffe6af72748 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f544c09b4b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffe6af749b0
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f544c0ee520
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe6af749ca
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f544c129320
[Tue Mar 26 16:43:30 2019] rsync D 0 8086 7962 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_start_transaction_fallback_global_rsv+0x34/0x200
[Tue Mar 26 16:43:30 2019] btrfs_unlink+0x30/0xd0
[Tue Mar 26 16:43:30 2019] vfs_unlink+0xf8/0x190
[Tue Mar 26 16:43:30 2019] do_unlinkat+0x287/0x300
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f9fa27c4ea7
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffd1cf9a008 EFLAGS: 00000246 ORIG_RAX: 0000000000000057
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f9fa27c4ea7
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000000 RSI: 00000000000081a4 RDI: 00007ffd1cf9b0c0
[Tue Mar 26 16:43:30 2019] RBP: 00000000000081a4 R08: 000000000000fcff R09: 00007f9fa2816100
[Tue Mar 26 16:43:30 2019] R10: 2f6d762f746f6f72 R11: 0000000000000246 R12: 00007ffd1cf9b0c0
[Tue Mar 26 16:43:30 2019] R13: 0000000000008000 R14: 0000000000000000 R15: 00000000000081a4
[Tue Mar 26 16:43:30 2019] kworker/u8:10 D 0 27109 2 0x80000000
[Tue Mar 26 16:43:30 2019] Workqueue: writeback wb_workfn (flush-btrfs-2)
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] io_schedule+0x12/0x40
[Tue Mar 26 16:43:30 2019] __lock_page+0x110/0x140
[Tue Mar 26 16:43:30 2019] ? add_to_page_cache_lru+0xc0/0xc0
[Tue Mar 26 16:43:30 2019] extent_write_cache_pages.constprop.27+0x2d9/0x3b0
[Tue Mar 26 16:43:30 2019] ? writeback_sb_inodes+0x123/0x550
[Tue Mar 26 16:43:30 2019] extent_writepages+0x4e/0x70
[Tue Mar 26 16:43:30 2019] ? btrfs_direct_IO+0x3e0/0x3e0
[Tue Mar 26 16:43:30 2019] do_writepages+0x23/0x80
[Tue Mar 26 16:43:30 2019] __writeback_single_inode+0x54/0x740
[Tue Mar 26 16:43:30 2019] writeback_sb_inodes+0x26e/0x550
[Tue Mar 26 16:43:30 2019] __writeback_inodes_wb+0x87/0xb0
[Tue Mar 26 16:43:30 2019] wb_writeback+0x3e9/0x550
[Tue Mar 26 16:43:30 2019] ? wb_workfn+0x93/0x5c0
[Tue Mar 26 16:43:30 2019] wb_workfn+0x93/0x5c0
[Tue Mar 26 16:43:30 2019] ? lock_acquire+0xbc/0x200
[Tue Mar 26 16:43:30 2019] process_one_work+0x204/0x650
[Tue Mar 26 16:43:30 2019] worker_thread+0x4d/0x3b0
[Tue Mar 26 16:43:30 2019] kthread+0x10d/0x140
[Tue Mar 26 16:43:30 2019] ? process_one_work+0x650/0x650
[Tue Mar 26 16:43:30 2019] ? kthread_create_on_node+0x40/0x40
[Tue Mar 26 16:43:30 2019] ret_from_fork+0x27/0x50
[Tue Mar 26 16:43:30 2019] rsync D 0 1942 1935 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f71546634b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffc31490108 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f71546634b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc31492370
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f71546b5d30
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc31492382
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f71546f1320
[Tue Mar 26 16:43:30 2019] rsync D 0 4476 4465 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f91a26c34b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffe9c29d2d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f91a26c34b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffe9c29f540
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f91a2715950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe9c29f557
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f91a2751320
[Tue Mar 26 16:43:30 2019] mkdir D 0 4816 4813 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_mkdir+0x50/0x1d0
[Tue Mar 26 16:43:30 2019] vfs_mkdir+0x134/0x1c0
[Tue Mar 26 16:43:30 2019] SyS_mkdirat+0xb3/0xf0
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f36b3ccb447
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffef220ed58 EFLAGS: 00000206 ORIG_RAX: 0000000000000053
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00007ffef2210b28 RCX: 00007f36b3ccb447
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007ffef2210b4d
[Tue Mar 26 16:43:30 2019] RBP: 00007ffef220eed0 R08: 00000000000001ff R09: 0000559b36eafe10
[Tue Mar 26 16:43:30 2019] R10: 0000000000000376 R11: 0000000000000206 R12: 0000559b36eafe50
[Tue Mar 26 16:43:30 2019] R13: 00000000000001ff R14: 00007ffef220f070 R15: 0000000000000001
[Tue Mar 26 16:43:30 2019] rsync D 0 6356 6343 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7fd4586134b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffd1348b1a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fd4586134b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffd1348d410
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007fd458665950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd1348d427
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fd4586a1320
[Tue Mar 26 16:43:30 2019] rsync D 0 6626 6625 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7fb8e7f0b4b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffd490ae4b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb8e7f0b4b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffd490b0720
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007fb8e7f5d950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd490b0737
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007fb8e7f99320
[Tue Mar 26 16:43:30 2019] rsync D 0 17256 17192 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7ff14b15b4b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffc30512e28 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff14b15b4b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc30515090
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000fff8 R09: 00007ff14b1adf10
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc305150ad
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007ff14b1e9320
[Tue Mar 26 16:43:30 2019] rsync D 0 29615 29614 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? __switch_to_asm+0x40/0x70
[Tue Mar 26 16:43:30 2019] ? __switch_to_asm+0x34/0x70
[Tue Mar 26 16:43:30 2019] ? __switch_to_asm+0x40/0x70
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f8c4eb534b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffe290f35b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f8c4eb534b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffe290f5820
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000ffff R09: 00007f8c4eba6520
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe290f5833
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f8c4ebe1320
[Tue Mar 26 16:43:30 2019] rsync D 0 7453 7452 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f444631b4b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffdf129fa38 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f444631b4b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffdf12a1ca0
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f444636d950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffdf12a1cb7
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f44463a9320
[Tue Mar 26 16:43:30 2019] rsync D 0 29610 29604 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_mkdir+0x50/0x1d0
[Tue Mar 26 16:43:30 2019] vfs_mkdir+0x134/0x1c0
[Tue Mar 26 16:43:30 2019] SyS_mkdirat+0xb3/0xf0
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f0d5147b447
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007fff61e45d08 EFLAGS: 00000202 ORIG_RAX: 0000000000000053
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00000000000001c0 RCX: 00007f0d5147b447
[Tue Mar 26 16:43:30 2019] RDX: fec4ff0000000000 RSI: 00000000000001c0 RDI: 00007fff61e48060
[Tue Mar 26 16:43:30 2019] RBP: 00007f0d51f3d7a0 R08: 000000000000ff00 R09: 00000000ffffffff
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
[Tue Mar 26 16:43:30 2019] R13: 0000000000000000 R14: 00007fff61e48060 R15: 00000000ffffffff
[Tue Mar 26 16:43:30 2019] rsync D 0 29638 29610 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f0d5147b4b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007fff61e44df8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0d5147b4b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007fff61e47060
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000ffff R09: 00007f0d514cdd40
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff61e47075
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f0d51509320
[Tue Mar 26 16:43:30 2019] rsync D 0 10214 10213 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f73b40db4b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffc80e88938 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f73b40db4b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc80e8aba0
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f73b412d950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc80e8abb7
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f73b4169320
[Tue Mar 26 16:43:30 2019] rsync D 0 29837 29758 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f3dd5d734b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffdceac6b38 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f3dd5d734b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffdceac8da0
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 0000000000000000 R09: 00007f3dd5dc6100
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffdceac8dcb
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f3dd5e01320
[Tue Mar 26 16:43:30 2019] rsync D 0 31885 31792 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f80265f34b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007fffeeff2558 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f80265f34b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007fffeeff47c0
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c000 R09: 00007f8026646100
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fffeeff47e8
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f8026681320
[Tue Mar 26 16:43:30 2019] rsync D 0 7092 7091 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f26f47eb4b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffd94a0fe08 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f26f47eb4b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffd94a12070
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f26f483d950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd94a12087
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f26f4879320
[Tue Mar 26 16:43:30 2019] rsync D 0 10725 10712 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7ff22e1c34b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffc04b5c158 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff22e1c34b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffc04b5e3c0
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007ff22e215950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc04b5e3d7
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007ff22e251320
[Tue Mar 26 16:43:30 2019] rsync D 0 27579 27578 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f046d3a34b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffca2430798 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f046d3a34b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffca2432a00
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000c0ff R09: 00007f046d3f5950
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffca2432a17
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f046d431320
[Tue Mar 26 16:43:30 2019] rsync D 0 27949 27856 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_create+0x47/0x1d0
[Tue Mar 26 16:43:30 2019] lookup_open+0x404/0x760
[Tue Mar 26 16:43:30 2019] ? __wake_up_common_lock+0x53/0x90
[Tue Mar 26 16:43:30 2019] path_openat+0x4e9/0xcc0
[Tue Mar 26 16:43:30 2019] ? filemap_map_pages+0x312/0x520
[Tue Mar 26 16:43:30 2019] do_filp_open+0x8a/0xf0
[Tue Mar 26 16:43:30 2019] ? __alloc_fd+0xee/0x200
[Tue Mar 26 16:43:30 2019] ? do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_sys_open+0x130/0x200
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f7421ff34b0
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffe22c079d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000002
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7421ff34b0
[Tue Mar 26 16:43:30 2019] RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007ffe22c09c40
[Tue Mar 26 16:43:30 2019] RBP: 000000000003a2f8 R08: 000000000000ffff R09: 00007f7422045b30
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe22c09c80
[Tue Mar 26 16:43:30 2019] R13: 8421084210842109 R14: 00000000000000c2 R15: 00007f7422081320
[Tue Mar 26 16:43:30 2019] rsync D 0 7504 7417 0x00000000
[Tue Mar 26 16:43:30 2019] Call Trace:
[Tue Mar 26 16:43:30 2019] ? __schedule+0x429/0xbb0
[Tue Mar 26 16:43:30 2019] ? prepare_to_wait_event+0x97/0x190
[Tue Mar 26 16:43:30 2019] schedule+0x39/0x90
[Tue Mar 26 16:43:30 2019] wait_current_trans+0xa3/0xf0
[Tue Mar 26 16:43:30 2019] ? wait_woken+0xa0/0xa0
[Tue Mar 26 16:43:30 2019] start_transaction+0x29b/0x4d0
[Tue Mar 26 16:43:30 2019] btrfs_mkdir+0x50/0x1d0
[Tue Mar 26 16:43:30 2019] vfs_mkdir+0x134/0x1c0
[Tue Mar 26 16:43:30 2019] SyS_mkdirat+0xb3/0xf0
[Tue Mar 26 16:43:30 2019] do_syscall_64+0x76/0x180
[Tue Mar 26 16:43:30 2019] entry_SYSCALL_64_after_hwframe+0x42/0xb7
[Tue Mar 26 16:43:30 2019] RIP: 0033:0x7f92bc98b447
[Tue Mar 26 16:43:30 2019] RSP: 002b:00007ffde7124dd8 EFLAGS: 00000202 ORIG_RAX: 0000000000000053
[Tue Mar 26 16:43:30 2019] RAX: ffffffffffffffda RBX: 00000000000001c0 RCX: 00007f92bc98b447
[Tue Mar 26 16:43:30 2019] RDX: 0c00000000000000 RSI: 00000000000001c0 RDI: 00007ffde7127130
[Tue Mar 26 16:43:30 2019] RBP: 000055b32c935790 R08: 0000000000000000 R09: 00000000ffffffff
[Tue Mar 26 16:43:30 2019] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000002
[Tue Mar 26 16:43:30 2019] R13: 0000000000000000 R14: 00007ffde7127130 R15: 00000000ffffffff
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: A collection of btrfs lockup stack traces (4.14..4.20.17, but not later kernels)
2019-03-20 3:49 ` A collection of btrfs lockup stack traces (4.20.13) Zygo Blaxell
@ 2020-02-04 3:34 ` Zygo Blaxell
0 siblings, 0 replies; 6+ messages in thread
From: Zygo Blaxell @ 2020-02-04 3:34 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2639 bytes --]
On Tue, Mar 19, 2019 at 11:49:51PM -0400, Zygo Blaxell wrote:
> On Tue, Mar 19, 2019 at 11:39:59PM -0400, Zygo Blaxell wrote:
> > I haven't been able to easily reproduce these in a test environment;
> > however, they have been happening several times a year on servers in
> > production.
> >
> > Kernel: most recent observation on 4.14.105 + cherry-picked deadlock
> > and misc hang fixes:
> >
> > btrfs: wakeup cleaner thread when adding delayed iput
> > Btrfs: fix deadlock when allocating tree block during leaf/node split
> > Btrfs: use nofs context when initializing security xattrs to avoid deadlock
> > Btrfs: fix deadlock with memory reclaim during scrub
> > Btrfs: fix deadlock between clone/dedupe and rename
> >
> > Also observed on 4.20.13, and 4.14.0..4.14.105 (4.14.106 is currently
> > running, but hasn't locked up yet).
> >
> > Filesystem mount flags: compress=zstd,ssd,flushoncommit,space_cache=v2.
> > Configuration is either -draid1/-mraid1 or -dsingle/-mraid1. I've
> > also reproduced a lockup without flushoncommit.
> >
> > The machines that are locking up all run the same workload:
> >
> > rsync receiving data continuously (gigabytes aren't enough,
> > I can barely reproduce this once a month with 2TB of rsync
> > traffic from 10 simulated clients)
> >
> > bees doing continuous dedupe
> >
> > snapshots daily and after each rsync
> >
> > snapshot deletes as required to maintain free space
> >
> > scrubs twice monthly plus after each crash
> >
> > watchdog does a 'mkdir foo; rmdir foo' every few seconds.
> > If this takes longer than 50 minutes, collect a stack trace;
> > longer than 60 minutes, reboot the machine.
These deadlocks still occur on the LTS kernels 4.14 and 4.19 (I have
not tested earlier LTS kernels). The deadlocks also occur on 4.15..4.18
and 4.20, but the deadlocks stopped on 5.0 and haven't occurred since.
I'm running a bisect to see if I can find where in 5.0 this was fixed,
and whether it is something that can be backported to stable. This might
take a month or so, as it takes a few days to get a negative deadlock
result and I can only spare one VM for the tests.
In the meantime several other bugs have been fixed and those _have_
been backported to 4.14 and 4.19, so the test case for the bisect is
more aggressive:
10x rsync write/updates
bees dedupe, 4 threads
snapshot create at random intervals (60-600 seconds)
snapshot delete at random intervals
balance start continuously
balance cancel at random intervals
scrub start continuously
scrub cancel at random intervals
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-02-04 3:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-03-20 3:39 A collection of btrfs lockup stack traces Zygo Blaxell
2019-03-20 3:45 ` Zygo Blaxell
2019-03-20 3:49 ` A collection of btrfs lockup stack traces (4.20.13) Zygo Blaxell
2020-02-04 3:34 ` A collection of btrfs lockup stack traces (4.14..4.20.17, but not later kernels) Zygo Blaxell
2019-03-27 2:37 ` A collection of btrfs lockup stack traces (4.14.106, no __sb_start_write) Zygo Blaxell
2019-03-27 20:04 ` A collection of btrfs lockup stack traces (4.14.107, kcompactd and two processes running __lock_page) Zygo Blaxell
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.