From: Drazen Kacar <drazen.kacar@oradian.com>
To: "linux-btrfs@vger.kernel.org" <linux-btrfs@vger.kernel.org>
Subject: Re: Deadlock between btrfs-transacti and userland on 5.2.1
Date: Fri, 19 Jul 2019 12:12:51 +0000 [thread overview]
Message-ID: <DB8PR03MB5628B467AB2EC04D41DB115599CB0@DB8PR03MB5628.eurprd03.prod.outlook.com> (raw)
In-Reply-To: <DB8PR03MB562876ECE2319B3E579590F799C80@DB8PR03MB5628.eurprd03.prod.outlook.com>
[-- Attachment #1: Type: text/plain, Size: 215 bytes --]
I've got more hung tasks. Log is attached. Several empty lines
in the file mean that there was a reboot in between.
btrfs scrub -Bdr says there are no errors on disks.
btrfs device stats also shows no errors.
[-- Attachment #2: messages_more_hungs.txt --]
[-- Type: text/plain, Size: 60687 bytes --]
Jul 19 11:47:36 prod-dbsnap-01 systemd: Started PostgreSQL snapshot service for ardcimicrofinance_6432.
Jul 19 11:47:36 prod-dbsnap-01 systemd: Reloading.
Jul 19 11:47:36 prod-dbsnap-01 systemd: Binding to IPv6 address not available since kernel does not support IPv6.
Jul 19 11:47:36 prod-dbsnap-01 systemd: Binding to IPv6 address not available since kernel does not support IPv6.
Jul 19 11:50:01 prod-dbsnap-01 systemd: Created slice User Slice of root.
Jul 19 11:50:01 prod-dbsnap-01 systemd: Started Session 241 of user root.
Jul 19 11:50:01 prod-dbsnap-01 systemd: Removed slice User Slice of root.
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:20710 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 20710 20672 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7fc16f67bd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffee0d14198 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000000f9 RCX: 00007fc16f67bd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007fc150c5c580 RDI: 00000000000000f9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 00007fc16fa877f0 R08: 00007fc150c5c690 R09: 00007fc150c5c588
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000019abc10bce0
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 000000000206dba0 R15: 0000000000000f9d
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:28814 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 28814 28787 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f1e421e4d00
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffc18ebe718 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000003e0 RCX: 00007f1e421e4d00
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f1e237c5580 RDI: 00000000000003e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 00007f1e425f07f0 R08: 00007f1e237c5690 R09: 00007f1e237c5588
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000006053e93948
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 0000000000e21f20 R15: 0000000000003042
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:31379 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 31379 28787 0x00004004
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: io_schedule+0x16/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_on_page_bit+0x1c0/0x270
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? file_fdatawait_range+0x30/0x30
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_on_page_writeback+0x48/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: extent_write_cache_pages+0x214/0x4c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: extent_writepages+0x4f/0x90 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? tcp_push+0xde/0x110
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_writepages+0xe/0x10 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_writepages+0x1f/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __filemap_fdatawrite_range+0xc6/0x100
Jul 19 11:52:01 prod-dbsnap-01 kernel: filemap_fdatawrite_range+0x13/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_fdatawrite_range+0x20/0x50 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_ordered_ops+0x4c/0x90 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __sys_sendto+0x107/0x1a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0xce/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f1e3fd58310
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffc18ebf888 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000006053 RCX: 00007f1e3fd58310
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000006053 RDI: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000e208e9 R14: 00007f1e2fbb9280 R15: 0000000000001e00
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:31526 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 31526 20672 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7fc16d1ef310
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffee0d15308 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000019abc RCX: 00007fc16d1ef310
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000019abc RDI: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 000000000208a5d9 R14: 00007fc15d050280 R15: 0000000000012000
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:36718 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 36718 36697 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f577cf4cd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc0a03fb8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f577cf4cd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f575e52d580 RDI: 0000000000000003
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 00007f577d3587f0 R08: 00007f575e52d690 R09: 00007f575e52d588
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 00000000000032c8 R11: 0000000000000246 R12: 000000930ca33770
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 0000009308e80f30 R15: 0000000000000001
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:41263 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 41263 36697 0x00004004
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f577aac0310
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc0a03bc8 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 000000000000930c RCX: 00007f577aac0310
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 000000000000930c RDI: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 00000000015392c9 R14: 00007f576a921280 R15: 00000000000000e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:5:118303 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:5 D 0 118303 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:4:6869 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:4 D 0 6869 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:14:8117 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:14 D 0 8117 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:6:10389 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:6 D 0 10389 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? syscall_return_via_sysret+0xf/0x7f
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:1840 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 1840 1796 0x00000000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: rwsem_down_write_failed+0x1f3/0x380
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? new_sync_read+0x11a/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: down_write+0x3e/0x40
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_file_llseek+0x4c/0x2b0 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? vfs_read+0x8e/0x140
Jul 19 12:20:13 prod-dbsnap-01 kernel: ksys_lseek+0x8a/0xb0
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_lseek+0x1a/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e7ff0d60
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c1358 EFLAGS: 00000246 ORIG_RAX: 0000000000000008
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000390000 RCX: 00007f49e7ff0d60
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000000 RSI: 0000000000390000 RDI: 0000000000000003
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 0000019ac1390000 R08: 0000000002c29758 R09: 0000000002c2050c
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 0000000002c2b730 R11: 0000000000000246 R12: 0000000000000003
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000000390000 R14: 0000000002c2050c R15: 0000000000000026
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2037 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2037 1801 0x00004000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f866ab96d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc52426c8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000341 RCX: 00007f866ab96d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f864c177580 RDI: 0000000000000341
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 00007f866afa27f0 R08: 00007f864c177690 R09: 00007f864c177588
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 00000093121d6de0
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 0000000000000058 R15: 0000000000001eae
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2061 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2061 1808 0x00004000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7fb113087d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007fffba1597c8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000003e0 RCX: 00007fb113087d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007fb0f4668580 RDI: 00000000000003e0
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 00007fb1134937f0 R08: 00007fb0f4668690 R09: 00007fb0f4668588
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 0000006055e08180
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 000000000000126f R15: 000000000000158d
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2135 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2135 1801 0x00004004
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: io_schedule+0x16/0x40
Jul 19 12:20:13 prod-dbsnap-01 kernel: wait_on_page_bit+0x1c0/0x270
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? file_fdatawait_range+0x30/0x30
Jul 19 12:20:13 prod-dbsnap-01 kernel: wait_on_page_writeback+0x48/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: extent_write_cache_pages+0x214/0x4c0 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: extent_writepages+0x4f/0x90 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? tcp_push+0xde/0x110
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_writepages+0xe/0x10 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_writepages+0x1f/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __filemap_fdatawrite_range+0xc6/0x100
Jul 19 12:20:13 prod-dbsnap-01 kernel: filemap_fdatawrite_range+0x13/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_fdatawrite_range+0x20/0x50 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: start_ordered_ops+0x4c/0x90 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? __sys_sendto+0x107/0x1a0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0xce/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f866870a310
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc5243748 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000009312 RCX: 00007f866870a310
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000009312 RDI: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000002c2a009 R14: 00007f865856b280 R15: 0000000000002430
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2165 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2165 1796 0x00004000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e5b64310
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c4088 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000019ac1 RCX: 00007f49e5b64310
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000019ac1 RDI: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000002c370c9 R14: 00007f49d59c5280 R15: 000000000000abb0
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:1961 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 1961 1807 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f99ee448d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffdb772d008 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000389 RCX: 00007f99ee448d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 00007f99df2a8070 RSI: 000000000a00000e RDI: 000000000000038f
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 000000000a00000e R08: 000000000007cc24 R09: 00007ffdb77810b0
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 000000000000010c R11: 0000000000000246 R12: 0000000000000009
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 00000000010ee588 R14: 00000000010a7190 R15: 00000000010a7190
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2032 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2032 1796 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e7ff0d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c3008 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000003e0 RCX: 00007f49e7ff0d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f49c95d1580 RDI: 00000000000003e0
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 00007f49e83fc7f0 R08: 00007f49c95d1690 R09: 00007f49c95d1588
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 0000019ac2a59b20
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 00000000000002de R15: 0000000000000307
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2061 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2061 1808 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: lock_and_cleanup_extent_if_need+0x1b7/0x230 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_buffered_write.isra.25+0x22e/0x710 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_file_write_iter+0x20d/0x580 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: new_sync_write+0x120/0x1a0
Jul 19 12:24:18 prod-dbsnap-01 kernel: __vfs_write+0x29/0x40
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_write+0xb2/0x1b0
Jul 19 12:24:18 prod-dbsnap-01 kernel: ksys_write+0x61/0xd0
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_write+0x1a/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7fb113087680
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007fffba159788 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00007fb1133e4ad0 RCX: 00007fb113087680
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000002000 RSI: 00000000014bb758 RDI: 00000000000003a7
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 0000000000002000 R08: 0000000000000009 R09: 0000000000000000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 0000000000002000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 00000000014bb758 R14: 000000000a000010 R15: 00007fb1134937f0
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2444 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2444 1796 0x00004004
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: io_schedule+0x16/0x40
Jul 19 12:24:18 prod-dbsnap-01 kernel: wait_on_page_bit+0x1c0/0x270
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? file_fdatawait_range+0x30/0x30
Jul 19 12:24:18 prod-dbsnap-01 kernel: wait_on_page_writeback+0x48/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: extent_write_cache_pages+0x214/0x4c0 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: extent_writepages+0x4f/0x90 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? tcp_push+0xde/0x110
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_writepages+0xe/0x10 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_writepages+0x1f/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __filemap_fdatawrite_range+0xc6/0x100
Jul 19 12:24:18 prod-dbsnap-01 kernel: filemap_fdatawrite_range+0x13/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_fdatawrite_range+0x20/0x50 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: start_ordered_ops+0x4c/0x90 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? __sys_sendto+0x107/0x1a0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0xce/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e5b64310
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c4088 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000019ac2 RCX: 00007f49e5b64310
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000019ac2 RDI: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 0000000002c208f9 R14: 00007f49d59c5280 R15: 0000000000003d20
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2445 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2445 1801 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f866870a310
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc5243748 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000009312 RCX: 00007f866870a310
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000009312 RDI: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 0000000002c30571 R14: 00007f865856b280 R15: 0000000000000038
Jul 19 13:02:23 prod-dbsnap-01 kernel: INFO: task btrfs-cleaner:1400 blocked for more than 122 seconds.
Jul 19 13:02:23 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:02:23 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs-cleaner D 0 1400 2 0x80004000
Jul 19 13:02:23 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:02:23 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:02:23 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:02:23 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:02:23 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_drop_snapshot+0x500/0x960 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_clean_one_deleted_snapshot+0xcb/0x120 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: cleaner_kthread+0xeb/0x140 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? btree_invalidatepage+0xd0/0xd0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:02:23 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:02:23 prod-dbsnap-01 kernel: INFO: task postgres:3423 blocked for more than 122 seconds.
Jul 19 13:02:23 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:02:23 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:02:23 prod-dbsnap-01 kernel: postgres D 0 3423 3269 0x00004004
Jul 19 13:02:23 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:02:23 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:02:23 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:02:23 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:02:23 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_sync_file+0x335/0x410 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 13:02:23 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 13:02:23 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 13:02:23 prod-dbsnap-01 kernel: RIP: 0033:0x7fb03c2f2310
Jul 19 13:02:23 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 13:02:23 prod-dbsnap-01 kernel: RSP: 002b:00007ffdbacb2988 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 13:02:23 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 000000000000931e RCX: 00007fb03c2f2310
Jul 19 13:02:23 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 000000000000931e RDI: 0000000000000004
Jul 19 13:02:23 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 13:02:23 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 13:02:23 prod-dbsnap-01 kernel: R13: 0000000000ee19f1 R14: 00007fb02c153280 R15: 0000000000000f08
Jul 19 13:02:23 prod-dbsnap-01 kernel: INFO: task postgres:4039 blocked for more than 122 seconds.
Jul 19 13:02:23 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:02:23 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:02:23 prod-dbsnap-01 kernel: postgres D 0 4039 4036 0x00004000
Jul 19 13:02:23 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:02:23 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:02:23 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:02:23 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:02:23 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_sync_file+0x335/0x410 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 13:02:23 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 13:02:23 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 13:02:23 prod-dbsnap-01 kernel: RIP: 0033:0x7f31f038bd00
Jul 19 13:02:23 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 13:02:23 prod-dbsnap-01 kernel: RSP: 002b:00007ffff1c80508 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 13:02:23 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f31f038bd00
Jul 19 13:02:23 prod-dbsnap-01 kernel: RDX: 0000000000000180 RSI: 0000000000000002 RDI: 0000000000000003
Jul 19 13:02:23 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 4231303030303030 R09: 00007f31ede5720d
Jul 19 13:02:23 prod-dbsnap-01 kernel: R10: 00007f31ee1d1c8c R11: 0000000000000246 R12: 00007ffff1c80ed0
Jul 19 13:02:23 prod-dbsnap-01 kernel: R13: 0000000000000000 R14: 0000000000000014 R15: 00007ffff1c80ed0
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task btrfs-cleaner:1400 blocked for more than 245 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs-cleaner D 0 1400 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_drop_snapshot+0x500/0x960 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_clean_one_deleted_snapshot+0xcb/0x120 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: cleaner_kthread+0xeb/0x140 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? btree_invalidatepage+0xd0/0xd0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:3:2678 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:3 D 0 2678 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:8:2683 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:8 D 0 2683 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:9:2684 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:9 D 0 2684 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:13:3321 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:13 D 0 3321 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task postgres:3423 blocked for more than 245 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: postgres D 0 3423 3269 0x00004004
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_sync_file+0x335/0x410 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 13:04:26 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 13:04:26 prod-dbsnap-01 kernel: RIP: 0033:0x7fb03c2f2310
Jul 19 13:04:26 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 13:04:26 prod-dbsnap-01 kernel: RSP: 002b:00007ffdbacb2988 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 13:04:26 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 000000000000931e RCX: 00007fb03c2f2310
Jul 19 13:04:26 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 000000000000931e RDI: 0000000000000004
Jul 19 13:04:26 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 13:04:26 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 13:04:26 prod-dbsnap-01 kernel: R13: 0000000000ee19f1 R14: 00007fb02c153280 R15: 0000000000000f08
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:1:3787 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:1 D 0 3787 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
next prev parent reply other threads:[~2019-07-19 12:13 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <DB8PR03MB56287A9E527898E5727BB41599C80@DB8PR03MB5628.eurprd03.prod.outlook.com>
2019-07-18 12:23 ` Deadlock between btrfs-transacti and userland on 5.2.1 Drazen Kacar
2019-07-19 12:12 ` Drazen Kacar [this message]
2019-09-11 14:59 ` Filipe Manana
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DB8PR03MB5628B467AB2EC04D41DB115599CB0@DB8PR03MB5628.eurprd03.prod.outlook.com \
--to=drazen.kacar@oradian.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).