* Strange XFS problem
@ 2018-09-12 8:07 Troels Hansen
2018-09-12 10:59 ` Carlos Maiolino
2018-09-13 4:19 ` Dave Chinner
0 siblings, 2 replies; 19+ messages in thread
From: Troels Hansen @ 2018-09-12 8:07 UTC (permalink / raw)
To: linux-xfs
Hi, we are facing an issue where we can't figure out if its XFS software related, or actually related to hardware, and can't quite figure out why we are facing the issues, though is doesn't seem hardware related.
The issue is with a 102Tb array on a Dell branded LSISAS 3508 (Perc H840).
Running Ubuntu with a 4.15.0-32 (Ubuntu branded), but we have also been running a number of 4.4.0-x with the same issues.
The XFS filsusyem is on a very busy NFS server, and when the issue occurs we see strange issues with NFS, while the system seems healthy on the local server, but at the same time some programs are having problems accessing the fs.
It occure roughly every 14 days, where we have to restart the fs to come back fully working.
Sometimes refusing to unmount cleanly during shutdown, forcing us to fsck the fs on startup.
It looks like its hanging in xlog_grant_head_wait, but I don't know enough to determine what can make it hang there.
Hoping someone in here could have a look and point me in the right direction.
Below is a trace from the last crash we had:
Sep 9 23:23:51 ged kernel: [1436769.178935] INFO: task mysqld:2847 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.178999] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.179047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.179105] mysqld D 0 2847 1 0x00000000
Sep 9 23:23:51 ged kernel: [1436769.179111] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.179123] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.179127] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.179216] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179277] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179330] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179381] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179428] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179478] xfs_vn_update_time+0x5d/0x160 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179486] file_update_time+0xbe/0x110
Sep 9 23:23:51 ged kernel: [1436769.179493] ? tcp_recvmsg+0x317/0xab0
Sep 9 23:23:51 ged kernel: [1436769.179542] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179588] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179632] xfs_file_write_iter+0x103/0x150 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179637] new_sync_write+0xe5/0x140
Sep 9 23:23:51 ged kernel: [1436769.179641] __vfs_write+0x29/0x40
Sep 9 23:23:51 ged kernel: [1436769.179645] vfs_write+0xb8/0x1b0
Sep 9 23:23:51 ged kernel: [1436769.179649] SyS_pwrite64+0x95/0xb0
Sep 9 23:23:51 ged kernel: [1436769.179655] do_syscall_64+0x73/0x130
Sep 9 23:23:51 ged kernel: [1436769.179661] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Sep 9 23:23:51 ged kernel: [1436769.179664] RIP: 0033:0x7f16be838da3
Sep 9 23:23:51 ged kernel: [1436769.179667] RSP: 002b:00007f148c31b950 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
Sep 9 23:23:51 ged kernel: [1436769.179671] RAX: ffffffffffffffda RBX: 00007f148c31ba50 RCX: 00007f16be838da3
Sep 9 23:23:51 ged kernel: [1436769.179673] RDX: 0000000000000200 RSI: 00007f149ea80200 RDI: 0000000000000008
Sep 9 23:23:51 ged kernel: [1436769.179675] RBP: 00007f148c31bc00 R08: 000000000019d200 R09: 00007f148c31bc3c
Sep 9 23:23:51 ged kernel: [1436769.179677] R10: 000000000019d200 R11: 0000000000000293 R12: 0000000000000200
Sep 9 23:23:51 ged kernel: [1436769.179679] R13: 000000000019d200 R14: 0000000000000000 R15: 0000000000000000
Sep 9 23:23:51 ged kernel: [1436769.179686] INFO: task nfsd:1937 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.179738] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.179784] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.179842] nfsd D 0 1937 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.179846] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.179852] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.179857] ? sch_direct_xmit+0x73/0x1b0
Sep 9 23:23:51 ged kernel: [1436769.179860] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.179910] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.179956] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180005] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180051] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180096] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180145] xfs_vn_update_time+0x5d/0x160 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180150] file_update_time+0xbe/0x110
Sep 9 23:23:51 ged kernel: [1436769.180199] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180244] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180248] ? _cond_resched+0x1a/0x50
Sep 9 23:23:51 ged kernel: [1436769.180291] xfs_file_write_iter+0x103/0x150 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180295] do_iter_readv_writev+0x111/0x180
Sep 9 23:23:51 ged kernel: [1436769.180299] do_iter_write+0x87/0x1a0
Sep 9 23:23:51 ged kernel: [1436769.180302] vfs_iter_write+0x19/0x30
Sep 9 23:23:51 ged kernel: [1436769.180321] nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.180338] nfsd4_write+0x16c/0x1a0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.180352] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.180365] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.180395] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.180416] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.180428] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.180434] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.180445] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.180451] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.180456] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.180461] INFO: task nfsd:1938 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.180511] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.180557] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.180615] nfsd D 0 1938 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.180619] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.180623] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.180628] ? sch_direct_xmit+0x73/0x1b0
Sep 9 23:23:51 ged kernel: [1436769.180631] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.180689] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180737] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180786] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180832] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180876] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180926] xfs_vn_update_time+0x5d/0x160 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.180931] file_update_time+0xbe/0x110
Sep 9 23:23:51 ged kernel: [1436769.180980] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181025] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181029] ? _cond_resched+0x1a/0x50
Sep 9 23:23:51 ged kernel: [1436769.181073] xfs_file_write_iter+0x103/0x150 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181077] do_iter_readv_writev+0x111/0x180
Sep 9 23:23:51 ged kernel: [1436769.181080] do_iter_write+0x87/0x1a0
Sep 9 23:23:51 ged kernel: [1436769.181083] vfs_iter_write+0x19/0x30
Sep 9 23:23:51 ged kernel: [1436769.181101] nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181117] nfsd4_write+0x16c/0x1a0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181131] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181143] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181169] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.181189] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.181200] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181206] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.181217] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181222] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.181228] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.181233] INFO: task nfsd:1939 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.181282] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.181328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.181385] nfsd D 0 1939 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.181389] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.181394] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.181398] ? sch_direct_xmit+0x73/0x1b0
Sep 9 23:23:51 ged kernel: [1436769.181401] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.181455] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181503] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181551] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181599] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181643] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181692] xfs_vn_update_time+0x5d/0x160 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181698] file_update_time+0xbe/0x110
Sep 9 23:23:51 ged kernel: [1436769.181747] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181792] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181796] ? _cond_resched+0x1a/0x50
Sep 9 23:23:51 ged kernel: [1436769.181839] xfs_file_write_iter+0x103/0x150 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.181843] do_iter_readv_writev+0x111/0x180
Sep 9 23:23:51 ged kernel: [1436769.181846] do_iter_write+0x87/0x1a0
Sep 9 23:23:51 ged kernel: [1436769.181849] vfs_iter_write+0x19/0x30
Sep 9 23:23:51 ged kernel: [1436769.181864] nfsd_vfs_write+0xbe/0x380 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181880] nfsd4_write+0x16c/0x1a0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181894] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181906] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181930] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.181951] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.181962] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181967] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.181978] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.181984] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.181988] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.181993] INFO: task nfsd:1940 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.182041] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.182088] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.182146] nfsd D 0 1940 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.182149] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.182153] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.182158] ? __switch_to_asm+0x40/0x70
Sep 9 23:23:51 ged kernel: [1436769.182162] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.182213] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182259] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182308] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182354] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182398] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182448] xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182494] xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182539] xfs_vn_setattr+0x66/0xa0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.182544] notify_change+0x2e1/0x410
Sep 9 23:23:51 ged kernel: [1436769.182558] nfsd_setattr+0x390/0x4f0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.182562] ? _cond_resched+0x1a/0x50
Sep 9 23:23:51 ged kernel: [1436769.182577] nfsd4_setattr+0x10e/0x140 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.182591] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.182603] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.182626] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.182646] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.182657] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.182662] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.182673] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.182710] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.182715] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.182720] INFO: task nfsd:1941 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.182771] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.182820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.182884] nfsd D 0 1941 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.182897] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.182905] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.182914] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.182972] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183022] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183075] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183126] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183174] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183229] xfs_create+0x1bf/0x6b0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183239] ? get_acl+0x1d/0x100
Sep 9 23:23:51 ged kernel: [1436769.183248] ? posix_acl_create+0x109/0x160
Sep 9 23:23:51 ged kernel: [1436769.183300] xfs_generic_create+0x21a/0x2f0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183350] xfs_vn_mknod+0x14/0x20 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183396] xfs_vn_create+0x13/0x20 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183405] vfs_create+0x12d/0x190
Sep 9 23:23:51 ged kernel: [1436769.183424] do_nfsd_create+0x43f/0x580 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.183446] nfsd4_open+0x20c/0x730 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.183466] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.183482] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.183509] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.183535] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.183553] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.183562] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.183578] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.183588] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.183596] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.183607] INFO: task nfsd:1942 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.183657] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.183703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.183760] nfsd D 0 1942 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.183763] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.183768] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.183772] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.183823] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183870] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183917] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.183964] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184008] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184056] xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184077] ? nfsd_setuser+0x11f/0x2b0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.184127] xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184174] xfs_vn_setattr+0x66/0xa0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184184] ? current_time+0x32/0x70
Sep 9 23:23:51 ged kernel: [1436769.184194] notify_change+0x2e1/0x410
Sep 9 23:23:51 ged kernel: [1436769.184212] nfsd_setattr+0x390/0x4f0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.184222] ? _cond_resched+0x1a/0x50
Sep 9 23:23:51 ged kernel: [1436769.184241] nfsd4_setattr+0x10e/0x140 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.184260] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.184278] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.184306] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.184330] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.184345] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.184356] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.184374] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.184384] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.184395] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.184406] INFO: task nfsd:1943 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.184460] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.184508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.184565] nfsd D 0 1943 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.184568] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.184572] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.184577] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.184628] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184673] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184721] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184768] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184812] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184867] xfs_setattr_nonsize+0x242/0x5a0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184918] xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184966] xfs_vn_setattr+0x66/0xa0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.184977] notify_change+0x2e1/0x410
Sep 9 23:23:51 ged kernel: [1436769.184997] nfsd_setattr+0x390/0x4f0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185004] ? _cond_resched+0x1a/0x50
Sep 9 23:23:51 ged kernel: [1436769.185020] nfsd4_setattr+0x10e/0x140 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185034] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185046] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185074] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.185098] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.185113] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185119] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.185135] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185143] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.185151] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.185156] INFO: task nfsd:1944 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.185205] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.185251] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.185308] nfsd D 0 1944 2 0x80000000
Sep 9 23:23:51 ged kernel: [1436769.185311] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.185315] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.185321] ? __slab_free+0xb2/0x2f0
Sep 9 23:23:51 ged kernel: [1436769.185324] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.185374] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.185420] xlog_grant_head_check+0xf1/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.185467] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.185513] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.185556] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.185606] xfs_rename+0x133/0xab0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.185610] ? _cond_resched+0x1a/0x50
Sep 9 23:23:51 ged kernel: [1436769.185614] ? sort+0x165/0x250
Sep 9 23:23:51 ged kernel: [1436769.185618] ? iter_div_u64_rem+0x20/0x20
Sep 9 23:23:51 ged kernel: [1436769.185665] xfs_vn_rename+0xd4/0x150 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.185670] vfs_rename+0x6a5/0x8c0
Sep 9 23:23:51 ged kernel: [1436769.185675] ? lookup_one_len+0xf5/0x120
Sep 9 23:23:51 ged kernel: [1436769.185689] nfsd_rename+0x359/0x3a0 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185705] nfsd4_rename+0x6e/0x190 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185719] nfsd4_proc_compound+0x33c/0x630 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185731] nfsd_dispatch+0xbb/0x250 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185753] svc_process_common+0x380/0x690 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.185773] svc_process+0xfa/0x1c0 [sunrpc]
Sep 9 23:23:51 ged kernel: [1436769.185785] nfsd+0xe9/0x150 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185790] kthread+0x105/0x140
Sep 9 23:23:51 ged kernel: [1436769.185801] ? nfsd_destroy+0x60/0x60 [nfsd]
Sep 9 23:23:51 ged kernel: [1436769.185807] ? kthread_destroy_worker+0x50/0x50
Sep 9 23:23:51 ged kernel: [1436769.185812] ret_from_fork+0x35/0x40
Sep 9 23:23:51 ged kernel: [1436769.185830] INFO: task bacula-fd:24743 blocked for more than 120 seconds.
Sep 9 23:23:51 ged kernel: [1436769.185881] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
Sep 9 23:23:51 ged kernel: [1436769.185927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 23:23:51 ged kernel: [1436769.185983] bacula-fd D 0 24743 1 0x00000000
Sep 9 23:23:51 ged kernel: [1436769.185987] Call Trace:
Sep 9 23:23:51 ged kernel: [1436769.185991] __schedule+0x3d6/0x8b0
Sep 9 23:23:51 ged kernel: [1436769.185995] schedule+0x36/0x80
Sep 9 23:23:51 ged kernel: [1436769.186045] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186090] xlog_grant_head_check+0x94/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186136] xfs_log_reserve+0xcb/0x1e0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186182] xfs_trans_reserve+0x169/0x1d0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186225] xfs_trans_alloc+0xbe/0x130 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186274] xfs_vn_update_time+0x5d/0x160 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186279] touch_atime+0xb2/0xe0
Sep 9 23:23:51 ged kernel: [1436769.186286] generic_file_read_iter+0x93a/0xba0
Sep 9 23:23:51 ged kernel: [1436769.186292] ? page_cache_tree_insert+0xe0/0xe0
Sep 9 23:23:51 ged kernel: [1436769.186341] xfs_file_buffered_aio_read+0x5d/0x100 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186385] xfs_file_read_iter+0x68/0xc0 [xfs]
Sep 9 23:23:51 ged kernel: [1436769.186389] new_sync_read+0xe2/0x130
Sep 9 23:23:51 ged kernel: [1436769.186393] __vfs_read+0x29/0x40
Sep 9 23:23:51 ged kernel: [1436769.186396] vfs_read+0x93/0x130
Sep 9 23:23:51 ged kernel: [1436769.186399] SyS_read+0x55/0xc0
Sep 9 23:23:51 ged kernel: [1436769.186405] do_syscall_64+0x73/0x130
Sep 9 23:23:51 ged kernel: [1436769.186409] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Sep 9 23:23:51 ged kernel: [1436769.186412] RIP: 0033:0x7f919219b51d
Sep 9 23:23:51 ged kernel: [1436769.186414] RSP: 002b:00007f918d90c9a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
Sep 9 23:23:51 ged kernel: [1436769.186418] RAX: ffffffffffffffda RBX: 00007f917c00b1a0 RCX: 00007f919219b51d
Sep 9 23:23:51 ged kernel: [1436769.186420] RDX: 0000000000010000 RSI: 00007f917c00c8c0 RDI: 000000000000000e
Sep 9 23:23:51 ged kernel: [1436769.186422] RBP: 00007f917e12de68 R08: 000000000000000a R09: 0000000000000000
Sep 9 23:23:51 ged kernel: [1436769.186424] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f917c096308
Sep 9 23:23:51 ged kernel: [1436769.186426] R13: 00007f917c00bfa8 R14: 00007f917c00b1a0 R15: 00007f917e12de68
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-09-12 8:07 Strange XFS problem Troels Hansen
@ 2018-09-12 10:59 ` Carlos Maiolino
2018-09-12 11:39 ` Troels Hansen
2018-09-13 4:19 ` Dave Chinner
1 sibling, 1 reply; 19+ messages in thread
From: Carlos Maiolino @ 2018-09-12 10:59 UTC (permalink / raw)
To: Troels Hansen; +Cc: linux-xfs
On Wed, Sep 12, 2018 at 10:07:55AM +0200, Troels Hansen wrote:
> Hi, we are facing an issue where we can't figure out if its XFS software related, or actually related to hardware, and can't quite figure out why we are facing the issues, though is doesn't seem hardware related.
>
> The issue is with a 102Tb array on a Dell branded LSISAS 3508 (Perc H840).
> Running Ubuntu with a 4.15.0-32 (Ubuntu branded), but we have also been running a number of 4.4.0-x with the same issues.
>
> The XFS filsusyem is on a very busy NFS server, and when the issue occurs we see strange issues with NFS, while the system seems healthy on the local server, but at the same time some programs are having problems accessing the fs.
>
> It occure roughly every 14 days, where we have to restart the fs to come back fully working.
> Sometimes refusing to unmount cleanly during shutdown, forcing us to fsck the fs on startup.
>
> It looks like its hanging in xlog_grant_head_wait, but I don't know enough to determine what can make it hang there.
>
> Hoping someone in here could have a look and point me in the right direction.
>
> Below is a trace from the last crash we had:
It's a long shot, but can't do much without more information. At first, your
filesystem is waiting for log space, so it could be your log is too small, your
storage is too slow and/or misconfigured, or maybe you you've been hitting a bug
somewhere!?
I don't remember any bug regarding log space issues or reservation problems
happening recently, but I don't use to work on log reservation issues by now,
so, maybe other can have more information if this might be any bug or not.
In the mean time, can you provide more information about the systems where you
are hitting this issue?
Mainle the storage configuration and the xfs_info output of the affected
filesystems.
FWIW:
http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
Cheers
>
>
> Sep 9 23:23:51 ged kernel: [1436769.178935] INFO: task mysqld:2847 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.178999] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.179047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.179105] mysqld D 0 2847 1 0x00000000
> Sep 9 23:23:51 ged kernel: [1436769.179111] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.179123] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.179127] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.179216] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179277] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179330] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179381] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179428] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179478] xfs_vn_update_time+0x5d/0x160 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179486] file_update_time+0xbe/0x110
> Sep 9 23:23:51 ged kernel: [1436769.179493] ? tcp_recvmsg+0x317/0xab0
> Sep 9 23:23:51 ged kernel: [1436769.179542] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179588] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179632] xfs_file_write_iter+0x103/0x150 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179637] new_sync_write+0xe5/0x140
> Sep 9 23:23:51 ged kernel: [1436769.179641] __vfs_write+0x29/0x40
> Sep 9 23:23:51 ged kernel: [1436769.179645] vfs_write+0xb8/0x1b0
> Sep 9 23:23:51 ged kernel: [1436769.179649] SyS_pwrite64+0x95/0xb0
> Sep 9 23:23:51 ged kernel: [1436769.179655] do_syscall_64+0x73/0x130
> Sep 9 23:23:51 ged kernel: [1436769.179661] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> Sep 9 23:23:51 ged kernel: [1436769.179664] RIP: 0033:0x7f16be838da3
> Sep 9 23:23:51 ged kernel: [1436769.179667] RSP: 002b:00007f148c31b950 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
> Sep 9 23:23:51 ged kernel: [1436769.179671] RAX: ffffffffffffffda RBX: 00007f148c31ba50 RCX: 00007f16be838da3
> Sep 9 23:23:51 ged kernel: [1436769.179673] RDX: 0000000000000200 RSI: 00007f149ea80200 RDI: 0000000000000008
> Sep 9 23:23:51 ged kernel: [1436769.179675] RBP: 00007f148c31bc00 R08: 000000000019d200 R09: 00007f148c31bc3c
> Sep 9 23:23:51 ged kernel: [1436769.179677] R10: 000000000019d200 R11: 0000000000000293 R12: 0000000000000200
> Sep 9 23:23:51 ged kernel: [1436769.179679] R13: 000000000019d200 R14: 0000000000000000 R15: 0000000000000000
> Sep 9 23:23:51 ged kernel: [1436769.179686] INFO: task nfsd:1937 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.179738] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.179784] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.179842] nfsd D 0 1937 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.179846] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.179852] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.179857] ? sch_direct_xmit+0x73/0x1b0
> Sep 9 23:23:51 ged kernel: [1436769.179860] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.179910] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179956] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180005] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180051] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180096] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180145] xfs_vn_update_time+0x5d/0x160 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180150] file_update_time+0xbe/0x110
> Sep 9 23:23:51 ged kernel: [1436769.180199] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180244] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180248] ? _cond_resched+0x1a/0x50
> Sep 9 23:23:51 ged kernel: [1436769.180291] xfs_file_write_iter+0x103/0x150 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180295] do_iter_readv_writev+0x111/0x180
> Sep 9 23:23:51 ged kernel: [1436769.180299] do_iter_write+0x87/0x1a0
> Sep 9 23:23:51 ged kernel: [1436769.180302] vfs_iter_write+0x19/0x30
> Sep 9 23:23:51 ged kernel: [1436769.180321] nfsd_vfs_write+0xbe/0x380 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.180338] nfsd4_write+0x16c/0x1a0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.180352] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.180365] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.180395] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.180416] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.180428] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.180434] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.180445] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.180451] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.180456] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.180461] INFO: task nfsd:1938 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.180511] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.180557] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.180615] nfsd D 0 1938 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.180619] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.180623] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.180628] ? sch_direct_xmit+0x73/0x1b0
> Sep 9 23:23:51 ged kernel: [1436769.180631] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.180689] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180737] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180786] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180832] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180876] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180926] xfs_vn_update_time+0x5d/0x160 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.180931] file_update_time+0xbe/0x110
> Sep 9 23:23:51 ged kernel: [1436769.180980] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181025] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181029] ? _cond_resched+0x1a/0x50
> Sep 9 23:23:51 ged kernel: [1436769.181073] xfs_file_write_iter+0x103/0x150 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181077] do_iter_readv_writev+0x111/0x180
> Sep 9 23:23:51 ged kernel: [1436769.181080] do_iter_write+0x87/0x1a0
> Sep 9 23:23:51 ged kernel: [1436769.181083] vfs_iter_write+0x19/0x30
> Sep 9 23:23:51 ged kernel: [1436769.181101] nfsd_vfs_write+0xbe/0x380 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181117] nfsd4_write+0x16c/0x1a0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181131] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181143] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181169] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.181189] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.181200] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181206] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.181217] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181222] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.181228] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.181233] INFO: task nfsd:1939 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.181282] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.181328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.181385] nfsd D 0 1939 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.181389] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.181394] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.181398] ? sch_direct_xmit+0x73/0x1b0
> Sep 9 23:23:51 ged kernel: [1436769.181401] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.181455] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181503] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181551] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181599] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181643] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181692] xfs_vn_update_time+0x5d/0x160 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181698] file_update_time+0xbe/0x110
> Sep 9 23:23:51 ged kernel: [1436769.181747] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181792] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181796] ? _cond_resched+0x1a/0x50
> Sep 9 23:23:51 ged kernel: [1436769.181839] xfs_file_write_iter+0x103/0x150 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.181843] do_iter_readv_writev+0x111/0x180
> Sep 9 23:23:51 ged kernel: [1436769.181846] do_iter_write+0x87/0x1a0
> Sep 9 23:23:51 ged kernel: [1436769.181849] vfs_iter_write+0x19/0x30
> Sep 9 23:23:51 ged kernel: [1436769.181864] nfsd_vfs_write+0xbe/0x380 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181880] nfsd4_write+0x16c/0x1a0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181894] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181906] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181930] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.181951] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.181962] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181967] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.181978] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.181984] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.181988] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.181993] INFO: task nfsd:1940 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.182041] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.182088] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.182146] nfsd D 0 1940 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.182149] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.182153] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.182158] ? __switch_to_asm+0x40/0x70
> Sep 9 23:23:51 ged kernel: [1436769.182162] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.182213] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182259] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182308] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182354] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182398] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182448] xfs_setattr_nonsize+0x242/0x5a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182494] xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182539] xfs_vn_setattr+0x66/0xa0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.182544] notify_change+0x2e1/0x410
> Sep 9 23:23:51 ged kernel: [1436769.182558] nfsd_setattr+0x390/0x4f0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.182562] ? _cond_resched+0x1a/0x50
> Sep 9 23:23:51 ged kernel: [1436769.182577] nfsd4_setattr+0x10e/0x140 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.182591] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.182603] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.182626] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.182646] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.182657] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.182662] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.182673] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.182710] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.182715] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.182720] INFO: task nfsd:1941 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.182771] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.182820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.182884] nfsd D 0 1941 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.182897] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.182905] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.182914] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.182972] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183022] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183075] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183126] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183174] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183229] xfs_create+0x1bf/0x6b0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183239] ? get_acl+0x1d/0x100
> Sep 9 23:23:51 ged kernel: [1436769.183248] ? posix_acl_create+0x109/0x160
> Sep 9 23:23:51 ged kernel: [1436769.183300] xfs_generic_create+0x21a/0x2f0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183350] xfs_vn_mknod+0x14/0x20 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183396] xfs_vn_create+0x13/0x20 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183405] vfs_create+0x12d/0x190
> Sep 9 23:23:51 ged kernel: [1436769.183424] do_nfsd_create+0x43f/0x580 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.183446] nfsd4_open+0x20c/0x730 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.183466] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.183482] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.183509] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.183535] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.183553] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.183562] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.183578] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.183588] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.183596] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.183607] INFO: task nfsd:1942 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.183657] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.183703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.183760] nfsd D 0 1942 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.183763] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.183768] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.183772] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.183823] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183870] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183917] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.183964] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184008] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184056] xfs_setattr_nonsize+0x242/0x5a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184077] ? nfsd_setuser+0x11f/0x2b0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.184127] xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184174] xfs_vn_setattr+0x66/0xa0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184184] ? current_time+0x32/0x70
> Sep 9 23:23:51 ged kernel: [1436769.184194] notify_change+0x2e1/0x410
> Sep 9 23:23:51 ged kernel: [1436769.184212] nfsd_setattr+0x390/0x4f0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.184222] ? _cond_resched+0x1a/0x50
> Sep 9 23:23:51 ged kernel: [1436769.184241] nfsd4_setattr+0x10e/0x140 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.184260] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.184278] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.184306] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.184330] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.184345] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.184356] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.184374] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.184384] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.184395] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.184406] INFO: task nfsd:1943 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.184460] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.184508] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.184565] nfsd D 0 1943 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.184568] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.184572] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.184577] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.184628] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184673] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184721] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184768] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184812] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184867] xfs_setattr_nonsize+0x242/0x5a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184918] xfs_vn_setattr_nonsize+0x42/0x90 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184966] xfs_vn_setattr+0x66/0xa0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.184977] notify_change+0x2e1/0x410
> Sep 9 23:23:51 ged kernel: [1436769.184997] nfsd_setattr+0x390/0x4f0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185004] ? _cond_resched+0x1a/0x50
> Sep 9 23:23:51 ged kernel: [1436769.185020] nfsd4_setattr+0x10e/0x140 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185034] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185046] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185074] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.185098] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.185113] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185119] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.185135] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185143] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.185151] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.185156] INFO: task nfsd:1944 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.185205] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.185251] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.185308] nfsd D 0 1944 2 0x80000000
> Sep 9 23:23:51 ged kernel: [1436769.185311] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.185315] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.185321] ? __slab_free+0xb2/0x2f0
> Sep 9 23:23:51 ged kernel: [1436769.185324] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.185374] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.185420] xlog_grant_head_check+0xf1/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.185467] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.185513] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.185556] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.185606] xfs_rename+0x133/0xab0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.185610] ? _cond_resched+0x1a/0x50
> Sep 9 23:23:51 ged kernel: [1436769.185614] ? sort+0x165/0x250
> Sep 9 23:23:51 ged kernel: [1436769.185618] ? iter_div_u64_rem+0x20/0x20
> Sep 9 23:23:51 ged kernel: [1436769.185665] xfs_vn_rename+0xd4/0x150 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.185670] vfs_rename+0x6a5/0x8c0
> Sep 9 23:23:51 ged kernel: [1436769.185675] ? lookup_one_len+0xf5/0x120
> Sep 9 23:23:51 ged kernel: [1436769.185689] nfsd_rename+0x359/0x3a0 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185705] nfsd4_rename+0x6e/0x190 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185719] nfsd4_proc_compound+0x33c/0x630 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185731] nfsd_dispatch+0xbb/0x250 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185753] svc_process_common+0x380/0x690 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.185773] svc_process+0xfa/0x1c0 [sunrpc]
> Sep 9 23:23:51 ged kernel: [1436769.185785] nfsd+0xe9/0x150 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185790] kthread+0x105/0x140
> Sep 9 23:23:51 ged kernel: [1436769.185801] ? nfsd_destroy+0x60/0x60 [nfsd]
> Sep 9 23:23:51 ged kernel: [1436769.185807] ? kthread_destroy_worker+0x50/0x50
> Sep 9 23:23:51 ged kernel: [1436769.185812] ret_from_fork+0x35/0x40
> Sep 9 23:23:51 ged kernel: [1436769.185830] INFO: task bacula-fd:24743 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.185881] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.185927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.185983] bacula-fd D 0 24743 1 0x00000000
> Sep 9 23:23:51 ged kernel: [1436769.185987] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.185991] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.185995] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.186045] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186090] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186136] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186182] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186225] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186274] xfs_vn_update_time+0x5d/0x160 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186279] touch_atime+0xb2/0xe0
> Sep 9 23:23:51 ged kernel: [1436769.186286] generic_file_read_iter+0x93a/0xba0
> Sep 9 23:23:51 ged kernel: [1436769.186292] ? page_cache_tree_insert+0xe0/0xe0
> Sep 9 23:23:51 ged kernel: [1436769.186341] xfs_file_buffered_aio_read+0x5d/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186385] xfs_file_read_iter+0x68/0xc0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.186389] new_sync_read+0xe2/0x130
> Sep 9 23:23:51 ged kernel: [1436769.186393] __vfs_read+0x29/0x40
> Sep 9 23:23:51 ged kernel: [1436769.186396] vfs_read+0x93/0x130
> Sep 9 23:23:51 ged kernel: [1436769.186399] SyS_read+0x55/0xc0
> Sep 9 23:23:51 ged kernel: [1436769.186405] do_syscall_64+0x73/0x130
> Sep 9 23:23:51 ged kernel: [1436769.186409] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> Sep 9 23:23:51 ged kernel: [1436769.186412] RIP: 0033:0x7f919219b51d
> Sep 9 23:23:51 ged kernel: [1436769.186414] RSP: 002b:00007f918d90c9a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000000
> Sep 9 23:23:51 ged kernel: [1436769.186418] RAX: ffffffffffffffda RBX: 00007f917c00b1a0 RCX: 00007f919219b51d
> Sep 9 23:23:51 ged kernel: [1436769.186420] RDX: 0000000000010000 RSI: 00007f917c00c8c0 RDI: 000000000000000e
> Sep 9 23:23:51 ged kernel: [1436769.186422] RBP: 00007f917e12de68 R08: 000000000000000a R09: 0000000000000000
> Sep 9 23:23:51 ged kernel: [1436769.186424] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f917c096308
> Sep 9 23:23:51 ged kernel: [1436769.186426] R13: 00007f917c00bfa8 R14: 00007f917c00b1a0 R15: 00007f917e12de68
>
--
Carlos
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-09-12 10:59 ` Carlos Maiolino
@ 2018-09-12 11:39 ` Troels Hansen
[not found] ` <notmuch-sha1-960c954e5404b5b2f083d150633af0b7848ec14c>
0 siblings, 1 reply; 19+ messages in thread
From: Troels Hansen @ 2018-09-12 11:39 UTC (permalink / raw)
To: Carlos Maiolino; +Cc: linux-xfs
Hi Carlos
----- On Sep 12, 2018, at 12:59 PM, Carlos Maiolino cmaiolino@redhat.com wrote:
> In the mean time, can you provide more information about the systems where you
> are hitting this issue?
>
> Mainle the storage configuration and the xfs_info output of the affected
> filesystems.
>
> FWIW:
> http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
$ uname -a
Linux ged 4.15.0-32-generic #35~16.04.1-Ubuntu SMP Fri Aug 10 21:54:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ sudo xfs_info /z/home
meta-data=/dev/sda isize=256 agcount=102, agsize=268435328 blks
= sectsz=512 attr=2, projid32bit=0
= crc=0 finobt=0 spinodes=0
data = bsize=4096 blocks=27345289216, imaxpct=1
= sunit=128 swidth=256 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
$ xfs_repair -V
xfs_repair version 4.3.0
CPU:
2 x Xeon(R) Gold 6142, total 32cores, 64 threads.
$ cat /proc/meminfo
MemTotal: 1583995652 kB
MemFree: 143862972 kB
MemAvailable: 1509722704 kB
Buffers: 388972 kB
Cached: 1237429160 kB
SwapCached: 0 kB
Active: 1026769972 kB
Inactive: 218745456 kB
Active(anon): 7542528 kB
Inactive(anon): 274984 kB
Active(file): 1019227444 kB
Inactive(file): 218470472 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 9765372 kB
SwapFree: 9765372 kB
Dirty: 4976 kB
Writeback: 0 kB
AnonPages: 7696592 kB
Mapped: 90216 kB
Shmem: 120212 kB
Slab: 144289884 kB
SReclaimable: 136424256 kB
SUnreclaim: 7865628 kB
KernelStack: 16176 kB
PageTables: 25844 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 801763196 kB
Committed_AS: 12122188 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 1826624 kB
DirectMap2M: 224344064 kB
DirectMap1G: 1385168896 kB
$ cat /proc/mounts
.....
/dev/sda /z/home xfs rw,relatime,attr2,inode64,sunit=1024,swidth=2048,noquota 0 0
$ cat /proc/partitions
major minor #blocks name
8 0 109381156864 sda
8 16 39048970240 sdb
8 32 585498624 sdc
8 33 9765376 sdc1
8 34 1 sdc2
8 37 575732480 sdc5
11 0 1048575 sr0
Hardware raid on Dell PERC H840
Raid50, 8 disk per span, 4 spans.
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 101.869 TB
Sector Size : 512
Parity Size : 14.552 TB
State : Optimal
Strip Size : 512 KB
Number Of Drives per span:8
Span Depth : 4
6Gb cache on controller, battery backed up.
Disks are 4Tb Toshiba 12G SAS disks.
Not running LVM.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
[not found] ` <notmuch-sha1-960c954e5404b5b2f083d150633af0b7848ec14c>
@ 2018-09-12 16:39 ` Carlos Maiolino
0 siblings, 0 replies; 19+ messages in thread
From: Carlos Maiolino @ 2018-09-12 16:39 UTC (permalink / raw)
To: Troels Hansen; +Cc: linux-xfs
I think it's quite clear I hit 'send' by mistake :)
On Wed, Sep 12, 2018 at 05:26:50PM +0200, Carlos Maiolino wrote:
> Hi,
>
> On Wed, Sep 12, 2018 at 01:39:59PM +0200, Troels Hansen wrote:
> > Hi Carlos
> >
> >
> > ----- On Sep 12, 2018, at 12:59 PM, Carlos Maiolino cmaiolino@redhat.com wrote:
> >
> > > In the mean time, can you provide more information about the systems where you
> > > are hitting this issue?
> > >
> > > Mainle the storage configuration and the xfs_info output of the affected
> > > filesystems.
> > >
> > > FWIW:
> > > http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F
> >
> >
> > $ uname -a
> > Linux ged 4.15.0-32-generic #35~16.04.1-Ubuntu SMP Fri Aug 10 21:54:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >
> >
> > $ sudo xfs_info /z/home
> > meta-data=/dev/sda isize=256 agcount=102, agsize=268435328 blks
> > = sectsz=512 attr=2, projid32bit=0
> > = crc=0 finobt=0 spinodes=0
> > data = bsize=4096 blocks=27345289216, imaxpct=1
> > = sunit=128 swidth=256 blks
> > naming =version 2 bsize=4096 ascii-ci=0 ftype=0
> > log =internal bsize=4096 blocks=521728, version=2
> > = sectsz=512 sunit=8 blks, lazy-count=1
> > realtime =none extsz=4096 blocks=0, rtextents=0
> >
>
> Ok, the settings look reasonable for the size of the filesystem, agcount and log
> size looks sane, the stripe configuration doesn't though, based on your raid
> configuration. And you're using V4 filesystem, which change things a bit.
> >
>
> > $ cat /proc/mounts
> > .....
> > /dev/sda /z/home xfs rw,relatime,attr2,inode64,sunit=1024,swidth=2048,noquota 0 0
>
> Mounted with different alignment. 512KiB sunit and a 1MiB width.
> >
> > Raid50, 8 disk per span, 4 spans.
> > RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
> > Size : 101.869 TB
> > Sector Size : 512
> > Parity Size : 14.552 TB
> > State : Optimal
> > Strip Size : 512 KB
> > Number Of Drives per span:8
> > Span Depth : 4
>
> So, if I understand correctly your configuration (I'm not really used to dell
> namings), you have:
>
> 4 RAID5 arrays using 8 disk each, nested in a RAID0 array, which has a 512KiB
> stripe size.
>
" If I'm right here, you should be using stripe unit 512K and stripe width
2MiB. (sunit 1024 swidth 4096)"
I didn't have too much time to think about it by now, and I am not sure if
setting the correct stripe alignment is the culprit here (most likely not), and
since I didn't think about it enough yet, I was postponing the e-mail and hit
'send' instead. My apologies about it.
I need to give it some extra thought to get some idea on how to help you here,
or have any reason to ask you to use a newer kernel. Again, my apologies for
hitting send too soon.
> >
> > 6Gb cache on controller, battery backed up.
> >
> > Disks are 4Tb Toshiba 12G SAS disks.
> >
> > Not running LVM.
> >
>
> --
> Carlos
--
Carlos
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-09-12 8:07 Strange XFS problem Troels Hansen
2018-09-12 10:59 ` Carlos Maiolino
@ 2018-09-13 4:19 ` Dave Chinner
2018-09-13 5:21 ` Troels Hansen
1 sibling, 1 reply; 19+ messages in thread
From: Dave Chinner @ 2018-09-13 4:19 UTC (permalink / raw)
To: Troels Hansen; +Cc: linux-xfs
On Wed, Sep 12, 2018 at 10:07:55AM +0200, Troels Hansen wrote:
> Hi, we are facing an issue where we can't figure out if its XFS software related, or actually related to hardware, and can't quite figure out why we are facing the issues, though is doesn't seem hardware related.
>
> The issue is with a 102Tb array on a Dell branded LSISAS 3508 (Perc H840).
> Running Ubuntu with a 4.15.0-32 (Ubuntu branded), but we have also been running a number of 4.4.0-x with the same issues.
Smells of an IO overload problem from that.
> The XFS filsusyem is on a very busy NFS server, and when the issue
> occurs we see strange issues with NFS, while the system seems
> healthy on the local server, but at the same time some programs
> are having problems accessing the fs.
>
> It occure roughly every 14 days, where we have to restart the fs to come back fully working.
What happens on your network every 14 days or so? Is there a rogue
client side backup or admin task running somewhere?
> Sometimes refusing to unmount cleanly during shutdown, forcing us to fsck the fs on startup.
Unclean shutdown doesn't require fsck to be run.
> It looks like its hanging in xlog_grant_head_wait, but I don't know enough to determine what can make it hang there.
>
> Hoping someone in here could have a look and point me in the right direction.
>
> Below is a trace from the last crash we had:
Not a crash - it's a hung task warning.
> Sep 9 23:23:51 ged kernel: [1436769.178935] INFO: task mysqld:2847 blocked for more than 120 seconds.
> Sep 9 23:23:51 ged kernel: [1436769.178999] Not tainted 4.15.0-32-generic #35~16.04.1-Ubuntu
> Sep 9 23:23:51 ged kernel: [1436769.179047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Sep 9 23:23:51 ged kernel: [1436769.179105] mysqld D 0 2847 1 0x00000000
> Sep 9 23:23:51 ged kernel: [1436769.179111] Call Trace:
> Sep 9 23:23:51 ged kernel: [1436769.179123] __schedule+0x3d6/0x8b0
> Sep 9 23:23:51 ged kernel: [1436769.179127] schedule+0x36/0x80
> Sep 9 23:23:51 ged kernel: [1436769.179216] xlog_grant_head_wait+0xb8/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179277] xlog_grant_head_check+0x94/0x100 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179330] xfs_log_reserve+0xcb/0x1e0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179381] xfs_trans_reserve+0x169/0x1d0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179428] xfs_trans_alloc+0xbe/0x130 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179478] xfs_vn_update_time+0x5d/0x160 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179486] file_update_time+0xbe/0x110
> Sep 9 23:23:51 ged kernel: [1436769.179493] ? tcp_recvmsg+0x317/0xab0
> Sep 9 23:23:51 ged kernel: [1436769.179542] xfs_file_aio_write_checks+0x13a/0x180 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179588] xfs_file_buffered_aio_write+0x89/0x2a0 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179632] xfs_file_write_iter+0x103/0x150 [xfs]
> Sep 9 23:23:51 ged kernel: [1436769.179637] new_sync_write+0xe5/0x140
> Sep 9 23:23:51 ged kernel: [1436769.179641] __vfs_write+0x29/0x40
> Sep 9 23:23:51 ged kernel: [1436769.179645] vfs_write+0xb8/0x1b0
> Sep 9 23:23:51 ged kernel: [1436769.179649] SyS_pwrite64+0x95/0xb0
> Sep 9 23:23:51 ged kernel: [1436769.179655] do_syscall_64+0x73/0x130
> Sep 9 23:23:51 ged kernel: [1436769.179661] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
.....
Does this repeat every 120s?
These hung task warnings can happen if your workload has overloaded
your raid array and everything doing IO hangs while it catches up.
e.g. you have 6GB of random 4k writes in the controller NV cache and
it takes minutes for it to flush (because random 4k writes are slow)
and make room for new incoming IO....
If the warnings don't repeat, then it means it was a temporary
overload. If the warnings repeat, but change processes and stack
traces then it's a sustained overload condition. If exactly the same
warnings repeat and/or has stalled and doesn't restart, then we've
got some kind of hang occurring and we'll need to look into it
further.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-09-13 4:19 ` Dave Chinner
@ 2018-09-13 5:21 ` Troels Hansen
2018-09-13 6:18 ` Dave Chinner
0 siblings, 1 reply; 19+ messages in thread
From: Troels Hansen @ 2018-09-13 5:21 UTC (permalink / raw)
To: Dave Chinner; +Cc: linux-xfs
>
> What happens on your network every 14 days or so? Is there a rogue
> client side backup or admin task running somewhere?
>
Well, we run nightly backups, but thats read ops.
When I look at the load, its not particular more loaded at that time, than normal work.
>
> Does this repeat every 120s?
No, what I sent is the full trace. It happened around 23:23, but no more XFS errors in the log (which is on the ext4 OS disk).
It was working when I came in the following morning aroung 6:45, and worked for some time, but initially failed, and we had to reboot the server to get NFS exports to work.
But, as I said, even though the fs was inaccessible from NFS I could `ls` the filesystem locally, but we really have no indication of it being an NFS problem, as we only see the XFS problem.
It could also boil down to a NFS problem, I just wasn't sure how to read the XFS trace.
> These hung task warnings can happen if your workload has overloaded
> your raid array and everything doing IO hangs while it catches up.
> e.g. you have 6GB of random 4k writes in the controller NV cache and
> it takes minutes for it to flush (because random 4k writes are slow)
> and make room for new incoming IO....
>
> If the warnings don't repeat, then it means it was a temporary
> overload. If the warnings repeat, but change processes and stack
> traces then it's a sustained overload condition. If exactly the same
> warnings repeat and/or has stalled and doesn't restart, then we've
> got some kind of hang occurring and we'll need to look into it
> further.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-09-13 5:21 ` Troels Hansen
@ 2018-09-13 6:18 ` Dave Chinner
0 siblings, 0 replies; 19+ messages in thread
From: Dave Chinner @ 2018-09-13 6:18 UTC (permalink / raw)
To: Troels Hansen; +Cc: linux-xfs
On Thu, Sep 13, 2018 at 07:21:59AM +0200, Troels Hansen wrote:
>
> >
> > What happens on your network every 14 days or so? Is there a rogue
> > client side backup or admin task running somewhere?
> >
>
> Well, we run nightly backups, but thats read ops.
Yup, but that can get stuck modifying atime, like the bacula process
in the hung process traces. :)
Hmmm - just a thought - it's hardware raid - it's not running a
background admin op like a media scrub every 14 days, is it?
> When I look at the load, its not particular more loaded at that time, than normal work.
OK.
> > Does this repeat every 120s?
>
> No, what I sent is the full trace. It happened around 23:23, but
> no more XFS errors in the log (which is on the ext4 OS disk).
Ok, so those processes reported as hung have been woken and made
progress again. It seems like a temporary overload situation.
> It was working when I came in the following morning aroung 6:45,
> and worked for some time, but initially failed, and we had to
> reboot the server to get NFS exports to work. But, as I said,
> even though the fs was inaccessible from NFS I could `ls` the
> filesystem locally, but we really have no indication of it being
> an NFS problem, as we only see the XFS problem.
That could be the same problem, with all the kernel nfsds blocked
waiting for the filesystem so no new NFS requests could be
processed. How many kernel nfsd threads do you run? Local
operations can still be done (don't go through nfsds), and they
won't be slow if they hit the caches rather than have to retreive
data from disk.
> It could also boil down to a NFS problem, I just wasn't sure how
> to read the XFS trace.
Like you, I don't think this is an NFS problem - it smells more of
how huge hardware writeback caches in front of slow disks using
RAID5/6 behave.
i.e. Flushing 100MB of sequential write data from the cache takes a
fraction or a second, flushing 100MB of random 4k write data to
RAID5 luns can take minutes. While the hardware cache and flushing is
supposed to be completely invisible to the OS, we can see it's
impact via unexpectedly high device utilisations and long IO times
for otherwise normal IO loads.
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-12 18:56 ` Eric Sandeen
2018-06-13 22:02 ` Dave Chinner
@ 2018-06-14 15:47 ` Stefan Ring
1 sibling, 0 replies; 19+ messages in thread
From: Stefan Ring @ 2018-06-14 15:47 UTC (permalink / raw)
To: Eric Sandeen; +Cc: linux-xfs
On Tue, Jun 12, 2018 at 8:56 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>
> Is the German translation bad/embarrassing enough that it should just
> be removed? Does it do more harm than good at this point?
I would not want to make that call. Certainly parts of it are
recyclable, so it would be wasteful to throw it all out.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-13 22:02 ` Dave Chinner
@ 2018-06-14 2:11 ` Eric Sandeen
0 siblings, 0 replies; 19+ messages in thread
From: Eric Sandeen @ 2018-06-14 2:11 UTC (permalink / raw)
To: Dave Chinner; +Cc: Stefan Ring, linux-xfs
On 6/13/18 5:02 PM, Dave Chinner wrote:
> On Tue, Jun 12, 2018 at 01:56:06PM -0500, Eric Sandeen wrote:
>>
>>
>> On 6/12/18 1:53 PM, Stefan Ring wrote:
>>> On Tue, Jun 5, 2018 at 4:19 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>>>> In all seriousness, if you would like to update the translations that would
>>>> be great. The Polish translation is the only one that receives regular
>>>> updates today.
>>>
>>> I've looked into it, but this seems to me like a gargantuan task
>>> because there is so much internal inconsistency that I wouldn't even
>>> know where to start. "Extent" for example: sometimes this is
>>> translated as "Erweiterung" (nonsense, that's extension). Sometimes
>>> it's left verbatim. This is what I'd be leaning towards as well. Other
>>> times, it got translated as "Ausmaß", which in another context is in
>>> fact a translation for "extent" (as in: the extent of a catastrophy),
>>> but makes no sense here. Also "Bereich" and "Umfang" are used
>>> sometimes, of which only the former makes a slight bit of sense. That
>>> in addition to my general under-appreciation of low-level technical
>>> translation. They are usually much less comprehensible and the
>>> gibberish needs to be translated back anyway in order to any sense of,
>>> apart from the fact that the original text is searchable, in stark
>>> contrast to the translation.
>>
>> Thanks for looking.
>>
>> Is the German translation bad/embarrassing enough that it should just
>> be removed? Does it do more harm than good at this point?
>
> Don't distro's have internationalisation teams precisely for doing
> translations of packages? If so, shouldn't we be asking for those
> teams to do translation work?
Sorta?
https://fedoraproject.org/wiki/L10N/Translate_on_Zanata
https://fedora.zanata.org
We could try to pursue that, but it currently has very few packages
on its "priority" list (anaconda, etc)
-Eric
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-12 18:56 ` Eric Sandeen
@ 2018-06-13 22:02 ` Dave Chinner
2018-06-14 2:11 ` Eric Sandeen
2018-06-14 15:47 ` Stefan Ring
1 sibling, 1 reply; 19+ messages in thread
From: Dave Chinner @ 2018-06-13 22:02 UTC (permalink / raw)
To: Eric Sandeen; +Cc: Stefan Ring, linux-xfs
On Tue, Jun 12, 2018 at 01:56:06PM -0500, Eric Sandeen wrote:
>
>
> On 6/12/18 1:53 PM, Stefan Ring wrote:
> > On Tue, Jun 5, 2018 at 4:19 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> >> In all seriousness, if you would like to update the translations that would
> >> be great. The Polish translation is the only one that receives regular
> >> updates today.
> >
> > I've looked into it, but this seems to me like a gargantuan task
> > because there is so much internal inconsistency that I wouldn't even
> > know where to start. "Extent" for example: sometimes this is
> > translated as "Erweiterung" (nonsense, that's extension). Sometimes
> > it's left verbatim. This is what I'd be leaning towards as well. Other
> > times, it got translated as "Ausmaß", which in another context is in
> > fact a translation for "extent" (as in: the extent of a catastrophy),
> > but makes no sense here. Also "Bereich" and "Umfang" are used
> > sometimes, of which only the former makes a slight bit of sense. That
> > in addition to my general under-appreciation of low-level technical
> > translation. They are usually much less comprehensible and the
> > gibberish needs to be translated back anyway in order to any sense of,
> > apart from the fact that the original text is searchable, in stark
> > contrast to the translation.
>
> Thanks for looking.
>
> Is the German translation bad/embarrassing enough that it should just
> be removed? Does it do more harm than good at this point?
Don't distro's have internationalisation teams precisely for doing
translations of packages? If so, shouldn't we be asking for those
teams to do translation work?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-12 18:53 ` Stefan Ring
@ 2018-06-12 18:56 ` Eric Sandeen
2018-06-13 22:02 ` Dave Chinner
2018-06-14 15:47 ` Stefan Ring
0 siblings, 2 replies; 19+ messages in thread
From: Eric Sandeen @ 2018-06-12 18:56 UTC (permalink / raw)
To: Stefan Ring; +Cc: linux-xfs
On 6/12/18 1:53 PM, Stefan Ring wrote:
> On Tue, Jun 5, 2018 at 4:19 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>> In all seriousness, if you would like to update the translations that would
>> be great. The Polish translation is the only one that receives regular
>> updates today.
>
> I've looked into it, but this seems to me like a gargantuan task
> because there is so much internal inconsistency that I wouldn't even
> know where to start. "Extent" for example: sometimes this is
> translated as "Erweiterung" (nonsense, that's extension). Sometimes
> it's left verbatim. This is what I'd be leaning towards as well. Other
> times, it got translated as "Ausmaß", which in another context is in
> fact a translation for "extent" (as in: the extent of a catastrophy),
> but makes no sense here. Also "Bereich" and "Umfang" are used
> sometimes, of which only the former makes a slight bit of sense. That
> in addition to my general under-appreciation of low-level technical
> translation. They are usually much less comprehensible and the
> gibberish needs to be translated back anyway in order to any sense of,
> apart from the fact that the original text is searchable, in stark
> contrast to the translation.
Thanks for looking.
Is the German translation bad/embarrassing enough that it should just
be removed? Does it do more harm than good at this point?
-Eric
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-05 14:19 ` Eric Sandeen
2018-06-05 14:46 ` Stefan Ring
@ 2018-06-12 18:53 ` Stefan Ring
2018-06-12 18:56 ` Eric Sandeen
1 sibling, 1 reply; 19+ messages in thread
From: Stefan Ring @ 2018-06-12 18:53 UTC (permalink / raw)
To: Eric Sandeen; +Cc: linux-xfs
On Tue, Jun 5, 2018 at 4:19 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> In all seriousness, if you would like to update the translations that would
> be great. The Polish translation is the only one that receives regular
> updates today.
I've looked into it, but this seems to me like a gargantuan task
because there is so much internal inconsistency that I wouldn't even
know where to start. "Extent" for example: sometimes this is
translated as "Erweiterung" (nonsense, that's extension). Sometimes
it's left verbatim. This is what I'd be leaning towards as well. Other
times, it got translated as "Ausmaß", which in another context is in
fact a translation for "extent" (as in: the extent of a catastrophy),
but makes no sense here. Also "Bereich" and "Umfang" are used
sometimes, of which only the former makes a slight bit of sense. That
in addition to my general under-appreciation of low-level technical
translation. They are usually much less comprehensible and the
gibberish needs to be translated back anyway in order to any sense of,
apart from the fact that the original text is searchable, in stark
contrast to the translation.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-05 14:46 ` Stefan Ring
@ 2018-06-05 17:34 ` Chris Murphy
0 siblings, 0 replies; 19+ messages in thread
From: Chris Murphy @ 2018-06-05 17:34 UTC (permalink / raw)
To: Stefan Ring; +Cc: Eric Sandeen, xfs list
On Tue, Jun 5, 2018 at 8:46 AM, Stefan Ring <stefanrin@gmail.com> wrote:
> On Tue, Jun 5, 2018 at 4:19 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
>> It's been a very long time since the German translations were updated (2009);
>> I'm sure we'd love to have someone look after them, hint hint?
>
> Yes, I've anticipated this hint as soon as having typed the sentence ;).
>
>> Still, it can't be as bad as the e2fsprogs translation referring to
>> Böcke instead of Blöcke... ;)
>
> Haha (goats or rams vs. blocks)!
Wir sollen die Böcke von den Schafen trennen!
--
Chris Murphy
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-05 14:19 ` Eric Sandeen
@ 2018-06-05 14:46 ` Stefan Ring
2018-06-05 17:34 ` Chris Murphy
2018-06-12 18:53 ` Stefan Ring
1 sibling, 1 reply; 19+ messages in thread
From: Stefan Ring @ 2018-06-05 14:46 UTC (permalink / raw)
To: Eric Sandeen; +Cc: linux-xfs
On Tue, Jun 5, 2018 at 4:19 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> It's been a very long time since the German translations were updated (2009);
> I'm sure we'd love to have someone look after them, hint hint?
Yes, I've anticipated this hint as soon as having typed the sentence ;).
> Still, it can't be as bad as the e2fsprogs translation referring to
> Böcke instead of Blöcke... ;)
Haha (goats or rams vs. blocks)!
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-05 13:46 ` Stefan Ring
@ 2018-06-05 14:19 ` Eric Sandeen
2018-06-05 14:46 ` Stefan Ring
2018-06-12 18:53 ` Stefan Ring
0 siblings, 2 replies; 19+ messages in thread
From: Eric Sandeen @ 2018-06-05 14:19 UTC (permalink / raw)
To: Stefan Ring, linux-xfs
On 6/5/18 8:46 AM, Stefan Ring wrote:
> On Tue, Jun 5, 2018 at 10:29 AM, Carlos Maiolino <cmaiolino@redhat.com> wrote:
>> Please at least translate the message to english before posting it to the list
>>
>> Me and several other developers on the list doesn't speak a single word in
>> German, and going to the .po files to find the translation is too time
>> consuming.
>
> Also, the translation is cringe-worthy for someone with German as
> their mother tongue.
It's been a very long time since the German translations were updated (2009);
I'm sure we'd love to have someone look after them, hint hint? My German
is only good enough to recognize a few words here and there.
Still, it can't be as bad as the e2fsprogs translation referring to
Böcke instead of Blöcke... ;)
In all seriousness, if you would like to update the translations that would
be great. The Polish translation is the only one that receives regular
updates today.
-Eric
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-05 8:29 ` Carlos Maiolino
@ 2018-06-05 13:46 ` Stefan Ring
2018-06-05 14:19 ` Eric Sandeen
0 siblings, 1 reply; 19+ messages in thread
From: Stefan Ring @ 2018-06-05 13:46 UTC (permalink / raw)
To: linux-xfs
On Tue, Jun 5, 2018 at 10:29 AM, Carlos Maiolino <cmaiolino@redhat.com> wrote:
> Please at least translate the message to english before posting it to the list
>
> Me and several other developers on the list doesn't speak a single word in
> German, and going to the .po files to find the translation is too time
> consuming.
Also, the translation is cringe-worthy for someone with German as
their mother tongue.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-03 2:32 ` Thorsten Hufnagel
@ 2018-06-05 8:29 ` Carlos Maiolino
2018-06-05 13:46 ` Stefan Ring
0 siblings, 1 reply; 19+ messages in thread
From: Carlos Maiolino @ 2018-06-05 8:29 UTC (permalink / raw)
To: Thorsten Hufnagel; +Cc: linux-xfs
On Sun, Jun 03, 2018 at 04:32:24AM +0200, Thorsten Hufnagel wrote:
> I just Got it.
>
> The issue seemed to got fixed with 4.16.0-rc1.
>
> Downloading / compiling / using the 4.16.1 version solved my issue.
>
> Sorry for bothering,
>
> Thorsten
>
>
Good it worked out, but:
> > "
> > mount: /mnt/test: Falscher Dateisystemtyp, ungültige Optionen, der
> > Superblock von /dev/md2 ist beschädigt, fehlende Kodierungsseite oder
> > ein anderer Fehler.
> > "
> >
> > And "xfs_repair -L" results in
> >
> > "
> > Phase 1 - Superblock finden und überprüfen...
> > - Berichts-Prozess in Abständen von 15 Minutes
> > Phase 2 - ein internes Protokoll benutzen
> > - Null-Protokoll...
Please at least translate the message to english before posting it to the list
Me and several other developers on the list doesn't speak a single word in
German, and going to the .po files to find the translation is too time
consuming.
Cheers.
> > Log inconsistent or not a log (last==0, first!=1)
> > empty log check failed
> > zero_log: cannot find log head/tail (xlog_find_tail=22)
> > "
> >
> > Sounds linke some kind of "Hen-egg" problem.
> >
> > Is there any change to get the XFS file sytem back running without
> > recreating the filesystem?
> >
> > Kernel:
> >
> > "
> > uname -r
> > 4.16.12-1-default
> > "
> >
> > Distribution: "Opensuse - Tumbleween"
> >
> > XFS tools: "xfs_repair Version 4.15.1"
> >
> > Thank you in advance,
> > Kind regards,
> >
> > Thorsten Hufnagel
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Carlos
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Strange XFS problem
2018-06-03 1:51 Thorsten Hufnagel
@ 2018-06-03 2:32 ` Thorsten Hufnagel
2018-06-05 8:29 ` Carlos Maiolino
0 siblings, 1 reply; 19+ messages in thread
From: Thorsten Hufnagel @ 2018-06-03 2:32 UTC (permalink / raw)
To: linux-xfs
I just Got it.
The issue seemed to got fixed with 4.16.0-rc1.
Downloading / compiling / using the 4.16.1 version solved my issue.
Sorry for bothering,
Thorsten
Am Sonntag, den 03.06.2018, 03:51 +0200 schrieb Thorsten Hufnagel:
> Hi there,
>
> I've just a big problem with my XFS file system used on a mdraid raid
> 5
> array.
>
> It started with the situation that the mount of my device was not
> possible any more (don't know why, hdd's are OK).
>
> "xfs_repair" results in
>
> "
> ERROR: The log head and/or tail cannot be discovered. Attempt to
> mount
> the
> filesystem to replay the log or use the -L option to destroy the log
> and
> attempt a repair.
> "
>
> "xfs_repair -L " ran, but unfortunately it was interrupted by an
> automatic server shutdown after it stuck after stage 7.
>
> Now, "xfs_repair" results in
>
> "
> Phase 1 - Superblock finden und überprüfen...
> - Berichts-Prozess in Abständen von 15 Minutes
> Phase 2 - ein internes Protokoll benutzen
> - Null-Protokoll...
> Log inconsistent or not a log (last==0, first!=1)
> empty log check failed
> zero_log: cannot find log head/tail (xlog_find_tail=22)
> ERROR: The log head and/or tail cannot be discovered. Attempt to
> mount
> the
> filesystem to replay the log or use the -L option to destroy the log
> and
> attempt a repair.
> "
>
> A mount attemp results in
>
> "
> mount: /mnt/test: Falscher Dateisystemtyp, ungültige Optionen, der
> Superblock von /dev/md2 ist beschädigt, fehlende Kodierungsseite oder
> ein anderer Fehler.
> "
>
> And "xfs_repair -L" results in
>
> "
> Phase 1 - Superblock finden und überprüfen...
> - Berichts-Prozess in Abständen von 15 Minutes
> Phase 2 - ein internes Protokoll benutzen
> - Null-Protokoll...
> Log inconsistent or not a log (last==0, first!=1)
> empty log check failed
> zero_log: cannot find log head/tail (xlog_find_tail=22)
> "
>
> Sounds linke some kind of "Hen-egg" problem.
>
> Is there any change to get the XFS file sytem back running without
> recreating the filesystem?
>
> Kernel:
>
> "
> uname -r
> 4.16.12-1-default
> "
>
> Distribution: "Opensuse - Tumbleween"
>
> XFS tools: "xfs_repair Version 4.15.1"
>
> Thank you in advance,
> Kind regards,
>
> Thorsten Hufnagel
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Strange XFS problem
@ 2018-06-03 1:51 Thorsten Hufnagel
2018-06-03 2:32 ` Thorsten Hufnagel
0 siblings, 1 reply; 19+ messages in thread
From: Thorsten Hufnagel @ 2018-06-03 1:51 UTC (permalink / raw)
To: linux-xfs
Hi there,
I've just a big problem with my XFS file system used on a mdraid raid 5
array.
It started with the situation that the mount of my device was not
possible any more (don't know why, hdd's are OK).
"xfs_repair" results in
"
ERROR: The log head and/or tail cannot be discovered. Attempt to mount
the
filesystem to replay the log or use the -L option to destroy the log
and
attempt a repair.
"
"xfs_repair -L " ran, but unfortunately it was interrupted by an
automatic server shutdown after it stuck after stage 7.
Now, "xfs_repair" results in
"
Phase 1 - Superblock finden und überprüfen...
- Berichts-Prozess in Abständen von 15 Minutes
Phase 2 - ein internes Protokoll benutzen
- Null-Protokoll...
Log inconsistent or not a log (last==0, first!=1)
empty log check failed
zero_log: cannot find log head/tail (xlog_find_tail=22)
ERROR: The log head and/or tail cannot be discovered. Attempt to mount
the
filesystem to replay the log or use the -L option to destroy the log
and
attempt a repair.
"
A mount attemp results in
"
mount: /mnt/test: Falscher Dateisystemtyp, ungültige Optionen, der
Superblock von /dev/md2 ist beschädigt, fehlende Kodierungsseite oder
ein anderer Fehler.
"
And "xfs_repair -L" results in
"
Phase 1 - Superblock finden und überprüfen...
- Berichts-Prozess in Abständen von 15 Minutes
Phase 2 - ein internes Protokoll benutzen
- Null-Protokoll...
Log inconsistent or not a log (last==0, first!=1)
empty log check failed
zero_log: cannot find log head/tail (xlog_find_tail=22)
"
Sounds linke some kind of "Hen-egg" problem.
Is there any change to get the XFS file sytem back running without
recreating the filesystem?
Kernel:
"
uname -r
4.16.12-1-default
"
Distribution: "Opensuse - Tumbleween"
XFS tools: "xfs_repair Version 4.15.1"
Thank you in advance,
Kind regards,
Thorsten Hufnagel
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2018-09-13 11:27 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-09-12 8:07 Strange XFS problem Troels Hansen
2018-09-12 10:59 ` Carlos Maiolino
2018-09-12 11:39 ` Troels Hansen
[not found] ` <notmuch-sha1-960c954e5404b5b2f083d150633af0b7848ec14c>
2018-09-12 16:39 ` Carlos Maiolino
2018-09-13 4:19 ` Dave Chinner
2018-09-13 5:21 ` Troels Hansen
2018-09-13 6:18 ` Dave Chinner
-- strict thread matches above, loose matches on Subject: below --
2018-06-03 1:51 Thorsten Hufnagel
2018-06-03 2:32 ` Thorsten Hufnagel
2018-06-05 8:29 ` Carlos Maiolino
2018-06-05 13:46 ` Stefan Ring
2018-06-05 14:19 ` Eric Sandeen
2018-06-05 14:46 ` Stefan Ring
2018-06-05 17:34 ` Chris Murphy
2018-06-12 18:53 ` Stefan Ring
2018-06-12 18:56 ` Eric Sandeen
2018-06-13 22:02 ` Dave Chinner
2018-06-14 2:11 ` Eric Sandeen
2018-06-14 15:47 ` Stefan Ring
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.