* Kernel hangs in cifs_reconnect
@ 2019-10-13 15:55 Martijn de Gouw
2019-10-15 15:33 ` Aurélien Aptel
0 siblings, 1 reply; 9+ messages in thread
From: Martijn de Gouw @ 2019-10-13 15:55 UTC (permalink / raw)
To: linux-cifs
Hi,
Our Linux VMs reports call traces about processes being stuck.
I've attached the full dmesg of one of the call traces below.
The machine is running kernel 5.3.1 SMP. All mounts are mounted via the
dfs shares on our domain controller and have the following options in fstab:
nohandlecache,multiuser,sec=krb5,noperm,user=xxxx,file_mode=0600,dir_mode=0700,vers=3.0
According the mount, this unfolds into:
rw,relatime,vers=3.0,sec=krb5,cruid=0,cache=strict,multiuser,uid=0,noforceuid,gid=0,
noforcegid,addr=10.1.1.2,file_mode=0600,dir_mode=0700,soft,nounix,mapposix,
nohandlecache,noperm,rsize=4194304,wsize=4194304,bsize=1048576,
echo_interval=60,actimeo=1,user=xxxx
After this call trace dump, the log keeps filling with:
CIFS VFS: wait timed out after 60000 ms
Hereby the dmesg output:
[72735.369353] INFO: task node_exporter:3853 blocked for more than 120 seconds.
[72735.370371] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72735.370824] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72735.371230] node_exporter D 0 3853 1 0x00000000
[72735.371246] Call Trace:
[72735.371262] ? __schedule+0x3e9/0x670
[72735.371265] ? __switch_to_asm+0x34/0x70
[72735.371267] schedule+0x39/0xa0
[72735.371268] schedule_preempt_disabled+0xa/0x10
[72735.371270] __mutex_lock.isra.7+0x2a1/0x4d0
[72735.371271] ? __switch_to_asm+0x34/0x70
[72735.371277] ? __switch_to+0x101/0x3e0
[72735.371453] ? smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[72735.371467] smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[72735.371495] ? wait_for_response.isra.11+0x5b/0xf0 [cifs]
[72735.371509] smb2_plain_req_init+0x1d9/0x260 [cifs]
[72735.371523] SMB2_open_init+0x6d/0x870 [cifs]
[72735.371541] ? smb2_query_info_compound+0x173/0x370 [cifs]
[72735.371553] ? cifs_small_buf_release+0x16/0x60 [cifs]
[72735.371567] smb2_query_info_compound+0x173/0x370 [cifs]
[72735.371575] ? __switch_to_asm+0x40/0x70
[72735.371578] ? lookup_fast+0xc8/0x2c0
[72735.371580] ? lookup_fast+0xc8/0x2c0
[72735.371594] ? smb2_queryfs+0x70/0xf0 [cifs]
[72735.371607] smb2_queryfs+0x70/0xf0 [cifs]
[72735.371618] cifs_statfs+0xa9/0x2a0 [cifs]
[72735.371633] statfs_by_dentry+0x6a/0x90
[72735.371638] vfs_statfs+0x16/0xc0
[72735.371639] user_statfs+0x50/0xa0
[72735.371640] __do_sys_statfs+0x20/0x50
[72735.371643] do_syscall_64+0x55/0x110
[72735.371648] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[72735.371657] RIP: 0033:0x4a5c20
[72735.371662] Code: Bad RIP value.
[72735.371663] RSP: 002b:000000c0004c5490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[72735.371664] RAX: ffffffffffffffda RBX: 000000c00002c500 RCX: 00000000004a5c20
[72735.371665] RDX: 0000000000000000 RSI: 000000c0004c55c0 RDI: 000000c00036e180
[72735.371665] RBP: 000000c0004c54f0 R08: 0000000000000000 R09: 0000000000000000
[72735.371666] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[72735.371667] R13: 000000000000000d R14: 000000000000000c R15: 0000000000000100
[72735.371676] INFO: task cifsd:997 blocked for more than 120 seconds.
[72735.372090] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72735.372571] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72735.373032] cifsd D 0 997 2 0x80004000
[72735.373034] Call Trace:
[72735.373039] ? __schedule+0x3e9/0x670
[72735.373042] schedule+0x39/0xa0
[72735.373044] schedule_preempt_disabled+0xa/0x10
[72735.373045] __mutex_lock.isra.7+0x2a1/0x4d0
[72735.373049] ? account_entity_enqueue+0x9c/0xc0
[72735.373075] ? dfs_cache_noreq_find+0x90/0x170 [cifs]
[72735.373089] dfs_cache_noreq_find+0x90/0x170 [cifs]
[72735.373112] cifs_reconnect+0xc5/0xd20 [cifs]
[72735.373132] cifs_handle_standard+0x166/0x190 [cifs]
[72735.373145] cifs_demultiplex_thread+0xaca/0xc20 [cifs]
[72735.373146] ? __switch_to+0x7a/0x3e0
[72735.373158] ? cifs_handle_standard+0x190/0x190 [cifs]
[72735.373160] kthread+0xf8/0x130
[72735.373166] ? kthread_create_worker_on_cpu+0x70/0x70
[72735.373168] ret_from_fork+0x35/0x40
[72856.190414] INFO: task node_exporter:3853 blocked for more than 241 seconds.
[72856.190830] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72856.191171] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72856.191516] node_exporter D 0 3853 1 0x00000000
[72856.191518] Call Trace:
[72856.191533] ? __schedule+0x3e9/0x670
[72856.191535] ? __switch_to_asm+0x34/0x70
[72856.191536] schedule+0x39/0xa0
[72856.191537] schedule_preempt_disabled+0xa/0x10
[72856.191539] __mutex_lock.isra.7+0x2a1/0x4d0
[72856.191540] ? __switch_to_asm+0x34/0x70
[72856.191542] ? __switch_to+0x101/0x3e0
[72856.191571] ? smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[72856.191583] smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[72856.191595] ? wait_for_response.isra.11+0x5b/0xf0 [cifs]
[72856.191607] smb2_plain_req_init+0x1d9/0x260 [cifs]
[72856.191618] SMB2_open_init+0x6d/0x870 [cifs]
[72856.191631] ? smb2_query_info_compound+0x173/0x370 [cifs]
[72856.191641] ? cifs_small_buf_release+0x16/0x60 [cifs]
[72856.191651] smb2_query_info_compound+0x173/0x370 [cifs]
[72856.191653] ? __switch_to_asm+0x40/0x70
[72856.191655] ? lookup_fast+0xc8/0x2c0
[72856.191657] ? lookup_fast+0xc8/0x2c0
[72856.191669] ? smb2_queryfs+0x70/0xf0 [cifs]
[72856.191680] smb2_queryfs+0x70/0xf0 [cifs]
[72856.191688] cifs_statfs+0xa9/0x2a0 [cifs]
[72856.191690] statfs_by_dentry+0x6a/0x90
[72856.191691] vfs_statfs+0x16/0xc0
[72856.191692] user_statfs+0x50/0xa0
[72856.191693] __do_sys_statfs+0x20/0x50
[72856.191695] do_syscall_64+0x55/0x110
[72856.191697] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[72856.191698] RIP: 0033:0x4a5c20
[72856.191701] Code: ff ff ff 48 3b 61 10 0f 86 ba 00 00 00 48 83 ec 40 48 89 6c 24 38 48 8d 6c 24 38 48 8b 44 24 48 48 85 c0 0f 8c 99 00 00 00 48 <3d> 85 00 00 00 0f 8d 8d 00 00 00 48 89 c1 48 c1 e0 04 48 8d 15 c7
[72856.191701] RSP: 002b:000000c0004c5490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[72856.191702] RAX: ffffffffffffffda RBX: 000000c00002c500 RCX: 00000000004a5c20
[72856.191703] RDX: 0000000000000000 RSI: 000000c0004c55c0 RDI: 000000c00036e180
[72856.191703] RBP: 000000c0004c54f0 R08: 0000000000000000 R09: 0000000000000000
[72856.191704] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[72856.191704] R13: 000000000000000d R14: 000000000000000c R15: 0000000000000100
[72856.191720] INFO: task cifsd:997 blocked for more than 241 seconds.
[72856.192084] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72856.192426] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72856.192797] cifsd D 0 997 2 0x80004000
[72856.192799] Call Trace:
[72856.192802] ? __schedule+0x3e9/0x670
[72856.192804] schedule+0x39/0xa0
[72856.192805] schedule_preempt_disabled+0xa/0x10
[72856.192806] __mutex_lock.isra.7+0x2a1/0x4d0
[72856.192809] ? account_entity_enqueue+0x9c/0xc0
[72856.192832] ? dfs_cache_noreq_find+0x90/0x170 [cifs]
[72856.192844] dfs_cache_noreq_find+0x90/0x170 [cifs]
[72856.192853] cifs_reconnect+0xc5/0xd20 [cifs]
[72856.192862] cifs_handle_standard+0x166/0x190 [cifs]
[72856.192870] cifs_demultiplex_thread+0xaca/0xc20 [cifs]
[72856.192872] ? __switch_to+0x7a/0x3e0
[72856.192880] ? cifs_handle_standard+0x190/0x190 [cifs]
[72856.192881] kthread+0xf8/0x130
[72856.192883] ? kthread_create_worker_on_cpu+0x70/0x70
[72856.192884] ret_from_fork+0x35/0x40
[72856.192899] INFO: task kworker/0:1:21401 blocked for more than 120 seconds.
[72856.193253] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72856.193698] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72856.194073] kworker/0:1 D 0 21401 2 0x80004000
[72856.194097] Workqueue: cifsiod refresh_cache_worker [cifs]
[72856.194097] Call Trace:
[72856.194100] ? __schedule+0x3e9/0x670
[72856.194101] schedule+0x39/0xa0
[72856.194102] schedule_preempt_disabled+0xa/0x10
[72856.194104] __mutex_lock.isra.7+0x2a1/0x4d0
[72856.194112] ? __switch_to_asm+0x34/0x70
[72856.194114] ? __switch_to_asm+0x34/0x70
[72856.194115] ? __switch_to_asm+0x40/0x70
[72856.194127] ? refresh_cache_worker+0x2e2/0xee0 [cifs]
[72856.194138] refresh_cache_worker+0x2e2/0xee0 [cifs]
[72856.194140] ? __switch_to_asm+0x34/0x70
[72856.194141] ? __switch_to_asm+0x34/0x70
[72856.194142] ? __switch_to_asm+0x40/0x70
[72856.194143] ? __switch_to_asm+0x34/0x70
[72856.194144] ? __switch_to+0x7a/0x3e0
[72856.194145] ? __switch_to_asm+0x34/0x70
[72856.194148] process_one_work+0x191/0x370
[72856.194150] worker_thread+0x4f/0x3b0
[72856.194151] ? rescuer_thread+0x340/0x340
[72856.194152] kthread+0xf8/0x130
[72856.194154] ? kthread_create_worker_on_cpu+0x70/0x70
[72856.194155] ret_from_fork+0x35/0x40
[72977.011482] INFO: task node_exporter:3853 blocked for more than 362 seconds.
[72977.012158] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72977.012823] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72977.013440] node_exporter D 0 3853 1 0x00000000
[72977.013443] Call Trace:
[72977.013452] ? __schedule+0x3e9/0x670
[72977.013455] ? __switch_to_asm+0x34/0x70
[72977.013456] schedule+0x39/0xa0
[72977.013458] schedule_preempt_disabled+0xa/0x10
[72977.013460] __mutex_lock.isra.7+0x2a1/0x4d0
[72977.013462] ? __switch_to_asm+0x34/0x70
[72977.013465] ? __switch_to+0x101/0x3e0
[72977.013511] ? smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[72977.013529] smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[72977.013547] ? wait_for_response.isra.11+0x5b/0xf0 [cifs]
[72977.013563] smb2_plain_req_init+0x1d9/0x260 [cifs]
[72977.013580] SMB2_open_init+0x6d/0x870 [cifs]
[72977.013596] ? smb2_query_info_compound+0x173/0x370 [cifs]
[72977.013611] ? cifs_small_buf_release+0x16/0x60 [cifs]
[72977.013626] smb2_query_info_compound+0x173/0x370 [cifs]
[72977.013629] ? __switch_to_asm+0x40/0x70
[72977.013633] ? lookup_fast+0xc8/0x2c0
[72977.013635] ? lookup_fast+0xc8/0x2c0
[72977.013652] ? smb2_queryfs+0x70/0xf0 [cifs]
[72977.013669] smb2_queryfs+0x70/0xf0 [cifs]
[72977.013680] cifs_statfs+0xa9/0x2a0 [cifs]
[72977.013684] statfs_by_dentry+0x6a/0x90
[72977.013686] vfs_statfs+0x16/0xc0
[72977.013687] user_statfs+0x50/0xa0
[72977.013689] __do_sys_statfs+0x20/0x50
[72977.013692] do_syscall_64+0x55/0x110
[72977.013694] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[72977.013696] RIP: 0033:0x4a5c20
[72977.013703] Code: Bad RIP value.
[72977.013704] RSP: 002b:000000c0004c5490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[72977.013706] RAX: ffffffffffffffda RBX: 000000c00002c500 RCX: 00000000004a5c20
[72977.013707] RDX: 0000000000000000 RSI: 000000c0004c55c0 RDI: 000000c00036e180
[72977.013707] RBP: 000000c0004c54f0 R08: 0000000000000000 R09: 0000000000000000
[72977.013708] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[72977.013709] R13: 000000000000000d R14: 000000000000000c R15: 0000000000000100
[72977.013740] INFO: task cifsd:997 blocked for more than 362 seconds.
[72977.014226] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72977.014678] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72977.015176] cifsd D 0 997 2 0x80004000
[72977.015179] Call Trace:
[72977.015196] ? __schedule+0x3e9/0x670
[72977.015200] schedule+0x39/0xa0
[72977.015202] schedule_preempt_disabled+0xa/0x10
[72977.015203] __mutex_lock.isra.7+0x2a1/0x4d0
[72977.015207] ? account_entity_enqueue+0x9c/0xc0
[72977.015231] ? dfs_cache_noreq_find+0x90/0x170 [cifs]
[72977.015246] dfs_cache_noreq_find+0x90/0x170 [cifs]
[72977.015259] cifs_reconnect+0xc5/0xd20 [cifs]
[72977.015274] cifs_handle_standard+0x166/0x190 [cifs]
[72977.015288] cifs_demultiplex_thread+0xaca/0xc20 [cifs]
[72977.015290] ? __switch_to+0x7a/0x3e0
[72977.015309] ? cifs_handle_standard+0x190/0x190 [cifs]
[72977.015311] kthread+0xf8/0x130
[72977.015313] ? kthread_create_worker_on_cpu+0x70/0x70
[72977.015322] ret_from_fork+0x35/0x40
[72977.015349] INFO: task kworker/0:1:21401 blocked for more than 241 seconds.
[72977.016210] Tainted: G E 5.3.1-pd-5.3.y #20190925
[72977.016820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[72977.017520] kworker/0:1 D 0 21401 2 0x80004000
[72977.017551] Workqueue: cifsiod refresh_cache_worker [cifs]
[72977.017553] Call Trace:
[72977.017558] ? __schedule+0x3e9/0x670
[72977.017559] schedule+0x39/0xa0
[72977.017561] schedule_preempt_disabled+0xa/0x10
[72977.017562] __mutex_lock.isra.7+0x2a1/0x4d0
[72977.017565] ? __switch_to_asm+0x34/0x70
[72977.017567] ? __switch_to_asm+0x34/0x70
[72977.017568] ? __switch_to_asm+0x40/0x70
[72977.017584] ? refresh_cache_worker+0x2e2/0xee0 [cifs]
[72977.017599] refresh_cache_worker+0x2e2/0xee0 [cifs]
[72977.017603] ? __switch_to_asm+0x34/0x70
[72977.017604] ? __switch_to_asm+0x34/0x70
[72977.017605] ? __switch_to_asm+0x40/0x70
[72977.017606] ? __switch_to_asm+0x34/0x70
[72977.017608] ? __switch_to+0x7a/0x3e0
[72977.017610] ? __switch_to_asm+0x34/0x70
[72977.017614] process_one_work+0x191/0x370
[72977.017616] worker_thread+0x4f/0x3b0
[72977.017618] ? rescuer_thread+0x340/0x340
[72977.017620] kthread+0xf8/0x130
[72977.017621] ? kthread_create_worker_on_cpu+0x70/0x70
[72977.017623] ret_from_fork+0x35/0x40
[73097.836447] INFO: task node_exporter:3853 blocked for more than 483 seconds.
[73097.836911] Tainted: G E 5.3.1-pd-5.3.y #20190925
[73097.837353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[73097.837788] node_exporter D 0 3853 1 0x00000000
[73097.837790] Call Trace:
[73097.837806] ? __schedule+0x3e9/0x670
[73097.837809] ? __switch_to_asm+0x34/0x70
[73097.837810] schedule+0x39/0xa0
[73097.837811] schedule_preempt_disabled+0xa/0x10
[73097.837812] __mutex_lock.isra.7+0x2a1/0x4d0
[73097.837814] ? __switch_to_asm+0x34/0x70
[73097.837816] ? __switch_to+0x101/0x3e0
[73097.837844] ? smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[73097.837857] smb2_reconnect.part.23+0xe1/0x7a0 [cifs]
[73097.837868] ? wait_for_response.isra.11+0x5b/0xf0 [cifs]
[73097.837880] smb2_plain_req_init+0x1d9/0x260 [cifs]
[73097.837892] SMB2_open_init+0x6d/0x870 [cifs]
[73097.837904] ? smb2_query_info_compound+0x173/0x370 [cifs]
[73097.837914] ? cifs_small_buf_release+0x16/0x60 [cifs]
[73097.837925] smb2_query_info_compound+0x173/0x370 [cifs]
[73097.837927] ? __switch_to_asm+0x40/0x70
[73097.837929] ? lookup_fast+0xc8/0x2c0
[73097.837930] ? lookup_fast+0xc8/0x2c0
[73097.837942] ? smb2_queryfs+0x70/0xf0 [cifs]
[73097.837953] smb2_queryfs+0x70/0xf0 [cifs]
[73097.837961] cifs_statfs+0xa9/0x2a0 [cifs]
[73097.837963] statfs_by_dentry+0x6a/0x90
[73097.837964] vfs_statfs+0x16/0xc0
[73097.837965] user_statfs+0x50/0xa0
[73097.837966] __do_sys_statfs+0x20/0x50
[73097.837968] do_syscall_64+0x55/0x110
[73097.837970] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[73097.837971] RIP: 0033:0x4a5c20
[73097.837975] Code: Bad RIP value.
[73097.837976] RSP: 002b:000000c0004c5490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[73097.837977] RAX: ffffffffffffffda RBX: 000000c00002c500 RCX: 00000000004a5c20
[73097.837977] RDX: 0000000000000000 RSI: 000000c0004c55c0 RDI: 000000c00036e180
[73097.837978] RBP: 000000c0004c54f0 R08: 0000000000000000 R09: 0000000000000000
[73097.837978] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[73097.837979] R13: 000000000000000d R14: 000000000000000c R15: 0000000000000100
[73097.837992] INFO: task cifsd:997 blocked for more than 483 seconds.
[73097.838427] Tainted: G E 5.3.1-pd-5.3.y #20190925
[73097.838864] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[73097.839333] cifsd D 0 997 2 0x80004000
[73097.839335] Call Trace:
[73097.839339] ? __schedule+0x3e9/0x670
[73097.839341] schedule+0x39/0xa0
[73097.839342] schedule_preempt_disabled+0xa/0x10
[73097.839349] __mutex_lock.isra.7+0x2a1/0x4d0
[73097.839353] ? account_entity_enqueue+0x9c/0xc0
[73097.839376] ? dfs_cache_noreq_find+0x90/0x170 [cifs]
[73097.839387] dfs_cache_noreq_find+0x90/0x170 [cifs]
[73097.839397] cifs_reconnect+0xc5/0xd20 [cifs]
[73097.839406] cifs_handle_standard+0x166/0x190 [cifs]
[73097.839415] cifs_demultiplex_thread+0xaca/0xc20 [cifs]
[73097.839416] ? __switch_to+0x7a/0x3e0
[73097.839425] ? cifs_handle_standard+0x190/0x190 [cifs]
[73097.839426] kthread+0xf8/0x130
[73097.839427] ? kthread_create_worker_on_cpu+0x70/0x70
[73097.839429] ret_from_fork+0x35/0x40
--
Martijn de Gouw
Designer
Prodrive Technologies
Mobile: +31 63 17 76 161
Phone: +31 40 26 76 200
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Kernel hangs in cifs_reconnect
2019-10-13 15:55 Kernel hangs in cifs_reconnect Martijn de Gouw
@ 2019-10-15 15:33 ` Aurélien Aptel
2019-10-15 16:27 ` Paulo Alcantara
0 siblings, 1 reply; 9+ messages in thread
From: Aurélien Aptel @ 2019-10-15 15:33 UTC (permalink / raw)
To: Martijn de Gouw, linux-cifs
Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
> Our Linux VMs reports call traces about processes being stuck.
> I've attached the full dmesg of one of the call traces below.
>
> The machine is running kernel 5.3.1 SMP. All mounts are mounted via the
> dfs shares on our domain controller and have the following options in fstab:
> nohandlecache,multiuser,sec=krb5,noperm,user=xxxx,file_mode=0600,dir_mode=0700,vers=3.0
It looks like its DFS related. The DFS cache code takes the reconnect
mutex and crashes with no chance to give back the mutex, making all
other process hang while waiting for it.
Cheers,
--
Aurélien Aptel / SUSE Labs Samba Team
GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München)
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Kernel hangs in cifs_reconnect
2019-10-15 15:33 ` Aurélien Aptel
@ 2019-10-15 16:27 ` Paulo Alcantara
2019-10-20 8:13 ` Martijn de Gouw
0 siblings, 1 reply; 9+ messages in thread
From: Paulo Alcantara @ 2019-10-15 16:27 UTC (permalink / raw)
To: Aurélien Aptel, Martijn de Gouw, linux-cifs
Aurélien Aptel <aaptel@suse.com> writes:
> Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
>> Our Linux VMs reports call traces about processes being stuck.
>> I've attached the full dmesg of one of the call traces below.
>>
>> The machine is running kernel 5.3.1 SMP. All mounts are mounted via the
>> dfs shares on our domain controller and have the following options in fstab:
>> nohandlecache,multiuser,sec=krb5,noperm,user=xxxx,file_mode=0600,dir_mode=0700,vers=3.0
>
> It looks like its DFS related. The DFS cache code takes the reconnect
> mutex and crashes with no chance to give back the mutex, making all
> other process hang while waiting for it.
Yeah, makes sense.
Martijn,
Could you please provide us with some debug logs with the following:
# echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
# echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
# echo 1 > /proc/fs/cifs/cifsFYI
# echo 1 > /sys/module/dns_resolver/parameters/debug
Besides, if you could also enable KASAN, that would be great.
Thanks,
Paulo
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Kernel hangs in cifs_reconnect
2019-10-15 16:27 ` Paulo Alcantara
@ 2019-10-20 8:13 ` Martijn de Gouw
2019-10-29 11:07 ` Martijn de Gouw
0 siblings, 1 reply; 9+ messages in thread
From: Martijn de Gouw @ 2019-10-20 8:13 UTC (permalink / raw)
To: Paulo Alcantara, Aurélien Aptel, linux-cifs
Hi Paulo,
On 15-10-2019 18:27, Paulo Alcantara wrote:
> Aurélien Aptel <aaptel@suse.com> writes:
>
>> Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
>>> Our Linux VMs reports call traces about processes being stuck.
>>> I've attached the full dmesg of one of the call traces below.
>>>
>>> The machine is running kernel 5.3.1 SMP. All mounts are mounted via the
>>> dfs shares on our domain controller and have the following options in fstab:
>>> nohandlecache,multiuser,sec=krb5,noperm,user=xxxx,file_mode=0600,dir_mode=0700,vers=3.0
>>
>> It looks like its DFS related. The DFS cache code takes the reconnect
>> mutex and crashes with no chance to give back the mutex, making all
>> other process hang while waiting for it.
>
> Yeah, makes sense.
>
> Martijn,
>
> Could you please provide us with some debug logs with the following:
>
> # echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
> # echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
> # echo 1 > /proc/fs/cifs/cifsFYI
> # echo 1 > /sys/module/dns_resolver/parameters/debug
>
> Besides, if you could also enable KASAN, that would be great.
I recompiled the same kernel with KASAN enabled and have it run again for several days.
See below for the stackdump, I've added some of the messages before and after the event,
but if you need more information, please let me know.
[373004.186807] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 73254 with uid: 999
[373004.186883] fs/cifs/transport.c: Sending smb: smb_len=348
[373004.187349] fs/cifs/connect.c: RFC1002 header 0x1b8
[373004.187358] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[373004.187359] fs/cifs/smb2misc.c: SMB2 len 208
[373004.187370] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[373004.187371] fs/cifs/smb2misc.c: SMB2 len 104
[373004.187381] fs/cifs/smb2misc.c: SMB2 len 124
[373004.187383] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 759
[373004.187386] fs/cifs/smb2ops.c: add 30 credits total=7026
[373004.187401] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=757 state=4
[373004.187404] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=758 state=4
[373004.187405] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=759 state=4
[373004.187407] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373004.187414] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373004.187418] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373004.187451] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 73254) rc = 0
[373031.153835] fs/cifs/smb2pdu.c: In echo request
[373031.153853] fs/cifs/smb2pdu.c: Echo request failed: -11
[373031.153869] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373064.180943] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 73255 with uid: 999
[373064.181026] fs/cifs/transport.c: Sending smb: smb_len=348
[373064.181572] fs/cifs/connect.c: Received no data or error: -104
[373092.592054] fs/cifs/smb2pdu.c: In echo request
[373092.592067] fs/cifs/smb2pdu.c: Echo request failed: -11
[373092.592073] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373096.683689] INFO: task cifsd:789 blocked for more than 120 seconds.
[373096.684890] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373096.685859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373096.686809] cifsd D 0 789 2 0x80004000
[373096.686824] Call Trace:
[373096.686838] ? __schedule+0x540/0xac0
[373096.686841] ? firmware_map_remove+0xe9/0xe9
[373096.686850] ? string_nocheck+0xb0/0xd0
[373096.686857] ? _raw_spin_lock+0x7a/0xd0
[373096.686858] schedule+0x5e/0x100
[373096.686861] schedule_preempt_disabled+0xa/0x10
[373096.686863] __mutex_lock.isra.4+0x484/0x820
[373096.686866] ? mutex_trylock+0x90/0x90
[373096.686873] ? irq_work_claim+0x2e/0x50
[373096.686875] ? irq_work_queue+0x9/0x20
[373096.686878] ? mutex_lock+0xce/0xe0
[373096.686879] mutex_lock+0xce/0xe0
[373096.686881] ? __mutex_lock_slowpath+0x10/0x10
[373096.686934] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373096.686975] cifs_reconnect+0x16c/0x1360 [cifs]
[373096.687019] ? smb2_calc_size+0x15c/0x250 [cifs]
[373096.687058] ? extract_hostname+0xa0/0xa0 [cifs]
[373096.687060] ? _raw_spin_trylock+0x91/0xe0
[373096.687062] ? _raw_spin_trylock_bh+0x100/0x100
[373096.687064] ? ___ratelimit+0x106/0x190
[373096.687104] cifs_handle_standard+0x252/0x270 [cifs]
[373096.687144] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[373096.687184] ? cifs_handle_standard+0x270/0x270 [cifs]
[373096.687186] ? __switch_to_asm+0x40/0x70
[373096.687188] ? __switch_to_asm+0x34/0x70
[373096.687189] ? __switch_to_asm+0x40/0x70
[373096.687191] ? __switch_to_asm+0x34/0x70
[373096.687192] ? __switch_to_asm+0x40/0x70
[373096.687194] ? __switch_to_asm+0x34/0x70
[373096.687196] ? __switch_to_asm+0x40/0x70
[373096.687197] ? __switch_to_asm+0x34/0x70
[373096.687199] ? __switch_to_asm+0x40/0x70
[373096.687200] ? __switch_to_asm+0x34/0x70
[373096.687202] ? __switch_to_asm+0x40/0x70
[373096.687203] ? __switch_to_asm+0x34/0x70
[373096.687205] ? __switch_to_asm+0x40/0x70
[373096.687206] ? __switch_to_asm+0x34/0x70
[373096.687208] ? __switch_to_asm+0x40/0x70
[373096.687209] ? __switch_to_asm+0x34/0x70
[373096.687211] ? __switch_to_asm+0x40/0x70
[373096.687212] ? __switch_to_asm+0x34/0x70
[373096.687214] ? __switch_to_asm+0x40/0x70
[373096.687215] ? __switch_to_asm+0x34/0x70
[373096.687217] ? __switch_to_asm+0x40/0x70
[373096.687218] ? __switch_to_asm+0x34/0x70
[373096.687220] ? __switch_to_asm+0x40/0x70
[373096.687221] ? __switch_to_asm+0x34/0x70
[373096.687223] ? __switch_to_asm+0x40/0x70
[373096.687224] ? __switch_to_asm+0x34/0x70
[373096.687226] ? __switch_to_asm+0x40/0x70
[373096.687232] ? finish_task_switch+0x91/0x370
[373096.687234] ? __switch_to+0x2ec/0x5e0
[373096.687237] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373096.687239] ? _raw_write_lock_bh+0xe0/0xe0
[373096.687277] ? cifs_handle_standard+0x270/0x270 [cifs]
[373096.687280] kthread+0x192/0x1e0
[373096.687282] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373096.687284] ret_from_fork+0x35/0x40
[373096.687326] fs/cifs/smb2pdu.c: In echo request
[373096.687354] CIFS VFS: Error -32 sending data on socket to server
[373096.688357] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373096.688369] fs/cifs/smb2pdu.c: Echo request failed: -32
[373096.688375] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373149.926651] fs/cifs/connect.c: Existing tcp session with server found
[373149.926658] fs/cifs/dfs_cache.c: CIFS VFS: in do_refresh_tcon as Xid: 73256 with uid: 0
[373154.022274] fs/cifs/smb2pdu.c: In echo request
[373154.022290] fs/cifs/smb2pdu.c: Echo request failed: -11
[373154.022298] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373158.121876] fs/cifs/smb2pdu.c: In echo request
[373158.121904] CIFS VFS: Error -32 sending data on socket to server
[373158.123065] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373158.123079] fs/cifs/smb2pdu.c: Echo request failed: -32
[373158.123089] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373161.358480] fs/cifs/inode.c: CIFS VFS: in cifs_revalidate_dentry_attr as Xid: 73257 with uid: 11025
[373161.358496] fs/cifs/dir.c: name: \KAES6309
[373161.358500] fs/cifs/inode.c: Update attributes: \KAES6309 inode 0x00000000ae3f689c count 1 dentry: 0x00000000661f7ca9 d_time 4379115881 jiffies 4388191018
[373161.358503] fs/cifs/inode.c: Getting info on \KAES6309
[373161.358682] fs/cifs/transport.c: Sending smb: smb_len=388
[373215.456445] fs/cifs/smb2pdu.c: In echo request
[373215.456461] fs/cifs/smb2pdu.c: Echo request failed: -11
[373215.456467] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373217.504315] INFO: task cifsd:789 blocked for more than 241 seconds.
[373217.505647] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373217.506642] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373217.507653] cifsd D 0 789 2 0x80004000
[373217.507657] Call Trace:
[373217.507664] ? __schedule+0x540/0xac0
[373217.507667] ? firmware_map_remove+0xe9/0xe9
[373217.507670] ? string_nocheck+0xb0/0xd0
[373217.507676] ? _raw_spin_lock+0x7a/0xd0
[373217.507677] schedule+0x5e/0x100
[373217.507680] schedule_preempt_disabled+0xa/0x10
[373217.507682] __mutex_lock.isra.4+0x484/0x820
[373217.507685] ? mutex_trylock+0x90/0x90
[373217.507688] ? irq_work_claim+0x2e/0x50
[373217.507690] ? irq_work_queue+0x9/0x20
[373217.507693] ? mutex_lock+0xce/0xe0
[373217.507694] mutex_lock+0xce/0xe0
[373217.507696] ? __mutex_lock_slowpath+0x10/0x10
[373217.507750] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373217.507791] cifs_reconnect+0x16c/0x1360 [cifs]
[373217.507834] ? smb2_calc_size+0x15c/0x250 [cifs]
[373217.507873] ? extract_hostname+0xa0/0xa0 [cifs]
[373217.507875] ? _raw_spin_trylock+0x91/0xe0
[373217.507877] ? _raw_spin_trylock_bh+0x100/0x100
[373217.507879] ? ___ratelimit+0x106/0x190
[373217.507919] cifs_handle_standard+0x252/0x270 [cifs]
[373217.507959] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[373217.507999] ? cifs_handle_standard+0x270/0x270 [cifs]
[373217.508001] ? __switch_to_asm+0x40/0x70
[373217.508003] ? __switch_to_asm+0x34/0x70
[373217.508004] ? __switch_to_asm+0x40/0x70
[373217.508006] ? __switch_to_asm+0x34/0x70
[373217.508008] ? __switch_to_asm+0x40/0x70
[373217.508009] ? __switch_to_asm+0x34/0x70
[373217.508011] ? __switch_to_asm+0x40/0x70
[373217.508012] ? __switch_to_asm+0x34/0x70
[373217.508014] ? __switch_to_asm+0x40/0x70
[373217.508015] ? __switch_to_asm+0x34/0x70
[373217.508017] ? __switch_to_asm+0x40/0x70
[373217.508018] ? __switch_to_asm+0x34/0x70
[373217.508020] ? __switch_to_asm+0x40/0x70
[373217.508021] ? __switch_to_asm+0x34/0x70
[373217.508023] ? __switch_to_asm+0x40/0x70
[373217.508024] ? __switch_to_asm+0x34/0x70
[373217.508026] ? __switch_to_asm+0x40/0x70
[373217.508028] ? __switch_to_asm+0x34/0x70
[373217.508029] ? __switch_to_asm+0x40/0x70
[373217.508031] ? __switch_to_asm+0x34/0x70
[373217.508032] ? __switch_to_asm+0x40/0x70
[373217.508034] ? __switch_to_asm+0x34/0x70
[373217.508035] ? __switch_to_asm+0x40/0x70
[373217.508037] ? __switch_to_asm+0x34/0x70
[373217.508038] ? __switch_to_asm+0x40/0x70
[373217.508040] ? __switch_to_asm+0x34/0x70
[373217.508041] ? __switch_to_asm+0x40/0x70
[373217.508044] ? finish_task_switch+0x91/0x370
[373217.508046] ? __switch_to+0x2ec/0x5e0
[373217.508049] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373217.508050] ? _raw_write_lock_bh+0xe0/0xe0
[373217.508090] ? cifs_handle_standard+0x270/0x270 [cifs]
[373217.508092] kthread+0x192/0x1e0
[373217.508094] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373217.508096] ret_from_fork+0x35/0x40
[373217.508118] INFO: task cifsd:16935 blocked for more than 120 seconds.
[373217.509148] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373217.510152] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373217.511174] cifsd D 0 16935 2 0x80004000
[373217.511177] Call Trace:
[373217.511181] ? __schedule+0x540/0xac0
[373217.511184] ? firmware_map_remove+0xe9/0xe9
[373217.511187] ? vsnprintf+0x870/0x870
[373217.511189] ? _raw_spin_lock+0x7a/0xd0
[373217.511191] schedule+0x5e/0x100
[373217.511193] schedule_preempt_disabled+0xa/0x10
[373217.511194] __mutex_lock.isra.4+0x484/0x820
[373217.511197] ? mutex_trylock+0x90/0x90
[373217.511226] ? up+0x32/0x70
[373217.511229] ? irq_work_claim+0x2e/0x50
[373217.511231] ? irq_work_queue+0x9/0x20
[373217.511234] ? vprintk_emit+0x11d/0x2e0
[373217.511236] ? mutex_lock+0xce/0xe0
[373217.511238] mutex_lock+0xce/0xe0
[373217.511240] ? __mutex_lock_slowpath+0x10/0x10
[373217.511290] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373217.511330] cifs_reconnect+0x16c/0x1360 [cifs]
[373217.511371] ? extract_hostname+0xa0/0xa0 [cifs]
[373217.511374] ? _raw_spin_trylock+0x91/0xe0
[373217.511376] ? _raw_spin_trylock_bh+0x100/0x100
[373217.511379] ? aa_sk_perm+0xe4/0x1f0
[373217.511382] ? inet_release+0xc0/0xc0
[373217.511384] ? ___ratelimit+0x106/0x190
[373217.511423] cifs_readv_from_socket+0x319/0x390 [cifs]
[373217.511463] cifs_read_from_socket+0x9d/0xe0 [cifs]
[373217.511503] ? cifs_readv_from_socket+0x390/0x390 [cifs]
[373217.511507] ? refcount_sub_and_test_checked+0xae/0x140
[373217.511548] ? cifs_small_buf_get+0x37/0x50 [cifs]
[373217.511588] ? allocate_buffers+0x10a/0x170 [cifs]
[373217.511627] cifs_demultiplex_thread+0x241/0x13e0 [cifs]
[373217.511667] ? cifs_handle_standard+0x270/0x270 [cifs]
[373217.511671] ? sched_clock+0x5/0x10
[373217.511673] ? __switch_to_asm+0x40/0x70
[373217.511675] ? __switch_to_asm+0x34/0x70
[373217.511677] ? __switch_to_asm+0x40/0x70
[373217.511678] ? __switch_to_asm+0x34/0x70
[373217.511681] ? __switch_to_asm+0x40/0x70
[373217.511683] ? __switch_to_asm+0x34/0x70
[373217.511684] ? __switch_to_asm+0x40/0x70
[373217.511686] ? __switch_to_asm+0x34/0x70
[373217.511687] ? __switch_to_asm+0x40/0x70
[373217.511689] ? __switch_to_asm+0x34/0x70
[373217.511690] ? __switch_to_asm+0x40/0x70
[373217.511692] ? __switch_to_asm+0x34/0x70
[373217.511693] ? __switch_to_asm+0x40/0x70
[373217.511695] ? __switch_to_asm+0x34/0x70
[373217.511696] ? __switch_to_asm+0x40/0x70
[373217.511698] ? __switch_to_asm+0x34/0x70
[373217.511700] ? __switch_to_asm+0x40/0x70
[373217.511701] ? __switch_to_asm+0x34/0x70
[373217.511703] ? __switch_to_asm+0x40/0x70
[373217.511704] ? __switch_to_asm+0x34/0x70
[373217.511706] ? __switch_to_asm+0x40/0x70
[373217.511707] ? __switch_to_asm+0x34/0x70
[373217.511709] ? __switch_to_asm+0x40/0x70
[373217.511710] ? __switch_to_asm+0x34/0x70
[373217.511712] ? __switch_to_asm+0x40/0x70
[373217.511713] ? __switch_to_asm+0x34/0x70
[373217.511716] ? finish_task_switch+0xf6/0x370
[373217.511717] ? __switch_to+0x2ec/0x5e0
[373217.511720] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373217.511722] ? _raw_write_lock_bh+0xe0/0xe0
[373217.511761] ? cifs_handle_standard+0x270/0x270 [cifs]
[373217.511763] kthread+0x192/0x1e0
[373217.511766] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373217.511768] ret_from_fork+0x35/0x40
[373219.552028] fs/cifs/smb2pdu.c: In echo request
[373219.552061] CIFS VFS: Error -32 sending data on socket to server
[373219.553244] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373219.553254] fs/cifs/smb2pdu.c: Echo request failed: -32
[373219.553261] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373276.894645] fs/cifs/smb2pdu.c: In echo request
[373276.894669] fs/cifs/smb2pdu.c: Echo request failed: -11
[373276.894675] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373280.986191] fs/cifs/smb2pdu.c: In echo request
[373280.986219] CIFS VFS: Error -32 sending data on socket to server
[373280.987409] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373280.987420] fs/cifs/smb2pdu.c: Echo request failed: -32
[373280.987426] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373338.324793] fs/cifs/smb2pdu.c: In echo request
[373338.324832] fs/cifs/smb2pdu.c: Echo request failed: -11
[373338.324838] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373338.324918] INFO: task cifsd:789 blocked for more than 362 seconds.
[373338.326236] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373338.327346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373338.328455] cifsd D 0 789 2 0x80004000
[373338.328458] Call Trace:
[373338.328465] ? __schedule+0x540/0xac0
[373338.328468] ? firmware_map_remove+0xe9/0xe9
[373338.328471] ? string_nocheck+0xb0/0xd0
[373338.328475] ? _raw_spin_lock+0x7a/0xd0
[373338.328477] schedule+0x5e/0x100
[373338.328479] schedule_preempt_disabled+0xa/0x10
[373338.328481] __mutex_lock.isra.4+0x484/0x820
[373338.328484] ? mutex_trylock+0x90/0x90
[373338.328488] ? irq_work_claim+0x2e/0x50
[373338.328490] ? irq_work_queue+0x9/0x20
[373338.328492] ? mutex_lock+0xce/0xe0
[373338.328494] mutex_lock+0xce/0xe0
[373338.328496] ? __mutex_lock_slowpath+0x10/0x10
[373338.328547] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373338.328588] cifs_reconnect+0x16c/0x1360 [cifs]
[373338.328631] ? smb2_calc_size+0x15c/0x250 [cifs]
[373338.328670] ? extract_hostname+0xa0/0xa0 [cifs]
[373338.328672] ? _raw_spin_trylock+0x91/0xe0
[373338.328674] ? _raw_spin_trylock_bh+0x100/0x100
[373338.328676] ? ___ratelimit+0x106/0x190
[373338.328716] cifs_handle_standard+0x252/0x270 [cifs]
[373338.328758] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[373338.328813] ? cifs_handle_standard+0x270/0x270 [cifs]
[373338.328818] ? __switch_to_asm+0x40/0x70
[373338.328824] ? __switch_to_asm+0x34/0x70
[373338.328828] ? __switch_to_asm+0x40/0x70
[373338.328834] ? __switch_to_asm+0x34/0x70
[373338.328840] ? __switch_to_asm+0x40/0x70
[373338.328843] ? __switch_to_asm+0x34/0x70
[373338.328849] ? __switch_to_asm+0x40/0x70
[373338.328853] ? __switch_to_asm+0x34/0x70
[373338.328859] ? __switch_to_asm+0x40/0x70
[373338.328871] ? __switch_to_asm+0x34/0x70
[373338.328873] ? __switch_to_asm+0x40/0x70
[373338.328874] ? __switch_to_asm+0x34/0x70
[373338.328876] ? __switch_to_asm+0x40/0x70
[373338.328877] ? __switch_to_asm+0x34/0x70
[373338.328879] ? __switch_to_asm+0x40/0x70
[373338.328880] ? __switch_to_asm+0x34/0x70
[373338.328882] ? __switch_to_asm+0x40/0x70
[373338.328883] ? __switch_to_asm+0x34/0x70
[373338.328885] ? __switch_to_asm+0x40/0x70
[373338.328886] ? __switch_to_asm+0x34/0x70
[373338.328888] ? __switch_to_asm+0x40/0x70
[373338.328889] ? __switch_to_asm+0x34/0x70
[373338.328891] ? __switch_to_asm+0x40/0x70
[373338.328893] ? __switch_to_asm+0x34/0x70
[373338.328894] ? __switch_to_asm+0x40/0x70
[373338.328896] ? __switch_to_asm+0x34/0x70
[373338.328897] ? __switch_to_asm+0x40/0x70
[373338.328900] ? finish_task_switch+0x91/0x370
[373338.328903] ? __switch_to+0x2ec/0x5e0
[373338.328905] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373338.328907] ? _raw_write_lock_bh+0xe0/0xe0
[373338.328947] ? cifs_handle_standard+0x270/0x270 [cifs]
[373338.328949] kthread+0x192/0x1e0
[373338.328952] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373338.328954] ret_from_fork+0x35/0x40
[373338.328975] INFO: task cifsd:16935 blocked for more than 241 seconds.
[373338.330061] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373338.331156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373338.332283] cifsd D 0 16935 2 0x80004000
[373338.332286] Call Trace:
[373338.332296] ? __schedule+0x540/0xac0
[373338.332298] ? firmware_map_remove+0xe9/0xe9
[373338.332301] ? vsnprintf+0x870/0x870
[373338.332304] ? _raw_spin_lock+0x7a/0xd0
[373338.332306] schedule+0x5e/0x100
[373338.332308] schedule_preempt_disabled+0xa/0x10
[373338.332310] __mutex_lock.isra.4+0x484/0x820
[373338.332312] ? mutex_trylock+0x90/0x90
[373338.332315] ? up+0x32/0x70
[373338.332319] ? irq_work_claim+0x2e/0x50
[373338.332320] ? irq_work_queue+0x9/0x20
[373338.332323] ? vprintk_emit+0x11d/0x2e0
[373338.332325] ? mutex_lock+0xce/0xe0
[373338.332326] mutex_lock+0xce/0xe0
[373338.332328] ? __mutex_lock_slowpath+0x10/0x10
[373338.332381] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373338.332421] cifs_reconnect+0x16c/0x1360 [cifs]
[373338.332461] ? extract_hostname+0xa0/0xa0 [cifs]
[373338.332463] ? _raw_spin_trylock+0x91/0xe0
[373338.332465] ? _raw_spin_trylock_bh+0x100/0x100
[373338.332467] ? aa_sk_perm+0xe4/0x1f0
[373338.332470] ? inet_release+0xc0/0xc0
[373338.332472] ? ___ratelimit+0x106/0x190
[373338.332511] cifs_readv_from_socket+0x319/0x390 [cifs]
[373338.332551] cifs_read_from_socket+0x9d/0xe0 [cifs]
[373338.332590] ? cifs_readv_from_socket+0x390/0x390 [cifs]
[373338.332594] ? refcount_sub_and_test_checked+0xae/0x140
[373338.332635] ? cifs_small_buf_get+0x37/0x50 [cifs]
[373338.332674] ? allocate_buffers+0x10a/0x170 [cifs]
[373338.332713] cifs_demultiplex_thread+0x241/0x13e0 [cifs]
[373338.332755] ? cifs_handle_standard+0x270/0x270 [cifs]
[373338.332774] ? sched_clock+0x5/0x10
[373338.332776] ? __switch_to_asm+0x40/0x70
[373338.332778] ? __switch_to_asm+0x34/0x70
[373338.332779] ? __switch_to_asm+0x40/0x70
[373338.332781] ? __switch_to_asm+0x34/0x70
[373338.332782] ? __switch_to_asm+0x40/0x70
[373338.332784] ? __switch_to_asm+0x34/0x70
[373338.332786] ? __switch_to_asm+0x40/0x70
[373338.332787] ? __switch_to_asm+0x34/0x70
[373338.332791] ? __switch_to_asm+0x40/0x70
[373338.332798] ? __switch_to_asm+0x34/0x70
[373338.332802] ? __switch_to_asm+0x40/0x70
[373338.332808] ? __switch_to_asm+0x34/0x70
[373338.332810] ? __switch_to_asm+0x40/0x70
[373338.332813] ? __switch_to_asm+0x34/0x70
[373338.332822] ? __switch_to_asm+0x40/0x70
[373338.332828] ? __switch_to_asm+0x34/0x70
[373338.332830] ? __switch_to_asm+0x40/0x70
[373338.332834] ? __switch_to_asm+0x34/0x70
[373338.332840] ? __switch_to_asm+0x40/0x70
[373338.332856] ? __switch_to_asm+0x34/0x70
[373338.332858] ? __switch_to_asm+0x40/0x70
[373338.332859] ? __switch_to_asm+0x34/0x70
[373338.332861] ? __switch_to_asm+0x40/0x70
[373338.332862] ? __switch_to_asm+0x34/0x70
[373338.332864] ? __switch_to_asm+0x40/0x70
[373338.332865] ? __switch_to_asm+0x34/0x70
[373338.332868] ? finish_task_switch+0xf6/0x370
[373338.332870] ? __switch_to+0x2ec/0x5e0
[373338.332872] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373338.332874] ? _raw_write_lock_bh+0xe0/0xe0
[373338.332913] ? cifs_handle_standard+0x270/0x270 [cifs]
[373338.332923] kthread+0x192/0x1e0
[373338.332927] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373338.332930] ret_from_fork+0x35/0x40
[373338.332958] INFO: task kworker/2:1:31242 blocked for more than 120 seconds.
[373338.334112] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373338.335285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373338.336617] kworker/2:1 D 0 31242 2 0x80004000
[373338.336700] Workqueue: cifsiod refresh_cache_worker [cifs]
[373338.336702] Call Trace:
[373338.336709] ? __schedule+0x540/0xac0
[373338.336714] ? firmware_map_remove+0xe9/0xe9
[373338.336717] ? _raw_read_lock_irq+0x40/0x40
[373338.336721] schedule+0x5e/0x100
[373338.336725] schedule_preempt_disabled+0xa/0x10
[373338.336728] __mutex_lock.isra.4+0x484/0x820
[373338.336733] ? mutex_trylock+0x90/0x90
[373338.336737] ? dynamic_emit_prefix+0x29/0x220
[373338.336740] ? __dynamic_pr_debug+0xf8/0x140
[373338.336744] ? dynamic_emit_prefix+0x220/0x220
[373338.336770] ? mutex_lock+0xce/0xe0
[373338.336773] mutex_lock+0xce/0xe0
[373338.336777] ? __mutex_lock_slowpath+0x10/0x10
[373338.336856] refresh_cache_worker+0x48f/0x14a0 [cifs]
[373338.336864] ? __switch_to_asm+0x40/0x70
[373338.336875] ? __switch_to_asm+0x40/0x70
[373338.336883] ? __switch_to_asm+0x34/0x70
[373338.336885] ? __switch_to_asm+0x40/0x70
[373338.336888] ? __switch_to_asm+0x34/0x70
[373338.336891] ? __switch_to_asm+0x40/0x70
[373338.336964] ? find_root_ses.isra.9+0x320/0x320 [cifs]
[373338.336968] ? __switch_to_asm+0x40/0x70
[373338.336976] ? __switch_to_asm+0x40/0x70
[373338.336979] ? __switch_to_asm+0x34/0x70
[373338.336981] ? __switch_to_asm+0x40/0x70
[373338.336984] ? __switch_to_asm+0x34/0x70
[373338.336991] ? __switch_to_asm+0x40/0x70
[373338.336994] ? __switch_to_asm+0x34/0x70
[373338.336998] ? __switch_to_asm+0x40/0x70
[373338.337004] ? __switch_to_asm+0x40/0x70
[373338.337007] ? __switch_to_asm+0x34/0x70
[373338.337010] ? finish_task_switch+0xf6/0x370
[373338.337016] ? __switch_to+0x2ec/0x5e0
[373338.337019] ? __schedule+0x562/0xac0
[373338.337023] ? read_word_at_a_time+0xe/0x20
[373338.337028] ? strscpy+0xca/0x1d0
[373338.337034] process_one_work+0x373/0x6e0
[373338.337040] worker_thread+0x78/0x5b0
[373338.337047] ? rescuer_thread+0x5e0/0x5e0
[373338.337050] kthread+0x192/0x1e0
[373338.337054] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373338.337058] ret_from_fork+0x35/0x40
[373342.420422] fs/cifs/smb2pdu.c: In echo request
[373342.420455] CIFS VFS: Error -32 sending data on socket to server
[373342.421797] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373342.421809] fs/cifs/smb2pdu.c: Echo request failed: -32
[373342.421816] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373399.758993] fs/cifs/smb2pdu.c: In echo request
[373399.759010] fs/cifs/smb2pdu.c: Echo request failed: -11
[373399.759017] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373403.854642] fs/cifs/smb2pdu.c: In echo request
[373403.854666] CIFS VFS: Error -32 sending data on socket to server
[373403.856180] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373403.856195] fs/cifs/smb2pdu.c: Echo request failed: -32
[373403.856202] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373459.145452] INFO: task cifsd:789 blocked for more than 483 seconds.
[373459.147021] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373459.148332] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373459.149648] cifsd D 0 789 2 0x80004000
[373459.149657] Call Trace:
[373459.149682] ? __schedule+0x540/0xac0
[373459.149687] ? firmware_map_remove+0xe9/0xe9
[373459.149690] ? string_nocheck+0xb0/0xd0
[373459.149694] ? _raw_spin_lock+0x7a/0xd0
[373459.149698] schedule+0x5e/0x100
[373459.149700] schedule_preempt_disabled+0xa/0x10
[373459.149703] __mutex_lock.isra.4+0x484/0x820
[373459.149706] ? mutex_trylock+0x90/0x90
[373459.149714] ? irq_work_claim+0x2e/0x50
[373459.149716] ? irq_work_queue+0x9/0x20
[373459.149719] ? mutex_lock+0xce/0xe0
[373459.149720] mutex_lock+0xce/0xe0
[373459.149722] ? __mutex_lock_slowpath+0x10/0x10
[373459.149775] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373459.149818] cifs_reconnect+0x16c/0x1360 [cifs]
[373459.149870] ? smb2_calc_size+0x15c/0x250 [cifs]
[373459.149909] ? extract_hostname+0xa0/0xa0 [cifs]
[373459.149911] ? _raw_spin_trylock+0x91/0xe0
[373459.149913] ? _raw_spin_trylock_bh+0x100/0x100
[373459.149915] ? ___ratelimit+0x106/0x190
[373459.149960] cifs_handle_standard+0x252/0x270 [cifs]
[373459.150001] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[373459.150041] ? cifs_handle_standard+0x270/0x270 [cifs]
[373459.150043] ? __switch_to_asm+0x40/0x70
[373459.150045] ? __switch_to_asm+0x34/0x70
[373459.150047] ? __switch_to_asm+0x40/0x70
[373459.150050] ? __switch_to_asm+0x34/0x70
[373459.150051] ? __switch_to_asm+0x40/0x70
[373459.150053] ? __switch_to_asm+0x34/0x70
[373459.150055] ? __switch_to_asm+0x40/0x70
[373459.150058] ? __switch_to_asm+0x34/0x70
[373459.150061] ? __switch_to_asm+0x40/0x70
[373459.150063] ? __switch_to_asm+0x34/0x70
[373459.150066] ? __switch_to_asm+0x40/0x70
[373459.150068] ? __switch_to_asm+0x34/0x70
[373459.150070] ? __switch_to_asm+0x40/0x70
[373459.150072] ? __switch_to_asm+0x34/0x70
[373459.150074] ? __switch_to_asm+0x40/0x70
[373459.150077] ? __switch_to_asm+0x34/0x70
[373459.150078] ? __switch_to_asm+0x40/0x70
[373459.150080] ? __switch_to_asm+0x34/0x70
[373459.150081] ? __switch_to_asm+0x40/0x70
[373459.150083] ? __switch_to_asm+0x34/0x70
[373459.150084] ? __switch_to_asm+0x40/0x70
[373459.150086] ? __switch_to_asm+0x34/0x70
[373459.150087] ? __switch_to_asm+0x40/0x70
[373459.150089] ? __switch_to_asm+0x34/0x70
[373459.150090] ? __switch_to_asm+0x40/0x70
[373459.150092] ? __switch_to_asm+0x34/0x70
[373459.150093] ? __switch_to_asm+0x40/0x70
[373459.150102] ? finish_task_switch+0x91/0x370
[373459.150104] ? __switch_to+0x2ec/0x5e0
[373459.150106] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373459.150108] ? _raw_write_lock_bh+0xe0/0xe0
[373459.150149] ? cifs_handle_standard+0x270/0x270 [cifs]
[373459.150152] kthread+0x192/0x1e0
[373459.150154] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373459.150156] ret_from_fork+0x35/0x40
[373459.150175] INFO: task cifsd:16935 blocked for more than 362 seconds.
[373459.151549] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373459.152979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373459.154378] cifsd D 0 16935 2 0x80004000
[373459.154381] Call Trace:
[373459.154387] ? __schedule+0x540/0xac0
[373459.154483] ? firmware_map_remove+0xe9/0xe9
[373459.154487] ? vsnprintf+0x870/0x870
[373459.154491] ? _raw_spin_lock+0x7a/0xd0
[373459.154494] schedule+0x5e/0x100
[373459.154497] schedule_preempt_disabled+0xa/0x10
[373459.154499] __mutex_lock.isra.4+0x484/0x820
[373459.154502] ? mutex_trylock+0x90/0x90
[373459.154511] ? up+0x32/0x70
[373459.154514] ? irq_work_claim+0x2e/0x50
[373459.154515] ? irq_work_queue+0x9/0x20
[373459.154518] ? vprintk_emit+0x11d/0x2e0
[373459.154520] ? mutex_lock+0xce/0xe0
[373459.154521] mutex_lock+0xce/0xe0
[373459.154524] ? __mutex_lock_slowpath+0x10/0x10
[373459.154576] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373459.154616] cifs_reconnect+0x16c/0x1360 [cifs]
[373459.154658] ? extract_hostname+0xa0/0xa0 [cifs]
[373459.154661] ? _raw_spin_trylock+0x91/0xe0
[373459.154663] ? _raw_spin_trylock_bh+0x100/0x100
[373459.154671] ? aa_sk_perm+0xe4/0x1f0
[373459.154677] ? inet_release+0xc0/0xc0
[373459.154679] ? ___ratelimit+0x106/0x190
[373459.154717] cifs_readv_from_socket+0x319/0x390 [cifs]
[373459.154757] cifs_read_from_socket+0x9d/0xe0 [cifs]
[373459.154796] ? cifs_readv_from_socket+0x390/0x390 [cifs]
[373459.154799] ? refcount_sub_and_test_checked+0xae/0x140
[373459.154840] ? cifs_small_buf_get+0x37/0x50 [cifs]
[373459.154881] ? allocate_buffers+0x10a/0x170 [cifs]
[373459.154920] cifs_demultiplex_thread+0x241/0x13e0 [cifs]
[373459.154960] ? cifs_handle_standard+0x270/0x270 [cifs]
[373459.154963] ? sched_clock+0x5/0x10
[373459.154965] ? __switch_to_asm+0x40/0x70
[373459.154966] ? __switch_to_asm+0x34/0x70
[373459.154969] ? __switch_to_asm+0x40/0x70
[373459.154971] ? __switch_to_asm+0x34/0x70
[373459.154972] ? __switch_to_asm+0x40/0x70
[373459.154974] ? __switch_to_asm+0x34/0x70
[373459.154975] ? __switch_to_asm+0x40/0x70
[373459.154977] ? __switch_to_asm+0x34/0x70
[373459.154978] ? __switch_to_asm+0x40/0x70
[373459.154980] ? __switch_to_asm+0x34/0x70
[373459.154981] ? __switch_to_asm+0x40/0x70
[373459.154983] ? __switch_to_asm+0x34/0x70
[373459.154984] ? __switch_to_asm+0x40/0x70
[373459.154986] ? __switch_to_asm+0x34/0x70
[373459.154988] ? __switch_to_asm+0x40/0x70
[373459.154989] ? __switch_to_asm+0x34/0x70
[373459.154991] ? __switch_to_asm+0x40/0x70
[373459.154992] ? __switch_to_asm+0x34/0x70
[373459.154994] ? __switch_to_asm+0x40/0x70
[373459.154995] ? __switch_to_asm+0x34/0x70
[373459.154997] ? __switch_to_asm+0x40/0x70
[373459.154998] ? __switch_to_asm+0x34/0x70
[373459.155000] ? __switch_to_asm+0x40/0x70
[373459.155001] ? __switch_to_asm+0x34/0x70
[373459.155003] ? __switch_to_asm+0x40/0x70
[373459.155005] ? __switch_to_asm+0x34/0x70
[373459.155007] ? finish_task_switch+0xf6/0x370
[373459.155010] ? __switch_to+0x2ec/0x5e0
[373459.155013] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373459.155014] ? _raw_write_lock_bh+0xe0/0xe0
[373459.155054] ? cifs_handle_standard+0x270/0x270 [cifs]
[373459.155056] kthread+0x192/0x1e0
[373459.155059] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373459.155061] ret_from_fork+0x35/0x40
[373459.155082] INFO: task kworker/2:1:31242 blocked for more than 241 seconds.
[373459.156633] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373459.158106] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373459.159698] kworker/2:1 D 0 31242 2 0x80004000
[373459.159755] Workqueue: cifsiod refresh_cache_worker [cifs]
[373459.159757] Call Trace:
[373459.159763] ? __schedule+0x540/0xac0
[373459.159765] ? firmware_map_remove+0xe9/0xe9
[373459.159768] ? _raw_read_lock_irq+0x40/0x40
[373459.159770] schedule+0x5e/0x100
[373459.159772] schedule_preempt_disabled+0xa/0x10
[373459.159774] __mutex_lock.isra.4+0x484/0x820
[373459.159777] ? mutex_trylock+0x90/0x90
[373459.159780] ? dynamic_emit_prefix+0x29/0x220
[373459.159782] ? __dynamic_pr_debug+0xf8/0x140
[373459.159783] ? dynamic_emit_prefix+0x220/0x220
[373459.159786] ? mutex_lock+0xce/0xe0
[373459.159787] mutex_lock+0xce/0xe0
[373459.159790] ? __mutex_lock_slowpath+0x10/0x10
[373459.159833] refresh_cache_worker+0x48f/0x14a0 [cifs]
[373459.159836] ? __switch_to_asm+0x40/0x70
[373459.159838] ? __switch_to_asm+0x40/0x70
[373459.159840] ? __switch_to_asm+0x34/0x70
[373459.159842] ? __switch_to_asm+0x40/0x70
[373459.159844] ? __switch_to_asm+0x34/0x70
[373459.159846] ? __switch_to_asm+0x40/0x70
[373459.159888] ? find_root_ses.isra.9+0x320/0x320 [cifs]
[373459.159890] ? __switch_to_asm+0x40/0x70
[373459.159891] ? __switch_to_asm+0x40/0x70
[373459.159893] ? __switch_to_asm+0x34/0x70
[373459.159895] ? __switch_to_asm+0x40/0x70
[373459.159896] ? __switch_to_asm+0x34/0x70
[373459.159898] ? __switch_to_asm+0x40/0x70
[373459.159899] ? __switch_to_asm+0x34/0x70
[373459.159901] ? __switch_to_asm+0x40/0x70
[373459.159902] ? __switch_to_asm+0x40/0x70
[373459.159904] ? __switch_to_asm+0x34/0x70
[373459.159907] ? finish_task_switch+0xf6/0x370
[373459.159909] ? __switch_to+0x2ec/0x5e0
[373459.159911] ? __schedule+0x562/0xac0
[373459.159915] ? read_word_at_a_time+0xe/0x20
[373459.159916] ? strscpy+0xca/0x1d0
[373459.159921] process_one_work+0x373/0x6e0
[373459.159924] worker_thread+0x78/0x5b0
[373459.159927] ? rescuer_thread+0x5e0/0x5e0
[373459.159929] kthread+0x192/0x1e0
[373459.159931] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373459.159933] ret_from_fork+0x35/0x40
[373460.878890] fs/cifs/inode.c: CIFS VFS: in cifs_revalidate_dentry_attr as Xid: 73258 with uid: 11025
[373460.878909] fs/cifs/dir.c: name: \KAES6309
[373460.878913] fs/cifs/inode.c: Update attributes: \KAES6309 inode 0x00000000ae3f689c count 1 dentry: 0x00000000661f7ca9 d_time 4379115881 jiffies 4388265905
[373460.878916] fs/cifs/inode.c: Getting info on \KAES6309
[373460.879081] fs/cifs/transport.c: Sending smb: smb_len=388
[373461.193169] fs/cifs/smb2pdu.c: In echo request
[373461.193186] fs/cifs/smb2pdu.c: Echo request failed: -11
[373461.193192] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373465.288778] fs/cifs/smb2pdu.c: In echo request
[373465.288811] CIFS VFS: Error -32 sending data on socket to server
[373465.290238] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373465.290249] fs/cifs/smb2pdu.c: Echo request failed: -32
[373465.290256] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373522.627361] fs/cifs/smb2pdu.c: In echo request
[373522.627379] fs/cifs/smb2pdu.c: Echo request failed: -11
[373522.627386] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373526.722979] fs/cifs/smb2pdu.c: In echo request
[373526.723013] CIFS VFS: Error -32 sending data on socket to server
[373526.724550] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373526.724562] fs/cifs/smb2pdu.c: Echo request failed: -32
[373526.724568] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373579.965997] INFO: task cifsd:789 blocked for more than 604 seconds.
[373579.967521] Tainted: G E 5.3.1-pd-5.3.y #20191015
[373579.968896] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[373579.970287] cifsd D 0 789 2 0x80004000
[373579.970291] Call Trace:
[373579.970321] ? __schedule+0x540/0xac0
[373579.970324] ? firmware_map_remove+0xe9/0xe9
[373579.970327] ? string_nocheck+0xb0/0xd0
[373579.970334] ? _raw_spin_lock+0x7a/0xd0
[373579.970336] schedule+0x5e/0x100
[373579.970338] schedule_preempt_disabled+0xa/0x10
[373579.970340] __mutex_lock.isra.4+0x484/0x820
[373579.970343] ? mutex_trylock+0x90/0x90
[373579.970357] ? irq_work_claim+0x2e/0x50
[373579.970359] ? irq_work_queue+0x9/0x20
[373579.970362] ? mutex_lock+0xce/0xe0
[373579.970363] mutex_lock+0xce/0xe0
[373579.970365] ? __mutex_lock_slowpath+0x10/0x10
[373579.970418] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[373579.970459] cifs_reconnect+0x16c/0x1360 [cifs]
[373579.970503] ? smb2_calc_size+0x15c/0x250 [cifs]
[373579.970542] ? extract_hostname+0xa0/0xa0 [cifs]
[373579.970544] ? _raw_spin_trylock+0x91/0xe0
[373579.970546] ? _raw_spin_trylock_bh+0x100/0x100
[373579.970548] ? ___ratelimit+0x106/0x190
[373579.970588] cifs_handle_standard+0x252/0x270 [cifs]
[373579.970628] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[373579.970672] ? cifs_handle_standard+0x270/0x270 [cifs]
[373579.970674] ? __switch_to_asm+0x40/0x70
[373579.970675] ? __switch_to_asm+0x34/0x70
[373579.970677] ? __switch_to_asm+0x40/0x70
[373579.970678] ? __switch_to_asm+0x34/0x70
[373579.970680] ? __switch_to_asm+0x40/0x70
[373579.970681] ? __switch_to_asm+0x34/0x70
[373579.970683] ? __switch_to_asm+0x40/0x70
[373579.970684] ? __switch_to_asm+0x34/0x70
[373579.970686] ? __switch_to_asm+0x40/0x70
[373579.970687] ? __switch_to_asm+0x34/0x70
[373579.970689] ? __switch_to_asm+0x40/0x70
[373579.970690] ? __switch_to_asm+0x34/0x70
[373579.970692] ? __switch_to_asm+0x40/0x70
[373579.970693] ? __switch_to_asm+0x34/0x70
[373579.970695] ? __switch_to_asm+0x40/0x70
[373579.970697] ? __switch_to_asm+0x34/0x70
[373579.970698] ? __switch_to_asm+0x40/0x70
[373579.970700] ? __switch_to_asm+0x34/0x70
[373579.970701] ? __switch_to_asm+0x40/0x70
[373579.970703] ? __switch_to_asm+0x34/0x70
[373579.970704] ? __switch_to_asm+0x40/0x70
[373579.970706] ? __switch_to_asm+0x34/0x70
[373579.970707] ? __switch_to_asm+0x40/0x70
[373579.970709] ? __switch_to_asm+0x34/0x70
[373579.970710] ? __switch_to_asm+0x40/0x70
[373579.970712] ? __switch_to_asm+0x34/0x70
[373579.970713] ? __switch_to_asm+0x40/0x70
[373579.970724] ? finish_task_switch+0x91/0x370
[373579.970732] ? __switch_to+0x2ec/0x5e0
[373579.970735] ? _raw_spin_lock_irqsave+0x8d/0xf0
[373579.970736] ? _raw_write_lock_bh+0xe0/0xe0
[373579.970775] ? cifs_handle_standard+0x270/0x270 [cifs]
[373579.970780] kthread+0x192/0x1e0
[373579.970782] ? kthread_create_worker_on_cpu+0xc0/0xc0
[373579.970786] ret_from_fork+0x35/0x40
[373584.061557] fs/cifs/smb2pdu.c: In echo request
[373584.061573] fs/cifs/smb2pdu.c: Echo request failed: -11
[373584.061578] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373588.157191] fs/cifs/smb2pdu.c: In echo request
[373588.157233] CIFS VFS: Error -32 sending data on socket to server
[373588.158699] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[373588.158710] fs/cifs/smb2pdu.c: Echo request failed: -32
[373588.158718] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
[373645.495852] fs/cifs/smb2pdu.c: In echo request
[373645.495870] fs/cifs/smb2pdu.c: Echo request failed: -11
[373645.495876] fs/cifs/connect.c: Unable to send echo request to server: DC02
[373649.591382] fs/cifs/smb2pdu.c: In echo request
The last part repeated over and over again in the log.
Gr, Martijn de Gouw
--
Martijn de Gouw
Designer
Prodrive Technologies
Mobile: +31 63 17 76 161
Phone: +31 40 26 76 200
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Kernel hangs in cifs_reconnect
2019-10-20 8:13 ` Martijn de Gouw
@ 2019-10-29 11:07 ` Martijn de Gouw
2019-10-29 14:49 ` Paulo Alcantara
0 siblings, 1 reply; 9+ messages in thread
From: Martijn de Gouw @ 2019-10-29 11:07 UTC (permalink / raw)
To: Paulo Alcantara, Aurélien Aptel, linux-cifs
Hi,
Anybody any idea on what goes wrong here?
Is any of the recently posted patches related to my issue, because I'm
more that willing to test out patches if needed.
Regards, Martijn
On 20-10-2019 10:13, Martijn de Gouw wrote:
> Hi Paulo,
>
> On 15-10-2019 18:27, Paulo Alcantara wrote:
>> Aurélien Aptel <aaptel@suse.com> writes:
>>
>>> Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
>>>> Our Linux VMs reports call traces about processes being stuck.
>>>> I've attached the full dmesg of one of the call traces below.
>>>>
>>>> The machine is running kernel 5.3.1 SMP. All mounts are mounted via the
>>>> dfs shares on our domain controller and have the following options in fstab:
>>>> nohandlecache,multiuser,sec=krb5,noperm,user=xxxx,file_mode=0600,dir_mode=0700,vers=3.0
>>>
>>> It looks like its DFS related. The DFS cache code takes the reconnect
>>> mutex and crashes with no chance to give back the mutex, making all
>>> other process hang while waiting for it.
>>
>> Yeah, makes sense.
>>
>> Martijn,
>>
>> Could you please provide us with some debug logs with the following:
>>
>> # echo 'module cifs +p' > /sys/kernel/debug/dynamic_debug/control
>> # echo 'file fs/cifs/* +p' > /sys/kernel/debug/dynamic_debug/control
>> # echo 1 > /proc/fs/cifs/cifsFYI
>> # echo 1 > /sys/module/dns_resolver/parameters/debug
>>
>> Besides, if you could also enable KASAN, that would be great.
>
> I recompiled the same kernel with KASAN enabled and have it run again for several days.
> See below for the stackdump, I've added some of the messages before and after the event,
> but if you need more information, please let me know.
>
> [373004.186807] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 73254 with uid: 999
> [373004.186883] fs/cifs/transport.c: Sending smb: smb_len=348
> [373004.187349] fs/cifs/connect.c: RFC1002 header 0x1b8
> [373004.187358] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
> [373004.187359] fs/cifs/smb2misc.c: SMB2 len 208
> [373004.187370] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
> [373004.187371] fs/cifs/smb2misc.c: SMB2 len 104
> [373004.187381] fs/cifs/smb2misc.c: SMB2 len 124
> [373004.187383] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 759
> [373004.187386] fs/cifs/smb2ops.c: add 30 credits total=7026
> [373004.187401] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=757 state=4
> [373004.187404] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=758 state=4
> [373004.187405] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=759 state=4
> [373004.187407] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373004.187414] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373004.187418] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373004.187451] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 73254) rc = 0
> [373031.153835] fs/cifs/smb2pdu.c: In echo request
> [373031.153853] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373031.153869] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373064.180943] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 73255 with uid: 999
> [373064.181026] fs/cifs/transport.c: Sending smb: smb_len=348
> [373064.181572] fs/cifs/connect.c: Received no data or error: -104
> [373092.592054] fs/cifs/smb2pdu.c: In echo request
> [373092.592067] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373092.592073] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373096.683689] INFO: task cifsd:789 blocked for more than 120 seconds.
> [373096.684890] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373096.685859] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373096.686809] cifsd D 0 789 2 0x80004000
> [373096.686824] Call Trace:
> [373096.686838] ? __schedule+0x540/0xac0
> [373096.686841] ? firmware_map_remove+0xe9/0xe9
> [373096.686850] ? string_nocheck+0xb0/0xd0
> [373096.686857] ? _raw_spin_lock+0x7a/0xd0
> [373096.686858] schedule+0x5e/0x100
> [373096.686861] schedule_preempt_disabled+0xa/0x10
> [373096.686863] __mutex_lock.isra.4+0x484/0x820
> [373096.686866] ? mutex_trylock+0x90/0x90
> [373096.686873] ? irq_work_claim+0x2e/0x50
> [373096.686875] ? irq_work_queue+0x9/0x20
> [373096.686878] ? mutex_lock+0xce/0xe0
> [373096.686879] mutex_lock+0xce/0xe0
> [373096.686881] ? __mutex_lock_slowpath+0x10/0x10
> [373096.686934] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373096.686975] cifs_reconnect+0x16c/0x1360 [cifs]
> [373096.687019] ? smb2_calc_size+0x15c/0x250 [cifs]
> [373096.687058] ? extract_hostname+0xa0/0xa0 [cifs]
> [373096.687060] ? _raw_spin_trylock+0x91/0xe0
> [373096.687062] ? _raw_spin_trylock_bh+0x100/0x100
> [373096.687064] ? ___ratelimit+0x106/0x190
> [373096.687104] cifs_handle_standard+0x252/0x270 [cifs]
> [373096.687144] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
> [373096.687184] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373096.687186] ? __switch_to_asm+0x40/0x70
> [373096.687188] ? __switch_to_asm+0x34/0x70
> [373096.687189] ? __switch_to_asm+0x40/0x70
> [373096.687191] ? __switch_to_asm+0x34/0x70
> [373096.687192] ? __switch_to_asm+0x40/0x70
> [373096.687194] ? __switch_to_asm+0x34/0x70
> [373096.687196] ? __switch_to_asm+0x40/0x70
> [373096.687197] ? __switch_to_asm+0x34/0x70
> [373096.687199] ? __switch_to_asm+0x40/0x70
> [373096.687200] ? __switch_to_asm+0x34/0x70
> [373096.687202] ? __switch_to_asm+0x40/0x70
> [373096.687203] ? __switch_to_asm+0x34/0x70
> [373096.687205] ? __switch_to_asm+0x40/0x70
> [373096.687206] ? __switch_to_asm+0x34/0x70
> [373096.687208] ? __switch_to_asm+0x40/0x70
> [373096.687209] ? __switch_to_asm+0x34/0x70
> [373096.687211] ? __switch_to_asm+0x40/0x70
> [373096.687212] ? __switch_to_asm+0x34/0x70
> [373096.687214] ? __switch_to_asm+0x40/0x70
> [373096.687215] ? __switch_to_asm+0x34/0x70
> [373096.687217] ? __switch_to_asm+0x40/0x70
> [373096.687218] ? __switch_to_asm+0x34/0x70
> [373096.687220] ? __switch_to_asm+0x40/0x70
> [373096.687221] ? __switch_to_asm+0x34/0x70
> [373096.687223] ? __switch_to_asm+0x40/0x70
> [373096.687224] ? __switch_to_asm+0x34/0x70
> [373096.687226] ? __switch_to_asm+0x40/0x70
> [373096.687232] ? finish_task_switch+0x91/0x370
> [373096.687234] ? __switch_to+0x2ec/0x5e0
> [373096.687237] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373096.687239] ? _raw_write_lock_bh+0xe0/0xe0
> [373096.687277] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373096.687280] kthread+0x192/0x1e0
> [373096.687282] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373096.687284] ret_from_fork+0x35/0x40
> [373096.687326] fs/cifs/smb2pdu.c: In echo request
> [373096.687354] CIFS VFS: Error -32 sending data on socket to server
> [373096.688357] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373096.688369] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373096.688375] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373149.926651] fs/cifs/connect.c: Existing tcp session with server found
> [373149.926658] fs/cifs/dfs_cache.c: CIFS VFS: in do_refresh_tcon as Xid: 73256 with uid: 0
> [373154.022274] fs/cifs/smb2pdu.c: In echo request
> [373154.022290] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373154.022298] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373158.121876] fs/cifs/smb2pdu.c: In echo request
> [373158.121904] CIFS VFS: Error -32 sending data on socket to server
> [373158.123065] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373158.123079] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373158.123089] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373161.358480] fs/cifs/inode.c: CIFS VFS: in cifs_revalidate_dentry_attr as Xid: 73257 with uid: 11025
> [373161.358496] fs/cifs/dir.c: name: \KAES6309
> [373161.358500] fs/cifs/inode.c: Update attributes: \KAES6309 inode 0x00000000ae3f689c count 1 dentry: 0x00000000661f7ca9 d_time 4379115881 jiffies 4388191018
> [373161.358503] fs/cifs/inode.c: Getting info on \KAES6309
> [373161.358682] fs/cifs/transport.c: Sending smb: smb_len=388
> [373215.456445] fs/cifs/smb2pdu.c: In echo request
> [373215.456461] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373215.456467] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373217.504315] INFO: task cifsd:789 blocked for more than 241 seconds.
> [373217.505647] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373217.506642] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373217.507653] cifsd D 0 789 2 0x80004000
> [373217.507657] Call Trace:
> [373217.507664] ? __schedule+0x540/0xac0
> [373217.507667] ? firmware_map_remove+0xe9/0xe9
> [373217.507670] ? string_nocheck+0xb0/0xd0
> [373217.507676] ? _raw_spin_lock+0x7a/0xd0
> [373217.507677] schedule+0x5e/0x100
> [373217.507680] schedule_preempt_disabled+0xa/0x10
> [373217.507682] __mutex_lock.isra.4+0x484/0x820
> [373217.507685] ? mutex_trylock+0x90/0x90
> [373217.507688] ? irq_work_claim+0x2e/0x50
> [373217.507690] ? irq_work_queue+0x9/0x20
> [373217.507693] ? mutex_lock+0xce/0xe0
> [373217.507694] mutex_lock+0xce/0xe0
> [373217.507696] ? __mutex_lock_slowpath+0x10/0x10
> [373217.507750] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373217.507791] cifs_reconnect+0x16c/0x1360 [cifs]
> [373217.507834] ? smb2_calc_size+0x15c/0x250 [cifs]
> [373217.507873] ? extract_hostname+0xa0/0xa0 [cifs]
> [373217.507875] ? _raw_spin_trylock+0x91/0xe0
> [373217.507877] ? _raw_spin_trylock_bh+0x100/0x100
> [373217.507879] ? ___ratelimit+0x106/0x190
> [373217.507919] cifs_handle_standard+0x252/0x270 [cifs]
> [373217.507959] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
> [373217.507999] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373217.508001] ? __switch_to_asm+0x40/0x70
> [373217.508003] ? __switch_to_asm+0x34/0x70
> [373217.508004] ? __switch_to_asm+0x40/0x70
> [373217.508006] ? __switch_to_asm+0x34/0x70
> [373217.508008] ? __switch_to_asm+0x40/0x70
> [373217.508009] ? __switch_to_asm+0x34/0x70
> [373217.508011] ? __switch_to_asm+0x40/0x70
> [373217.508012] ? __switch_to_asm+0x34/0x70
> [373217.508014] ? __switch_to_asm+0x40/0x70
> [373217.508015] ? __switch_to_asm+0x34/0x70
> [373217.508017] ? __switch_to_asm+0x40/0x70
> [373217.508018] ? __switch_to_asm+0x34/0x70
> [373217.508020] ? __switch_to_asm+0x40/0x70
> [373217.508021] ? __switch_to_asm+0x34/0x70
> [373217.508023] ? __switch_to_asm+0x40/0x70
> [373217.508024] ? __switch_to_asm+0x34/0x70
> [373217.508026] ? __switch_to_asm+0x40/0x70
> [373217.508028] ? __switch_to_asm+0x34/0x70
> [373217.508029] ? __switch_to_asm+0x40/0x70
> [373217.508031] ? __switch_to_asm+0x34/0x70
> [373217.508032] ? __switch_to_asm+0x40/0x70
> [373217.508034] ? __switch_to_asm+0x34/0x70
> [373217.508035] ? __switch_to_asm+0x40/0x70
> [373217.508037] ? __switch_to_asm+0x34/0x70
> [373217.508038] ? __switch_to_asm+0x40/0x70
> [373217.508040] ? __switch_to_asm+0x34/0x70
> [373217.508041] ? __switch_to_asm+0x40/0x70
> [373217.508044] ? finish_task_switch+0x91/0x370
> [373217.508046] ? __switch_to+0x2ec/0x5e0
> [373217.508049] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373217.508050] ? _raw_write_lock_bh+0xe0/0xe0
> [373217.508090] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373217.508092] kthread+0x192/0x1e0
> [373217.508094] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373217.508096] ret_from_fork+0x35/0x40
> [373217.508118] INFO: task cifsd:16935 blocked for more than 120 seconds.
> [373217.509148] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373217.510152] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373217.511174] cifsd D 0 16935 2 0x80004000
> [373217.511177] Call Trace:
> [373217.511181] ? __schedule+0x540/0xac0
> [373217.511184] ? firmware_map_remove+0xe9/0xe9
> [373217.511187] ? vsnprintf+0x870/0x870
> [373217.511189] ? _raw_spin_lock+0x7a/0xd0
> [373217.511191] schedule+0x5e/0x100
> [373217.511193] schedule_preempt_disabled+0xa/0x10
> [373217.511194] __mutex_lock.isra.4+0x484/0x820
> [373217.511197] ? mutex_trylock+0x90/0x90
> [373217.511226] ? up+0x32/0x70
> [373217.511229] ? irq_work_claim+0x2e/0x50
> [373217.511231] ? irq_work_queue+0x9/0x20
> [373217.511234] ? vprintk_emit+0x11d/0x2e0
> [373217.511236] ? mutex_lock+0xce/0xe0
> [373217.511238] mutex_lock+0xce/0xe0
> [373217.511240] ? __mutex_lock_slowpath+0x10/0x10
> [373217.511290] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373217.511330] cifs_reconnect+0x16c/0x1360 [cifs]
> [373217.511371] ? extract_hostname+0xa0/0xa0 [cifs]
> [373217.511374] ? _raw_spin_trylock+0x91/0xe0
> [373217.511376] ? _raw_spin_trylock_bh+0x100/0x100
> [373217.511379] ? aa_sk_perm+0xe4/0x1f0
> [373217.511382] ? inet_release+0xc0/0xc0
> [373217.511384] ? ___ratelimit+0x106/0x190
> [373217.511423] cifs_readv_from_socket+0x319/0x390 [cifs]
> [373217.511463] cifs_read_from_socket+0x9d/0xe0 [cifs]
> [373217.511503] ? cifs_readv_from_socket+0x390/0x390 [cifs]
> [373217.511507] ? refcount_sub_and_test_checked+0xae/0x140
> [373217.511548] ? cifs_small_buf_get+0x37/0x50 [cifs]
> [373217.511588] ? allocate_buffers+0x10a/0x170 [cifs]
> [373217.511627] cifs_demultiplex_thread+0x241/0x13e0 [cifs]
> [373217.511667] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373217.511671] ? sched_clock+0x5/0x10
> [373217.511673] ? __switch_to_asm+0x40/0x70
> [373217.511675] ? __switch_to_asm+0x34/0x70
> [373217.511677] ? __switch_to_asm+0x40/0x70
> [373217.511678] ? __switch_to_asm+0x34/0x70
> [373217.511681] ? __switch_to_asm+0x40/0x70
> [373217.511683] ? __switch_to_asm+0x34/0x70
> [373217.511684] ? __switch_to_asm+0x40/0x70
> [373217.511686] ? __switch_to_asm+0x34/0x70
> [373217.511687] ? __switch_to_asm+0x40/0x70
> [373217.511689] ? __switch_to_asm+0x34/0x70
> [373217.511690] ? __switch_to_asm+0x40/0x70
> [373217.511692] ? __switch_to_asm+0x34/0x70
> [373217.511693] ? __switch_to_asm+0x40/0x70
> [373217.511695] ? __switch_to_asm+0x34/0x70
> [373217.511696] ? __switch_to_asm+0x40/0x70
> [373217.511698] ? __switch_to_asm+0x34/0x70
> [373217.511700] ? __switch_to_asm+0x40/0x70
> [373217.511701] ? __switch_to_asm+0x34/0x70
> [373217.511703] ? __switch_to_asm+0x40/0x70
> [373217.511704] ? __switch_to_asm+0x34/0x70
> [373217.511706] ? __switch_to_asm+0x40/0x70
> [373217.511707] ? __switch_to_asm+0x34/0x70
> [373217.511709] ? __switch_to_asm+0x40/0x70
> [373217.511710] ? __switch_to_asm+0x34/0x70
> [373217.511712] ? __switch_to_asm+0x40/0x70
> [373217.511713] ? __switch_to_asm+0x34/0x70
> [373217.511716] ? finish_task_switch+0xf6/0x370
> [373217.511717] ? __switch_to+0x2ec/0x5e0
> [373217.511720] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373217.511722] ? _raw_write_lock_bh+0xe0/0xe0
> [373217.511761] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373217.511763] kthread+0x192/0x1e0
> [373217.511766] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373217.511768] ret_from_fork+0x35/0x40
> [373219.552028] fs/cifs/smb2pdu.c: In echo request
> [373219.552061] CIFS VFS: Error -32 sending data on socket to server
> [373219.553244] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373219.553254] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373219.553261] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373276.894645] fs/cifs/smb2pdu.c: In echo request
> [373276.894669] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373276.894675] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373280.986191] fs/cifs/smb2pdu.c: In echo request
> [373280.986219] CIFS VFS: Error -32 sending data on socket to server
> [373280.987409] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373280.987420] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373280.987426] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373338.324793] fs/cifs/smb2pdu.c: In echo request
> [373338.324832] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373338.324838] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373338.324918] INFO: task cifsd:789 blocked for more than 362 seconds.
> [373338.326236] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373338.327346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373338.328455] cifsd D 0 789 2 0x80004000
> [373338.328458] Call Trace:
> [373338.328465] ? __schedule+0x540/0xac0
> [373338.328468] ? firmware_map_remove+0xe9/0xe9
> [373338.328471] ? string_nocheck+0xb0/0xd0
> [373338.328475] ? _raw_spin_lock+0x7a/0xd0
> [373338.328477] schedule+0x5e/0x100
> [373338.328479] schedule_preempt_disabled+0xa/0x10
> [373338.328481] __mutex_lock.isra.4+0x484/0x820
> [373338.328484] ? mutex_trylock+0x90/0x90
> [373338.328488] ? irq_work_claim+0x2e/0x50
> [373338.328490] ? irq_work_queue+0x9/0x20
> [373338.328492] ? mutex_lock+0xce/0xe0
> [373338.328494] mutex_lock+0xce/0xe0
> [373338.328496] ? __mutex_lock_slowpath+0x10/0x10
> [373338.328547] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373338.328588] cifs_reconnect+0x16c/0x1360 [cifs]
> [373338.328631] ? smb2_calc_size+0x15c/0x250 [cifs]
> [373338.328670] ? extract_hostname+0xa0/0xa0 [cifs]
> [373338.328672] ? _raw_spin_trylock+0x91/0xe0
> [373338.328674] ? _raw_spin_trylock_bh+0x100/0x100
> [373338.328676] ? ___ratelimit+0x106/0x190
> [373338.328716] cifs_handle_standard+0x252/0x270 [cifs]
> [373338.328758] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
> [373338.328813] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373338.328818] ? __switch_to_asm+0x40/0x70
> [373338.328824] ? __switch_to_asm+0x34/0x70
> [373338.328828] ? __switch_to_asm+0x40/0x70
> [373338.328834] ? __switch_to_asm+0x34/0x70
> [373338.328840] ? __switch_to_asm+0x40/0x70
> [373338.328843] ? __switch_to_asm+0x34/0x70
> [373338.328849] ? __switch_to_asm+0x40/0x70
> [373338.328853] ? __switch_to_asm+0x34/0x70
> [373338.328859] ? __switch_to_asm+0x40/0x70
> [373338.328871] ? __switch_to_asm+0x34/0x70
> [373338.328873] ? __switch_to_asm+0x40/0x70
> [373338.328874] ? __switch_to_asm+0x34/0x70
> [373338.328876] ? __switch_to_asm+0x40/0x70
> [373338.328877] ? __switch_to_asm+0x34/0x70
> [373338.328879] ? __switch_to_asm+0x40/0x70
> [373338.328880] ? __switch_to_asm+0x34/0x70
> [373338.328882] ? __switch_to_asm+0x40/0x70
> [373338.328883] ? __switch_to_asm+0x34/0x70
> [373338.328885] ? __switch_to_asm+0x40/0x70
> [373338.328886] ? __switch_to_asm+0x34/0x70
> [373338.328888] ? __switch_to_asm+0x40/0x70
> [373338.328889] ? __switch_to_asm+0x34/0x70
> [373338.328891] ? __switch_to_asm+0x40/0x70
> [373338.328893] ? __switch_to_asm+0x34/0x70
> [373338.328894] ? __switch_to_asm+0x40/0x70
> [373338.328896] ? __switch_to_asm+0x34/0x70
> [373338.328897] ? __switch_to_asm+0x40/0x70
> [373338.328900] ? finish_task_switch+0x91/0x370
> [373338.328903] ? __switch_to+0x2ec/0x5e0
> [373338.328905] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373338.328907] ? _raw_write_lock_bh+0xe0/0xe0
> [373338.328947] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373338.328949] kthread+0x192/0x1e0
> [373338.328952] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373338.328954] ret_from_fork+0x35/0x40
> [373338.328975] INFO: task cifsd:16935 blocked for more than 241 seconds.
> [373338.330061] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373338.331156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373338.332283] cifsd D 0 16935 2 0x80004000
> [373338.332286] Call Trace:
> [373338.332296] ? __schedule+0x540/0xac0
> [373338.332298] ? firmware_map_remove+0xe9/0xe9
> [373338.332301] ? vsnprintf+0x870/0x870
> [373338.332304] ? _raw_spin_lock+0x7a/0xd0
> [373338.332306] schedule+0x5e/0x100
> [373338.332308] schedule_preempt_disabled+0xa/0x10
> [373338.332310] __mutex_lock.isra.4+0x484/0x820
> [373338.332312] ? mutex_trylock+0x90/0x90
> [373338.332315] ? up+0x32/0x70
> [373338.332319] ? irq_work_claim+0x2e/0x50
> [373338.332320] ? irq_work_queue+0x9/0x20
> [373338.332323] ? vprintk_emit+0x11d/0x2e0
> [373338.332325] ? mutex_lock+0xce/0xe0
> [373338.332326] mutex_lock+0xce/0xe0
> [373338.332328] ? __mutex_lock_slowpath+0x10/0x10
> [373338.332381] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373338.332421] cifs_reconnect+0x16c/0x1360 [cifs]
> [373338.332461] ? extract_hostname+0xa0/0xa0 [cifs]
> [373338.332463] ? _raw_spin_trylock+0x91/0xe0
> [373338.332465] ? _raw_spin_trylock_bh+0x100/0x100
> [373338.332467] ? aa_sk_perm+0xe4/0x1f0
> [373338.332470] ? inet_release+0xc0/0xc0
> [373338.332472] ? ___ratelimit+0x106/0x190
> [373338.332511] cifs_readv_from_socket+0x319/0x390 [cifs]
> [373338.332551] cifs_read_from_socket+0x9d/0xe0 [cifs]
> [373338.332590] ? cifs_readv_from_socket+0x390/0x390 [cifs]
> [373338.332594] ? refcount_sub_and_test_checked+0xae/0x140
> [373338.332635] ? cifs_small_buf_get+0x37/0x50 [cifs]
> [373338.332674] ? allocate_buffers+0x10a/0x170 [cifs]
> [373338.332713] cifs_demultiplex_thread+0x241/0x13e0 [cifs]
> [373338.332755] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373338.332774] ? sched_clock+0x5/0x10
> [373338.332776] ? __switch_to_asm+0x40/0x70
> [373338.332778] ? __switch_to_asm+0x34/0x70
> [373338.332779] ? __switch_to_asm+0x40/0x70
> [373338.332781] ? __switch_to_asm+0x34/0x70
> [373338.332782] ? __switch_to_asm+0x40/0x70
> [373338.332784] ? __switch_to_asm+0x34/0x70
> [373338.332786] ? __switch_to_asm+0x40/0x70
> [373338.332787] ? __switch_to_asm+0x34/0x70
> [373338.332791] ? __switch_to_asm+0x40/0x70
> [373338.332798] ? __switch_to_asm+0x34/0x70
> [373338.332802] ? __switch_to_asm+0x40/0x70
> [373338.332808] ? __switch_to_asm+0x34/0x70
> [373338.332810] ? __switch_to_asm+0x40/0x70
> [373338.332813] ? __switch_to_asm+0x34/0x70
> [373338.332822] ? __switch_to_asm+0x40/0x70
> [373338.332828] ? __switch_to_asm+0x34/0x70
> [373338.332830] ? __switch_to_asm+0x40/0x70
> [373338.332834] ? __switch_to_asm+0x34/0x70
> [373338.332840] ? __switch_to_asm+0x40/0x70
> [373338.332856] ? __switch_to_asm+0x34/0x70
> [373338.332858] ? __switch_to_asm+0x40/0x70
> [373338.332859] ? __switch_to_asm+0x34/0x70
> [373338.332861] ? __switch_to_asm+0x40/0x70
> [373338.332862] ? __switch_to_asm+0x34/0x70
> [373338.332864] ? __switch_to_asm+0x40/0x70
> [373338.332865] ? __switch_to_asm+0x34/0x70
> [373338.332868] ? finish_task_switch+0xf6/0x370
> [373338.332870] ? __switch_to+0x2ec/0x5e0
> [373338.332872] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373338.332874] ? _raw_write_lock_bh+0xe0/0xe0
> [373338.332913] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373338.332923] kthread+0x192/0x1e0
> [373338.332927] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373338.332930] ret_from_fork+0x35/0x40
> [373338.332958] INFO: task kworker/2:1:31242 blocked for more than 120 seconds.
> [373338.334112] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373338.335285] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373338.336617] kworker/2:1 D 0 31242 2 0x80004000
> [373338.336700] Workqueue: cifsiod refresh_cache_worker [cifs]
> [373338.336702] Call Trace:
> [373338.336709] ? __schedule+0x540/0xac0
> [373338.336714] ? firmware_map_remove+0xe9/0xe9
> [373338.336717] ? _raw_read_lock_irq+0x40/0x40
> [373338.336721] schedule+0x5e/0x100
> [373338.336725] schedule_preempt_disabled+0xa/0x10
> [373338.336728] __mutex_lock.isra.4+0x484/0x820
> [373338.336733] ? mutex_trylock+0x90/0x90
> [373338.336737] ? dynamic_emit_prefix+0x29/0x220
> [373338.336740] ? __dynamic_pr_debug+0xf8/0x140
> [373338.336744] ? dynamic_emit_prefix+0x220/0x220
> [373338.336770] ? mutex_lock+0xce/0xe0
> [373338.336773] mutex_lock+0xce/0xe0
> [373338.336777] ? __mutex_lock_slowpath+0x10/0x10
> [373338.336856] refresh_cache_worker+0x48f/0x14a0 [cifs]
> [373338.336864] ? __switch_to_asm+0x40/0x70
> [373338.336875] ? __switch_to_asm+0x40/0x70
> [373338.336883] ? __switch_to_asm+0x34/0x70
> [373338.336885] ? __switch_to_asm+0x40/0x70
> [373338.336888] ? __switch_to_asm+0x34/0x70
> [373338.336891] ? __switch_to_asm+0x40/0x70
> [373338.336964] ? find_root_ses.isra.9+0x320/0x320 [cifs]
> [373338.336968] ? __switch_to_asm+0x40/0x70
> [373338.336976] ? __switch_to_asm+0x40/0x70
> [373338.336979] ? __switch_to_asm+0x34/0x70
> [373338.336981] ? __switch_to_asm+0x40/0x70
> [373338.336984] ? __switch_to_asm+0x34/0x70
> [373338.336991] ? __switch_to_asm+0x40/0x70
> [373338.336994] ? __switch_to_asm+0x34/0x70
> [373338.336998] ? __switch_to_asm+0x40/0x70
> [373338.337004] ? __switch_to_asm+0x40/0x70
> [373338.337007] ? __switch_to_asm+0x34/0x70
> [373338.337010] ? finish_task_switch+0xf6/0x370
> [373338.337016] ? __switch_to+0x2ec/0x5e0
> [373338.337019] ? __schedule+0x562/0xac0
> [373338.337023] ? read_word_at_a_time+0xe/0x20
> [373338.337028] ? strscpy+0xca/0x1d0
> [373338.337034] process_one_work+0x373/0x6e0
> [373338.337040] worker_thread+0x78/0x5b0
> [373338.337047] ? rescuer_thread+0x5e0/0x5e0
> [373338.337050] kthread+0x192/0x1e0
> [373338.337054] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373338.337058] ret_from_fork+0x35/0x40
> [373342.420422] fs/cifs/smb2pdu.c: In echo request
> [373342.420455] CIFS VFS: Error -32 sending data on socket to server
> [373342.421797] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373342.421809] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373342.421816] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373399.758993] fs/cifs/smb2pdu.c: In echo request
> [373399.759010] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373399.759017] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373403.854642] fs/cifs/smb2pdu.c: In echo request
> [373403.854666] CIFS VFS: Error -32 sending data on socket to server
> [373403.856180] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373403.856195] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373403.856202] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373459.145452] INFO: task cifsd:789 blocked for more than 483 seconds.
> [373459.147021] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373459.148332] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373459.149648] cifsd D 0 789 2 0x80004000
> [373459.149657] Call Trace:
> [373459.149682] ? __schedule+0x540/0xac0
> [373459.149687] ? firmware_map_remove+0xe9/0xe9
> [373459.149690] ? string_nocheck+0xb0/0xd0
> [373459.149694] ? _raw_spin_lock+0x7a/0xd0
> [373459.149698] schedule+0x5e/0x100
> [373459.149700] schedule_preempt_disabled+0xa/0x10
> [373459.149703] __mutex_lock.isra.4+0x484/0x820
> [373459.149706] ? mutex_trylock+0x90/0x90
> [373459.149714] ? irq_work_claim+0x2e/0x50
> [373459.149716] ? irq_work_queue+0x9/0x20
> [373459.149719] ? mutex_lock+0xce/0xe0
> [373459.149720] mutex_lock+0xce/0xe0
> [373459.149722] ? __mutex_lock_slowpath+0x10/0x10
> [373459.149775] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373459.149818] cifs_reconnect+0x16c/0x1360 [cifs]
> [373459.149870] ? smb2_calc_size+0x15c/0x250 [cifs]
> [373459.149909] ? extract_hostname+0xa0/0xa0 [cifs]
> [373459.149911] ? _raw_spin_trylock+0x91/0xe0
> [373459.149913] ? _raw_spin_trylock_bh+0x100/0x100
> [373459.149915] ? ___ratelimit+0x106/0x190
> [373459.149960] cifs_handle_standard+0x252/0x270 [cifs]
> [373459.150001] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
> [373459.150041] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373459.150043] ? __switch_to_asm+0x40/0x70
> [373459.150045] ? __switch_to_asm+0x34/0x70
> [373459.150047] ? __switch_to_asm+0x40/0x70
> [373459.150050] ? __switch_to_asm+0x34/0x70
> [373459.150051] ? __switch_to_asm+0x40/0x70
> [373459.150053] ? __switch_to_asm+0x34/0x70
> [373459.150055] ? __switch_to_asm+0x40/0x70
> [373459.150058] ? __switch_to_asm+0x34/0x70
> [373459.150061] ? __switch_to_asm+0x40/0x70
> [373459.150063] ? __switch_to_asm+0x34/0x70
> [373459.150066] ? __switch_to_asm+0x40/0x70
> [373459.150068] ? __switch_to_asm+0x34/0x70
> [373459.150070] ? __switch_to_asm+0x40/0x70
> [373459.150072] ? __switch_to_asm+0x34/0x70
> [373459.150074] ? __switch_to_asm+0x40/0x70
> [373459.150077] ? __switch_to_asm+0x34/0x70
> [373459.150078] ? __switch_to_asm+0x40/0x70
> [373459.150080] ? __switch_to_asm+0x34/0x70
> [373459.150081] ? __switch_to_asm+0x40/0x70
> [373459.150083] ? __switch_to_asm+0x34/0x70
> [373459.150084] ? __switch_to_asm+0x40/0x70
> [373459.150086] ? __switch_to_asm+0x34/0x70
> [373459.150087] ? __switch_to_asm+0x40/0x70
> [373459.150089] ? __switch_to_asm+0x34/0x70
> [373459.150090] ? __switch_to_asm+0x40/0x70
> [373459.150092] ? __switch_to_asm+0x34/0x70
> [373459.150093] ? __switch_to_asm+0x40/0x70
> [373459.150102] ? finish_task_switch+0x91/0x370
> [373459.150104] ? __switch_to+0x2ec/0x5e0
> [373459.150106] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373459.150108] ? _raw_write_lock_bh+0xe0/0xe0
> [373459.150149] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373459.150152] kthread+0x192/0x1e0
> [373459.150154] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373459.150156] ret_from_fork+0x35/0x40
> [373459.150175] INFO: task cifsd:16935 blocked for more than 362 seconds.
> [373459.151549] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373459.152979] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373459.154378] cifsd D 0 16935 2 0x80004000
> [373459.154381] Call Trace:
> [373459.154387] ? __schedule+0x540/0xac0
> [373459.154483] ? firmware_map_remove+0xe9/0xe9
> [373459.154487] ? vsnprintf+0x870/0x870
> [373459.154491] ? _raw_spin_lock+0x7a/0xd0
> [373459.154494] schedule+0x5e/0x100
> [373459.154497] schedule_preempt_disabled+0xa/0x10
> [373459.154499] __mutex_lock.isra.4+0x484/0x820
> [373459.154502] ? mutex_trylock+0x90/0x90
> [373459.154511] ? up+0x32/0x70
> [373459.154514] ? irq_work_claim+0x2e/0x50
> [373459.154515] ? irq_work_queue+0x9/0x20
> [373459.154518] ? vprintk_emit+0x11d/0x2e0
> [373459.154520] ? mutex_lock+0xce/0xe0
> [373459.154521] mutex_lock+0xce/0xe0
> [373459.154524] ? __mutex_lock_slowpath+0x10/0x10
> [373459.154576] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373459.154616] cifs_reconnect+0x16c/0x1360 [cifs]
> [373459.154658] ? extract_hostname+0xa0/0xa0 [cifs]
> [373459.154661] ? _raw_spin_trylock+0x91/0xe0
> [373459.154663] ? _raw_spin_trylock_bh+0x100/0x100
> [373459.154671] ? aa_sk_perm+0xe4/0x1f0
> [373459.154677] ? inet_release+0xc0/0xc0
> [373459.154679] ? ___ratelimit+0x106/0x190
> [373459.154717] cifs_readv_from_socket+0x319/0x390 [cifs]
> [373459.154757] cifs_read_from_socket+0x9d/0xe0 [cifs]
> [373459.154796] ? cifs_readv_from_socket+0x390/0x390 [cifs]
> [373459.154799] ? refcount_sub_and_test_checked+0xae/0x140
> [373459.154840] ? cifs_small_buf_get+0x37/0x50 [cifs]
> [373459.154881] ? allocate_buffers+0x10a/0x170 [cifs]
> [373459.154920] cifs_demultiplex_thread+0x241/0x13e0 [cifs]
> [373459.154960] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373459.154963] ? sched_clock+0x5/0x10
> [373459.154965] ? __switch_to_asm+0x40/0x70
> [373459.154966] ? __switch_to_asm+0x34/0x70
> [373459.154969] ? __switch_to_asm+0x40/0x70
> [373459.154971] ? __switch_to_asm+0x34/0x70
> [373459.154972] ? __switch_to_asm+0x40/0x70
> [373459.154974] ? __switch_to_asm+0x34/0x70
> [373459.154975] ? __switch_to_asm+0x40/0x70
> [373459.154977] ? __switch_to_asm+0x34/0x70
> [373459.154978] ? __switch_to_asm+0x40/0x70
> [373459.154980] ? __switch_to_asm+0x34/0x70
> [373459.154981] ? __switch_to_asm+0x40/0x70
> [373459.154983] ? __switch_to_asm+0x34/0x70
> [373459.154984] ? __switch_to_asm+0x40/0x70
> [373459.154986] ? __switch_to_asm+0x34/0x70
> [373459.154988] ? __switch_to_asm+0x40/0x70
> [373459.154989] ? __switch_to_asm+0x34/0x70
> [373459.154991] ? __switch_to_asm+0x40/0x70
> [373459.154992] ? __switch_to_asm+0x34/0x70
> [373459.154994] ? __switch_to_asm+0x40/0x70
> [373459.154995] ? __switch_to_asm+0x34/0x70
> [373459.154997] ? __switch_to_asm+0x40/0x70
> [373459.154998] ? __switch_to_asm+0x34/0x70
> [373459.155000] ? __switch_to_asm+0x40/0x70
> [373459.155001] ? __switch_to_asm+0x34/0x70
> [373459.155003] ? __switch_to_asm+0x40/0x70
> [373459.155005] ? __switch_to_asm+0x34/0x70
> [373459.155007] ? finish_task_switch+0xf6/0x370
> [373459.155010] ? __switch_to+0x2ec/0x5e0
> [373459.155013] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373459.155014] ? _raw_write_lock_bh+0xe0/0xe0
> [373459.155054] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373459.155056] kthread+0x192/0x1e0
> [373459.155059] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373459.155061] ret_from_fork+0x35/0x40
> [373459.155082] INFO: task kworker/2:1:31242 blocked for more than 241 seconds.
> [373459.156633] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373459.158106] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373459.159698] kworker/2:1 D 0 31242 2 0x80004000
> [373459.159755] Workqueue: cifsiod refresh_cache_worker [cifs]
> [373459.159757] Call Trace:
> [373459.159763] ? __schedule+0x540/0xac0
> [373459.159765] ? firmware_map_remove+0xe9/0xe9
> [373459.159768] ? _raw_read_lock_irq+0x40/0x40
> [373459.159770] schedule+0x5e/0x100
> [373459.159772] schedule_preempt_disabled+0xa/0x10
> [373459.159774] __mutex_lock.isra.4+0x484/0x820
> [373459.159777] ? mutex_trylock+0x90/0x90
> [373459.159780] ? dynamic_emit_prefix+0x29/0x220
> [373459.159782] ? __dynamic_pr_debug+0xf8/0x140
> [373459.159783] ? dynamic_emit_prefix+0x220/0x220
> [373459.159786] ? mutex_lock+0xce/0xe0
> [373459.159787] mutex_lock+0xce/0xe0
> [373459.159790] ? __mutex_lock_slowpath+0x10/0x10
> [373459.159833] refresh_cache_worker+0x48f/0x14a0 [cifs]
> [373459.159836] ? __switch_to_asm+0x40/0x70
> [373459.159838] ? __switch_to_asm+0x40/0x70
> [373459.159840] ? __switch_to_asm+0x34/0x70
> [373459.159842] ? __switch_to_asm+0x40/0x70
> [373459.159844] ? __switch_to_asm+0x34/0x70
> [373459.159846] ? __switch_to_asm+0x40/0x70
> [373459.159888] ? find_root_ses.isra.9+0x320/0x320 [cifs]
> [373459.159890] ? __switch_to_asm+0x40/0x70
> [373459.159891] ? __switch_to_asm+0x40/0x70
> [373459.159893] ? __switch_to_asm+0x34/0x70
> [373459.159895] ? __switch_to_asm+0x40/0x70
> [373459.159896] ? __switch_to_asm+0x34/0x70
> [373459.159898] ? __switch_to_asm+0x40/0x70
> [373459.159899] ? __switch_to_asm+0x34/0x70
> [373459.159901] ? __switch_to_asm+0x40/0x70
> [373459.159902] ? __switch_to_asm+0x40/0x70
> [373459.159904] ? __switch_to_asm+0x34/0x70
> [373459.159907] ? finish_task_switch+0xf6/0x370
> [373459.159909] ? __switch_to+0x2ec/0x5e0
> [373459.159911] ? __schedule+0x562/0xac0
> [373459.159915] ? read_word_at_a_time+0xe/0x20
> [373459.159916] ? strscpy+0xca/0x1d0
> [373459.159921] process_one_work+0x373/0x6e0
> [373459.159924] worker_thread+0x78/0x5b0
> [373459.159927] ? rescuer_thread+0x5e0/0x5e0
> [373459.159929] kthread+0x192/0x1e0
> [373459.159931] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373459.159933] ret_from_fork+0x35/0x40
> [373460.878890] fs/cifs/inode.c: CIFS VFS: in cifs_revalidate_dentry_attr as Xid: 73258 with uid: 11025
> [373460.878909] fs/cifs/dir.c: name: \KAES6309
> [373460.878913] fs/cifs/inode.c: Update attributes: \KAES6309 inode 0x00000000ae3f689c count 1 dentry: 0x00000000661f7ca9 d_time 4379115881 jiffies 4388265905
> [373460.878916] fs/cifs/inode.c: Getting info on \KAES6309
> [373460.879081] fs/cifs/transport.c: Sending smb: smb_len=388
> [373461.193169] fs/cifs/smb2pdu.c: In echo request
> [373461.193186] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373461.193192] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373465.288778] fs/cifs/smb2pdu.c: In echo request
> [373465.288811] CIFS VFS: Error -32 sending data on socket to server
> [373465.290238] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373465.290249] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373465.290256] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373522.627361] fs/cifs/smb2pdu.c: In echo request
> [373522.627379] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373522.627386] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373526.722979] fs/cifs/smb2pdu.c: In echo request
> [373526.723013] CIFS VFS: Error -32 sending data on socket to server
> [373526.724550] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373526.724562] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373526.724568] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373579.965997] INFO: task cifsd:789 blocked for more than 604 seconds.
> [373579.967521] Tainted: G E 5.3.1-pd-5.3.y #20191015
> [373579.968896] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [373579.970287] cifsd D 0 789 2 0x80004000
> [373579.970291] Call Trace:
> [373579.970321] ? __schedule+0x540/0xac0
> [373579.970324] ? firmware_map_remove+0xe9/0xe9
> [373579.970327] ? string_nocheck+0xb0/0xd0
> [373579.970334] ? _raw_spin_lock+0x7a/0xd0
> [373579.970336] schedule+0x5e/0x100
> [373579.970338] schedule_preempt_disabled+0xa/0x10
> [373579.970340] __mutex_lock.isra.4+0x484/0x820
> [373579.970343] ? mutex_trylock+0x90/0x90
> [373579.970357] ? irq_work_claim+0x2e/0x50
> [373579.970359] ? irq_work_queue+0x9/0x20
> [373579.970362] ? mutex_lock+0xce/0xe0
> [373579.970363] mutex_lock+0xce/0xe0
> [373579.970365] ? __mutex_lock_slowpath+0x10/0x10
> [373579.970418] dfs_cache_noreq_find+0xa7/0x190 [cifs]
> [373579.970459] cifs_reconnect+0x16c/0x1360 [cifs]
> [373579.970503] ? smb2_calc_size+0x15c/0x250 [cifs]
> [373579.970542] ? extract_hostname+0xa0/0xa0 [cifs]
> [373579.970544] ? _raw_spin_trylock+0x91/0xe0
> [373579.970546] ? _raw_spin_trylock_bh+0x100/0x100
> [373579.970548] ? ___ratelimit+0x106/0x190
> [373579.970588] cifs_handle_standard+0x252/0x270 [cifs]
> [373579.970628] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
> [373579.970672] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373579.970674] ? __switch_to_asm+0x40/0x70
> [373579.970675] ? __switch_to_asm+0x34/0x70
> [373579.970677] ? __switch_to_asm+0x40/0x70
> [373579.970678] ? __switch_to_asm+0x34/0x70
> [373579.970680] ? __switch_to_asm+0x40/0x70
> [373579.970681] ? __switch_to_asm+0x34/0x70
> [373579.970683] ? __switch_to_asm+0x40/0x70
> [373579.970684] ? __switch_to_asm+0x34/0x70
> [373579.970686] ? __switch_to_asm+0x40/0x70
> [373579.970687] ? __switch_to_asm+0x34/0x70
> [373579.970689] ? __switch_to_asm+0x40/0x70
> [373579.970690] ? __switch_to_asm+0x34/0x70
> [373579.970692] ? __switch_to_asm+0x40/0x70
> [373579.970693] ? __switch_to_asm+0x34/0x70
> [373579.970695] ? __switch_to_asm+0x40/0x70
> [373579.970697] ? __switch_to_asm+0x34/0x70
> [373579.970698] ? __switch_to_asm+0x40/0x70
> [373579.970700] ? __switch_to_asm+0x34/0x70
> [373579.970701] ? __switch_to_asm+0x40/0x70
> [373579.970703] ? __switch_to_asm+0x34/0x70
> [373579.970704] ? __switch_to_asm+0x40/0x70
> [373579.970706] ? __switch_to_asm+0x34/0x70
> [373579.970707] ? __switch_to_asm+0x40/0x70
> [373579.970709] ? __switch_to_asm+0x34/0x70
> [373579.970710] ? __switch_to_asm+0x40/0x70
> [373579.970712] ? __switch_to_asm+0x34/0x70
> [373579.970713] ? __switch_to_asm+0x40/0x70
> [373579.970724] ? finish_task_switch+0x91/0x370
> [373579.970732] ? __switch_to+0x2ec/0x5e0
> [373579.970735] ? _raw_spin_lock_irqsave+0x8d/0xf0
> [373579.970736] ? _raw_write_lock_bh+0xe0/0xe0
> [373579.970775] ? cifs_handle_standard+0x270/0x270 [cifs]
> [373579.970780] kthread+0x192/0x1e0
> [373579.970782] ? kthread_create_worker_on_cpu+0xc0/0xc0
> [373579.970786] ret_from_fork+0x35/0x40
> [373584.061557] fs/cifs/smb2pdu.c: In echo request
> [373584.061573] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373584.061578] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373588.157191] fs/cifs/smb2pdu.c: In echo request
> [373588.157233] CIFS VFS: Error -32 sending data on socket to server
> [373588.158699] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
> [373588.158710] fs/cifs/smb2pdu.c: Echo request failed: -32
> [373588.158718] fs/cifs/connect.c: Unable to send echo request to server: stor02.bk.prodrive.nl
> [373645.495852] fs/cifs/smb2pdu.c: In echo request
> [373645.495870] fs/cifs/smb2pdu.c: Echo request failed: -11
> [373645.495876] fs/cifs/connect.c: Unable to send echo request to server: DC02
> [373649.591382] fs/cifs/smb2pdu.c: In echo request
>
> The last part repeated over and over again in the log.
>
> Gr, Martijn de Gouw
>
--
Martijn de Gouw
Designer
Prodrive Technologies
Mobile: +31 63 17 76 161
Phone: +31 40 26 76 200
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Kernel hangs in cifs_reconnect
2019-10-29 11:07 ` Martijn de Gouw
@ 2019-10-29 14:49 ` Paulo Alcantara
2019-10-29 18:38 ` Martijn de Gouw
2019-10-30 19:17 ` Martijn de Gouw
0 siblings, 2 replies; 9+ messages in thread
From: Paulo Alcantara @ 2019-10-29 14:49 UTC (permalink / raw)
To: Martijn de Gouw, Aurélien Aptel, linux-cifs
[-- Attachment #1: Type: text/plain, Size: 482 bytes --]
Hi Martijn,
Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
> Anybody any idea on what goes wrong here?
Looks like an use-after-free bug in cifs_reconnect(). cifs superblock
gets freed due to automount expiration and then we dereference it in
dfs_cache_noreq_find().
> Is any of the recently posted patches related to my issue, because I'm
> more that willing to test out patches if needed.
Could you please test it again with below patch?
cheers,
Paulo
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-cifs-Fix-use-after-free-bug-in-cifs_reconnect.patch --]
[-- Type: text/x-patch, Size: 8076 bytes --]
From 6143e4f4af194ebd1ec58d51510e0caf5917786d Mon Sep 17 00:00:00 2001
From: "Paulo Alcantara (SUSE)" <pc@cjr.nz>
Date: Wed, 23 Oct 2019 18:00:37 -0300
Subject: [PATCH] cifs: Fix use-after-free bug in cifs_reconnect()
Ensure we grab an active reference in cifs superblock while doing
failover to prevent automounts (DFS links) of expiring and then
destroying the superblock pointer.
This patch fixes the following KASAN report:
[ 464.301462] BUG: KASAN: use-after-free in
cifs_reconnect+0x6ab/0x1350
[ 464.303052] Read of size 8 at addr ffff888155e580d0 by task
cifsd/1107
[ 464.304682] CPU: 3 PID: 1107 Comm: cifsd Not tainted 5.4.0-rc4+ #13
[ 464.305552] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS rel-1.12.1-0-ga5cab58-rebuilt.opensuse.org 04/01/2014
[ 464.307146] Call Trace:
[ 464.307875] dump_stack+0x5b/0x90
[ 464.308631] print_address_description.constprop.0+0x16/0x200
[ 464.309478] ? cifs_reconnect+0x6ab/0x1350
[ 464.310253] ? cifs_reconnect+0x6ab/0x1350
[ 464.311040] __kasan_report.cold+0x1a/0x41
[ 464.311811] ? cifs_reconnect+0x6ab/0x1350
[ 464.312563] kasan_report+0xe/0x20
[ 464.313300] cifs_reconnect+0x6ab/0x1350
[ 464.314062] ? extract_hostname.part.0+0x90/0x90
[ 464.314829] ? printk+0xad/0xde
[ 464.315525] ? _raw_spin_lock+0x7c/0xd0
[ 464.316252] ? _raw_read_lock_irq+0x40/0x40
[ 464.316961] ? ___ratelimit+0xed/0x182
[ 464.317655] cifs_readv_from_socket+0x289/0x3b0
[ 464.318386] cifs_read_from_socket+0x98/0xd0
[ 464.319078] ? cifs_readv_from_socket+0x3b0/0x3b0
[ 464.319782] ? try_to_wake_up+0x43c/0xa90
[ 464.320463] ? cifs_small_buf_get+0x4b/0x60
[ 464.321173] ? allocate_buffers+0x98/0x1a0
[ 464.321856] cifs_demultiplex_thread+0x218/0x14a0
[ 464.322558] ? cifs_handle_standard+0x270/0x270
[ 464.323237] ? __switch_to_asm+0x40/0x70
[ 464.323893] ? __switch_to_asm+0x34/0x70
[ 464.324554] ? __switch_to_asm+0x40/0x70
[ 464.325226] ? __switch_to_asm+0x40/0x70
[ 464.325863] ? __switch_to_asm+0x34/0x70
[ 464.326505] ? __switch_to_asm+0x40/0x70
[ 464.327161] ? __switch_to_asm+0x34/0x70
[ 464.327784] ? finish_task_switch+0xa1/0x330
[ 464.328414] ? __switch_to+0x363/0x640
[ 464.329044] ? __schedule+0x575/0xaf0
[ 464.329655] ? _raw_spin_lock_irqsave+0x82/0xe0
[ 464.330301] kthread+0x1a3/0x1f0
[ 464.330884] ? cifs_handle_standard+0x270/0x270
[ 464.331624] ? kthread_create_on_node+0xd0/0xd0
[ 464.332347] ret_from_fork+0x35/0x40
[ 464.333577] Allocated by task 1110:
[ 464.334381] save_stack+0x1b/0x80
[ 464.335123] __kasan_kmalloc.constprop.0+0xc2/0xd0
[ 464.335848] cifs_smb3_do_mount+0xd4/0xb00
[ 464.336619] legacy_get_tree+0x6b/0xa0
[ 464.337235] vfs_get_tree+0x41/0x110
[ 464.337975] fc_mount+0xa/0x40
[ 464.338557] vfs_kern_mount.part.0+0x6c/0x80
[ 464.339227] cifs_dfs_d_automount+0x336/0xd29
[ 464.339846] follow_managed+0x1b1/0x450
[ 464.340449] lookup_fast+0x231/0x4a0
[ 464.341039] path_openat+0x240/0x1fd0
[ 464.341634] do_filp_open+0x126/0x1c0
[ 464.342277] do_sys_open+0x1eb/0x2c0
[ 464.342957] do_syscall_64+0x5e/0x190
[ 464.343555] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 464.344772] Freed by task 0:
[ 464.345347] save_stack+0x1b/0x80
[ 464.345966] __kasan_slab_free+0x12c/0x170
[ 464.346576] kfree+0xa6/0x270
[ 464.347211] rcu_core+0x39c/0xc80
[ 464.347800] __do_softirq+0x10d/0x3da
[ 464.348919] The buggy address belongs to the object at
ffff888155e58000
which belongs to the cache kmalloc-256 of size 256
[ 464.350222] The buggy address is located 208 bytes inside of
256-byte region [ffff888155e58000, ffff888155e58100)
[ 464.351575] The buggy address belongs to the page:
[ 464.352333] page:ffffea0005579600 refcount:1 mapcount:0
mapping:ffff88815a803400 index:0x0 compound_mapcount: 0
[ 464.353583] flags: 0x200000000010200(slab|head)
[ 464.354209] raw: 0200000000010200 ffffea0005576200 0000000400000004
ffff88815a803400
[ 464.355353] raw: 0000000000000000 0000000080100010 00000001ffffffff
0000000000000000
[ 464.356458] page dumped because: kasan: bad access detected
[ 464.367005] Memory state around the buggy address:
[ 464.367787] ffff888155e57f80: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 464.368877] ffff888155e58000: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 464.369967] >ffff888155e58080: fb fb fb fb fb fb fb fb fb fb fb fb
fb fb fb fb
[ 464.371111] ^
[ 464.371775] ffff888155e58100: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 464.372893] ffff888155e58180: fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc fc
[ 464.373983] ==================================================================
Signed-off-by: Paulo Alcantara (SUSE) <pc@cjr.nz>
---
fs/cifs/connect.c | 46 +++++++++++++++++++++++++++++++++++-----------
1 file changed, 35 insertions(+), 11 deletions(-)
diff --git a/fs/cifs/connect.c b/fs/cifs/connect.c
index ccaa8bad336f..a4ae4d944a3a 100644
--- a/fs/cifs/connect.c
+++ b/fs/cifs/connect.c
@@ -387,7 +387,7 @@ static inline int reconn_set_ipaddr(struct TCP_Server_Info *server)
#ifdef CONFIG_CIFS_DFS_UPCALL
struct super_cb_data {
struct TCP_Server_Info *server;
- struct cifs_sb_info *cifs_sb;
+ struct super_block *sb;
};
/* These functions must be called with server->srv_mutex held */
@@ -398,25 +398,39 @@ static void super_cb(struct super_block *sb, void *arg)
struct cifs_sb_info *cifs_sb;
struct cifs_tcon *tcon;
- if (d->cifs_sb)
+ if (d->sb)
return;
cifs_sb = CIFS_SB(sb);
tcon = cifs_sb_master_tcon(cifs_sb);
if (tcon->ses->server == d->server)
- d->cifs_sb = cifs_sb;
+ d->sb = sb;
}
-static inline struct cifs_sb_info *
-find_super_by_tcp(struct TCP_Server_Info *server)
+static struct super_block *get_tcp_super(struct TCP_Server_Info *server)
{
struct super_cb_data d = {
.server = server,
- .cifs_sb = NULL,
+ .sb = NULL,
};
iterate_supers_type(&cifs_fs_type, super_cb, &d);
- return d.cifs_sb ? d.cifs_sb : ERR_PTR(-ENOENT);
+
+ if (unlikely(!d.sb))
+ return ERR_PTR(-ENOENT);
+ /*
+ * Grab an active reference in order to prevent automounts (DFS links)
+ * of expiring and then freeing up our cifs superblock pointer while
+ * we're doing failover.
+ */
+ cifs_sb_active(d.sb);
+ return d.sb;
+}
+
+static inline void put_tcp_super(struct super_block *sb)
+{
+ if (!IS_ERR_OR_NULL(sb))
+ cifs_sb_deactive(sb);
}
static void reconn_inval_dfs_target(struct TCP_Server_Info *server,
@@ -480,6 +494,7 @@ cifs_reconnect(struct TCP_Server_Info *server)
struct mid_q_entry *mid_entry;
struct list_head retry_list;
#ifdef CONFIG_CIFS_DFS_UPCALL
+ struct super_block *sb = NULL;
struct cifs_sb_info *cifs_sb = NULL;
struct dfs_cache_tgt_list tgt_list = {0};
struct dfs_cache_tgt_iterator *tgt_it = NULL;
@@ -489,13 +504,15 @@ cifs_reconnect(struct TCP_Server_Info *server)
server->nr_targets = 1;
#ifdef CONFIG_CIFS_DFS_UPCALL
spin_unlock(&GlobalMid_Lock);
- cifs_sb = find_super_by_tcp(server);
- if (IS_ERR(cifs_sb)) {
- rc = PTR_ERR(cifs_sb);
+ sb = get_tcp_super(server);
+ if (IS_ERR(sb)) {
+ rc = PTR_ERR(sb);
cifs_dbg(FYI, "%s: will not do DFS failover: rc = %d\n",
__func__, rc);
- cifs_sb = NULL;
+ sb = NULL;
} else {
+ cifs_sb = CIFS_SB(sb);
+
rc = reconn_setup_dfs_targets(cifs_sb, &tgt_list, &tgt_it);
if (rc && (rc != -EOPNOTSUPP)) {
cifs_server_dbg(VFS, "%s: no target servers for DFS failover\n",
@@ -512,6 +529,10 @@ cifs_reconnect(struct TCP_Server_Info *server)
/* the demux thread will exit normally
next time through the loop */
spin_unlock(&GlobalMid_Lock);
+#ifdef CONFIG_CIFS_DFS_UPCALL
+ dfs_cache_free_tgts(&tgt_list);
+ put_tcp_super(sb);
+#endif
return rc;
} else
server->tcpStatus = CifsNeedReconnect;
@@ -638,7 +659,10 @@ cifs_reconnect(struct TCP_Server_Info *server)
__func__, rc);
}
dfs_cache_free_tgts(&tgt_list);
+
}
+
+ put_tcp_super(sb);
#endif
if (server->tcpStatus == CifsNeedNegotiate)
mod_delayed_work(cifsiod_wq, &server->echo, 0);
--
2.23.0
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: Kernel hangs in cifs_reconnect
2019-10-29 14:49 ` Paulo Alcantara
@ 2019-10-29 18:38 ` Martijn de Gouw
2019-10-30 19:17 ` Martijn de Gouw
1 sibling, 0 replies; 9+ messages in thread
From: Martijn de Gouw @ 2019-10-29 18:38 UTC (permalink / raw)
To: Paulo Alcantara, Aurélien Aptel, linux-cifs
Hi Paulo
On 29-10-2019 15:49, Paulo Alcantara wrote:
> Hi Martijn,
>
> Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
>
>> Anybody any idea on what goes wrong here?
>
> Looks like an use-after-free bug in cifs_reconnect(). cifs superblock
> gets freed due to automount expiration and then we dereference it in
> dfs_cache_noreq_find().
>
>> Is any of the recently posted patches related to my issue, because I'm
>> more that willing to test out patches if needed.
>
> Could you please test it again with below patch?
It's running the same version+your patch now, will let you know within a
couple of days!
Thanks, Martijn
>
> cheers,
> Paulo
>
--
Martijn de Gouw
Designer
Prodrive Technologies
Mobile: +31 63 17 76 161
Phone: +31 40 26 76 200
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Kernel hangs in cifs_reconnect
2019-10-29 14:49 ` Paulo Alcantara
2019-10-29 18:38 ` Martijn de Gouw
@ 2019-10-30 19:17 ` Martijn de Gouw
2019-10-30 19:33 ` Paulo Alcantara
1 sibling, 1 reply; 9+ messages in thread
From: Martijn de Gouw @ 2019-10-30 19:17 UTC (permalink / raw)
To: Paulo Alcantara, Aurélien Aptel, linux-cifs
Hi Paulo,
On 29-10-2019 15:49, Paulo Alcantara wrote:
> Hi Martijn,
>
> Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
>
>> Anybody any idea on what goes wrong here?
>
> Looks like an use-after-free bug in cifs_reconnect(). cifs superblock
> gets freed due to automount expiration and then we dereference it in
> dfs_cache_noreq_find().
>
>> Is any of the recently posted patches related to my issue, because I'm
>> more that willing to test out patches if needed.
>
> Could you please test it again with below patch?
Again cifs stopped working, it looks more like a live/dead lock to me than a use-after-free.
I cut the dmesg from the moment the echo to DC02 starts failing.
[70939.980125] fs/cifs/smb2pdu.c: In echo request
[70939.980137] fs/cifs/smb2pdu.c: Echo request failed: -11
[70939.980143] fs/cifs/connect.c: Unable to send echo request to server: DC02
[70960.106599] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13353 with uid: 999
[70960.106697] fs/cifs/transport.c: Sending smb: smb_len=372
[70960.116162] fs/cifs/connect.c: RFC1002 header 0x1b8
[70960.116170] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[70960.116171] fs/cifs/smb2misc.c: SMB2 len 208
[70960.116185] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[70960.116186] fs/cifs/smb2misc.c: SMB2 len 104
[70960.116196] fs/cifs/smb2misc.c: SMB2 len 124
[70960.116197] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4066
[70960.116200] fs/cifs/smb2ops.c: add 3 credits total=512
[70960.116218] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4064 state=4
[70960.116227] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4065 state=4
[70960.116230] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4066 state=4
[70960.116234] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[70960.116242] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[70960.116247] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[70960.116290] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13353) rc = 0
[70960.116319] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13354 with uid: 999
[70960.116429] fs/cifs/transport.c: Sending smb: smb_len=348
[70960.116829] fs/cifs/connect.c: RFC1002 header 0x1b8
[70960.116834] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[70960.116835] fs/cifs/smb2misc.c: SMB2 len 208
[70960.116842] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[70960.116843] fs/cifs/smb2misc.c: SMB2 len 104
[70960.116856] fs/cifs/smb2misc.c: SMB2 len 124
[70960.116857] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 74
[70960.116859] fs/cifs/smb2ops.c: add 30 credits total=897
[70960.116872] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=72 state=4
[70960.116874] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=73 state=4
[70960.116876] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=74 state=4
[70960.116877] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[70960.116882] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[70960.116887] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[70960.116954] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13354) rc = 0
[71001.418549] fs/cifs/smb2pdu.c: In echo request
[71001.418565] fs/cifs/smb2pdu.c: Echo request failed: -11
[71001.418570] fs/cifs/connect.c: Unable to send echo request to server: DC02
[71020.097059] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13355 with uid: 999
[71020.097165] fs/cifs/transport.c: Sending smb: smb_len=372
[71020.097760] fs/cifs/connect.c: RFC1002 header 0x1b8
[71020.097767] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71020.097768] fs/cifs/smb2misc.c: SMB2 len 208
[71020.097777] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71020.097778] fs/cifs/smb2misc.c: SMB2 len 104
[71020.097788] fs/cifs/smb2misc.c: SMB2 len 124
[71020.097789] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4069
[71020.097796] fs/cifs/smb2ops.c: add 3 credits total=512
[71020.098387] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4067 state=4
[71020.098395] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4068 state=4
[71020.098398] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4069 state=4
[71020.098405] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71020.098414] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71020.098419] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71020.098461] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13355) rc = 0
[71020.098488] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13356 with uid: 999
[71020.098556] fs/cifs/transport.c: Sending smb: smb_len=348
[71020.098915] fs/cifs/connect.c: RFC1002 header 0x1b8
[71020.098921] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71020.098922] fs/cifs/smb2misc.c: SMB2 len 208
[71020.098932] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71020.098932] fs/cifs/smb2misc.c: SMB2 len 104
[71020.098942] fs/cifs/smb2misc.c: SMB2 len 124
[71020.098944] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 77
[71020.098946] fs/cifs/smb2ops.c: add 30 credits total=924
[71020.098956] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=75 state=4
[71020.098958] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=76 state=4
[71020.098959] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=77 state=4
[71020.098961] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71020.098968] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71020.098973] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71020.099011] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13356) rc = 0
[71044.418723] INFO: task node_exporter:7214 blocked for more than 120 seconds.
[71044.420037] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71044.421491] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71044.422762] node_exporter D 0 7214 1 0x00000000
[71044.422781] Call Trace:
[71044.422806] ? __schedule+0x540/0xac0
[71044.422813] ? firmware_map_remove+0xe9/0xe9
[71044.422821] ? vsnprintf+0x32c/0x870
[71044.422826] ? _raw_spin_lock+0x7a/0xd0
[71044.422829] schedule+0x5e/0x100
[71044.422837] schedule_preempt_disabled+0xa/0x10
[71044.422840] __mutex_lock.isra.4+0x484/0x820
[71044.422844] ? mutex_trylock+0x90/0x90
[71044.422848] ? string_nocheck+0xb0/0xd0
[71044.422854] ? pointer+0x387/0x460
[71044.422859] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71044.422863] ? _raw_write_lock_bh+0xe0/0xe0
[71044.422868] ? mutex_lock+0xce/0xe0
[71044.422871] mutex_lock+0xce/0xe0
[71044.422875] ? __mutex_lock_slowpath+0x10/0x10
[71044.422892] ? find_nls+0x7d/0xa0
[71044.422976] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
[71044.422992] ? deref_stack_reg+0x88/0xd0
[71044.422995] ? 0xffffffffa1e00000
[71044.423065] ? SMB2_tcon+0xab0/0xab0 [cifs]
[71044.423067] ? unwind_next_frame+0x90a/0x980
[71044.423070] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71044.423081] ? __module_text_address+0x11/0xa0
[71044.423087] ? __is_insn_slot_addr+0x1b/0x70
[71044.423094] ? is_bpf_text_address+0xa/0x20
[71044.423101] ? kernel_text_address+0xe2/0xf0
[71044.423104] ? __kernel_text_address+0xe/0x30
[71044.423108] ? unwind_get_return_address+0x2f/0x50
[71044.423114] ? create_prof_cpu_mask+0x20/0x20
[71044.423120] ? arch_stack_walk+0x92/0xe0
[71044.423125] ? stack_trace_save+0x8a/0xb0
[71044.423195] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
[71044.423272] SMB2_open_init+0x143/0x12b0 [cifs]
[71044.423339] ? cifs_statfs+0x13a/0x420 [cifs]
[71044.423352] ? statfs_by_dentry+0xa5/0xf0
[71044.423354] ? vfs_statfs+0x28/0x110
[71044.423357] ? __do_sys_statfs+0x64/0xc0
[71044.423413] ? smb2_parse_contexts+0x270/0x270 [cifs]
[71044.423416] ? _raw_write_trylock+0xe0/0xe0
[71044.423419] ? _raw_spin_lock+0x7a/0xd0
[71044.423421] ? _raw_write_trylock+0xe0/0xe0
[71044.423429] ? memset+0x1f/0x40
[71044.423433] ? stack_access_ok+0x35/0x90
[71044.423437] ? deref_stack_reg+0x88/0xd0
[71044.423439] ? stack_access_ok+0x35/0x90
[71044.423442] ? deref_stack_reg+0x88/0xd0
[71044.423446] ? __read_once_size_nocheck.constprop.6+0x10/0x10
[71044.423450] ? put_dec_trunc8+0x73/0x110
[71044.423453] ? number+0x356/0x4b0
[71044.423456] ? widen_string+0x23/0xf0
[71044.423517] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
[71044.423558] smb2_query_info_compound+0x242/0x4d0 [cifs]
[71044.423601] ? smb2_query_symlink+0xc70/0xc70 [cifs]
[71044.423604] ? pointer+0x460/0x460
[71044.423606] ? kernel_text_address+0xe2/0xf0
[71044.423608] ? va_format.isra.12+0xee/0x100
[71044.423612] ? vsnprintf+0x870/0x870
[71044.423614] ? string_nocheck+0xb0/0xd0
[71044.423616] ? pointer+0x387/0x460
[71044.423619] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71044.423620] ? _raw_write_lock_bh+0xe0/0xe0
[71044.423622] ? _raw_spin_lock+0x7a/0xd0
[71044.423624] ? _raw_write_trylock+0xe0/0xe0
[71044.423626] ? up+0x32/0x70
[71044.423630] ? __switch_to_asm+0x34/0x70
[71044.423636] ? dynamic_emit_prefix+0x29/0x220
[71044.423684] ? smb2_queryfs+0xd9/0x1c0 [cifs]
[71044.423735] smb2_queryfs+0xd9/0x1c0 [cifs]
[71044.423776] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
[71044.423780] ? vfs_mknod+0xc0/0x320
[71044.423785] ? map_id_up+0x12f/0x1d0
[71044.423787] ? make_kprojid+0x20/0x20
[71044.423789] ? _raw_spin_lock+0x7a/0xd0
[71044.423837] cifs_statfs+0x13a/0x420 [cifs]
[71044.423844] statfs_by_dentry+0xa5/0xf0
[71044.423848] vfs_statfs+0x28/0x110
[71044.423851] user_statfs+0x91/0xf0
[71044.423853] ? vfs_statfs+0x110/0x110
[71044.423855] ? __schedule+0x562/0xac0
[71044.423857] __do_sys_statfs+0x64/0xc0
[71044.423859] ? user_statfs+0xf0/0xf0
[71044.423865] do_syscall_64+0x73/0x190
[71044.423868] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71044.423875] RIP: 0033:0x4a5c20
[71044.423880] Code: Bad RIP value.
[71044.423882] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[71044.423888] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX: 00000000004a5c20
[71044.423890] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI: 000000c0001c43c0
[71044.423892] RBP: 000000c0003414f0 R08: 0000000000000000 R09: 0000000000000000
[71044.423893] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[71044.423894] R13: 000000000000001f R14: 000000000000001e R15: 0000000000000100
[71044.423926] INFO: task cifsd:9136 blocked for more than 120 seconds.
[71044.425076] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71044.426421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71044.427718] cifsd D 0 9136 2 0x80004000
[71044.427721] Call Trace:
[71044.427738] ? __schedule+0x540/0xac0
[71044.427745] ? firmware_map_remove+0xe9/0xe9
[71044.427752] ? _raw_spin_lock+0x7a/0xd0
[71044.427757] schedule+0x5e/0x100
[71044.427765] schedule_preempt_disabled+0xa/0x10
[71044.427775] __mutex_lock.isra.4+0x484/0x820
[71044.427781] ? mutex_trylock+0x90/0x90
[71044.427787] ? irq_work_claim+0x2e/0x50
[71044.427792] ? vprintk_emit+0x11d/0x2e0
[71044.427795] ? mutex_lock+0xce/0xe0
[71044.427797] mutex_lock+0xce/0xe0
[71044.427800] ? __mutex_lock_slowpath+0x10/0x10
[71044.427877] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[71044.427951] cifs_reconnect+0x18c/0x1510 [cifs]
[71044.427970] ? ___ratelimit+0x106/0x190
[71044.428050] ? smb2_calc_size+0x15c/0x250 [cifs]
[71044.428107] ? extract_hostname+0xa0/0xa0 [cifs]
[71044.428121] ? _raw_spin_trylock+0x91/0xe0
[71044.428129] ? _raw_spin_trylock_bh+0x100/0x100
[71044.428134] ? ___ratelimit+0x106/0x190
[71044.428179] cifs_handle_standard+0x252/0x270 [cifs]
[71044.428224] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[71044.428267] ? cifs_handle_standard+0x270/0x270 [cifs]
[71044.428272] ? __switch_to_asm+0x40/0x70
[71044.428276] ? __switch_to_asm+0x34/0x70
[71044.428278] ? __switch_to_asm+0x40/0x70
[71044.428280] ? __switch_to_asm+0x34/0x70
[71044.428281] ? __switch_to_asm+0x40/0x70
[71044.428283] ? __switch_to_asm+0x34/0x70
[71044.428284] ? __switch_to_asm+0x40/0x70
[71044.428286] ? __switch_to_asm+0x34/0x70
[71044.428287] ? __switch_to_asm+0x40/0x70
[71044.428289] ? __switch_to_asm+0x34/0x70
[71044.428290] ? __switch_to_asm+0x40/0x70
[71044.428292] ? __switch_to_asm+0x34/0x70
[71044.428293] ? __switch_to_asm+0x40/0x70
[71044.428295] ? __switch_to_asm+0x34/0x70
[71044.428296] ? __switch_to_asm+0x40/0x70
[71044.428298] ? __switch_to_asm+0x34/0x70
[71044.428299] ? __switch_to_asm+0x40/0x70
[71044.428301] ? __switch_to_asm+0x34/0x70
[71044.428303] ? __switch_to_asm+0x40/0x70
[71044.428304] ? __switch_to_asm+0x34/0x70
[71044.428306] ? __switch_to_asm+0x40/0x70
[71044.428307] ? __switch_to_asm+0x34/0x70
[71044.428309] ? __switch_to_asm+0x40/0x70
[71044.428310] ? __switch_to_asm+0x34/0x70
[71044.428312] ? __switch_to_asm+0x40/0x70
[71044.428313] ? __switch_to_asm+0x34/0x70
[71044.428317] ? finish_task_switch+0xf6/0x370
[71044.428319] ? __switch_to+0x2ec/0x5e0
[71044.428321] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71044.428323] ? _raw_write_lock_bh+0xe0/0xe0
[71044.428375] ? cifs_handle_standard+0x270/0x270 [cifs]
[71044.428378] kthread+0x192/0x1e0
[71044.428381] ? kthread_create_worker_on_cpu+0xc0/0xc0
[71044.428384] ret_from_fork+0x35/0x40
[71062.848999] fs/cifs/smb2pdu.c: In echo request
[71062.849014] fs/cifs/smb2pdu.c: Echo request failed: -11
[71062.849025] fs/cifs/connect.c: Unable to send echo request to server: DC02
[71080.092803] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13357 with uid: 999
[71080.092915] fs/cifs/transport.c: Sending smb: smb_len=372
[71080.097568] fs/cifs/connect.c: RFC1002 header 0x1b8
[71080.097580] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71080.097581] fs/cifs/smb2misc.c: SMB2 len 208
[71080.097591] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71080.097592] fs/cifs/smb2misc.c: SMB2 len 104
[71080.097601] fs/cifs/smb2misc.c: SMB2 len 124
[71080.097603] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4072
[71080.097605] fs/cifs/smb2ops.c: add 3 credits total=512
[71080.097741] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4070 state=4
[71080.097749] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4071 state=4
[71080.097752] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4072 state=4
[71080.097758] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71080.097767] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71080.097771] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71080.097812] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13357) rc = 0
[71080.097992] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13358 with uid: 999
[71080.098070] fs/cifs/transport.c: Sending smb: smb_len=348
[71080.098487] fs/cifs/connect.c: RFC1002 header 0x1b8
[71080.098492] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71080.098493] fs/cifs/smb2misc.c: SMB2 len 208
[71080.098500] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71080.098501] fs/cifs/smb2misc.c: SMB2 len 104
[71080.098510] fs/cifs/smb2misc.c: SMB2 len 124
[71080.098512] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 80
[71080.098514] fs/cifs/smb2ops.c: add 30 credits total=951
[71080.098609] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=78 state=4
[71080.098610] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=79 state=4
[71080.098612] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=80 state=4
[71080.098613] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71080.098619] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71080.098624] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71080.098655] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13358) rc = 0
[71105.853107] fs/cifs/connect.c: Existing tcp session with server found
[71105.853114] fs/cifs/dfs_cache.c: CIFS VFS: in do_refresh_tcon as Xid: 13359 with uid: 0
[71116.587766] fs/cifs/inode.c: CIFS VFS: in cifs_revalidate_dentry_attr as Xid: 13360 with uid: 11025
[71116.587834] fs/cifs/dir.c: name: \SOMEDIRX
[71116.587842] fs/cifs/inode.c: Update attributes: \SOMEDIRX inode 0x000000009f726d03 count 1 dentry: 0x00000000d22e461a d_time 4312598279 jiffies 4312672891
[71116.587847] fs/cifs/inode.c: Getting info on \SOMEDIRX
[71116.588001] fs/cifs/transport.c: Sending smb: smb_len=388
[71124.283416] fs/cifs/smb2pdu.c: In echo request
[71124.283434] fs/cifs/smb2pdu.c: Echo request failed: -11
[71124.283442] fs/cifs/connect.c: Unable to send echo request to server: DC02
[71140.087648] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13361 with uid: 999
[71140.087743] fs/cifs/transport.c: Sending smb: smb_len=372
[71140.088088] fs/cifs/connect.c: RFC1002 header 0x1b8
[71140.088095] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71140.088096] fs/cifs/smb2misc.c: SMB2 len 208
[71140.088106] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71140.088107] fs/cifs/smb2misc.c: SMB2 len 104
[71140.088116] fs/cifs/smb2misc.c: SMB2 len 124
[71140.088118] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4075
[71140.088120] fs/cifs/smb2ops.c: add 3 credits total=512
[71140.088137] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4073 state=4
[71140.088141] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4074 state=4
[71140.088144] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4075 state=4
[71140.088147] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71140.088153] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71140.088158] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71140.088190] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13361) rc = 0
[71140.088218] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13362 with uid: 999
[71140.088304] fs/cifs/transport.c: Sending smb: smb_len=348
[71140.088643] fs/cifs/connect.c: RFC1002 header 0x1b8
[71140.088647] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71140.088648] fs/cifs/smb2misc.c: SMB2 len 208
[71140.088655] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71140.088656] fs/cifs/smb2misc.c: SMB2 len 104
[71140.088664] fs/cifs/smb2misc.c: SMB2 len 124
[71140.088665] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 83
[71140.088667] fs/cifs/smb2ops.c: add 30 credits total=978
[71140.088680] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=81 state=4
[71140.088682] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=82 state=4
[71140.088684] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=83 state=4
[71140.088686] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71140.088692] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71140.088696] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71140.088728] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13362) rc = 0
[71165.239724] INFO: task node_exporter:7214 blocked for more than 241 seconds.
[71165.240866] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71165.241935] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71165.243018] node_exporter D 0 7214 1 0x00000000
[71165.243022] Call Trace:
[71165.243028] ? __schedule+0x540/0xac0
[71165.243031] ? firmware_map_remove+0xe9/0xe9
[71165.243035] ? vsnprintf+0x32c/0x870
[71165.243038] ? _raw_spin_lock+0x7a/0xd0
[71165.243040] schedule+0x5e/0x100
[71165.243042] schedule_preempt_disabled+0xa/0x10
[71165.243044] __mutex_lock.isra.4+0x484/0x820
[71165.243047] ? mutex_trylock+0x90/0x90
[71165.243049] ? string_nocheck+0xb0/0xd0
[71165.243051] ? pointer+0x387/0x460
[71165.243054] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71165.243056] ? _raw_write_lock_bh+0xe0/0xe0
[71165.243059] ? mutex_lock+0xce/0xe0
[71165.243060] mutex_lock+0xce/0xe0
[71165.243062] ? __mutex_lock_slowpath+0x10/0x10
[71165.243065] ? find_nls+0x7d/0xa0
[71165.243133] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
[71165.243137] ? deref_stack_reg+0x88/0xd0
[71165.243139] ? 0xffffffffa1e00000
[71165.243180] ? SMB2_tcon+0xab0/0xab0 [cifs]
[71165.243181] ? unwind_next_frame+0x90a/0x980
[71165.243183] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71165.243187] ? __module_text_address+0x11/0xa0
[71165.243189] ? __is_insn_slot_addr+0x1b/0x70
[71165.243193] ? is_bpf_text_address+0xa/0x20
[71165.243195] ? kernel_text_address+0xe2/0xf0
[71165.243197] ? __kernel_text_address+0xe/0x30
[71165.243200] ? unwind_get_return_address+0x2f/0x50
[71165.243203] ? create_prof_cpu_mask+0x20/0x20
[71165.243207] ? arch_stack_walk+0x92/0xe0
[71165.243210] ? stack_trace_save+0x8a/0xb0
[71165.243251] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
[71165.243294] SMB2_open_init+0x143/0x12b0 [cifs]
[71165.243332] ? cifs_statfs+0x13a/0x420 [cifs]
[71165.243336] ? statfs_by_dentry+0xa5/0xf0
[71165.243338] ? vfs_statfs+0x28/0x110
[71165.243340] ? __do_sys_statfs+0x64/0xc0
[71165.243382] ? smb2_parse_contexts+0x270/0x270 [cifs]
[71165.243384] ? _raw_write_trylock+0xe0/0xe0
[71165.243385] ? _raw_spin_lock+0x7a/0xd0
[71165.243387] ? _raw_write_trylock+0xe0/0xe0
[71165.243389] ? memset+0x1f/0x40
[71165.243391] ? stack_access_ok+0x35/0x90
[71165.243393] ? deref_stack_reg+0x88/0xd0
[71165.243396] ? stack_access_ok+0x35/0x90
[71165.243399] ? deref_stack_reg+0x88/0xd0
[71165.243402] ? __read_once_size_nocheck.constprop.6+0x10/0x10
[71165.243404] ? put_dec_trunc8+0x73/0x110
[71165.243406] ? number+0x356/0x4b0
[71165.243408] ? widen_string+0x23/0xf0
[71165.243449] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
[71165.243490] smb2_query_info_compound+0x242/0x4d0 [cifs]
[71165.243533] ? smb2_query_symlink+0xc70/0xc70 [cifs]
[71165.243535] ? pointer+0x460/0x460
[71165.243537] ? kernel_text_address+0xe2/0xf0
[71165.243539] ? va_format.isra.12+0xee/0x100
[71165.243541] ? vsnprintf+0x870/0x870
[71165.243543] ? string_nocheck+0xb0/0xd0
[71165.243545] ? pointer+0x387/0x460
[71165.243547] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71165.243549] ? _raw_write_lock_bh+0xe0/0xe0
[71165.243551] ? _raw_spin_lock+0x7a/0xd0
[71165.243552] ? _raw_write_trylock+0xe0/0xe0
[71165.243555] ? up+0x32/0x70
[71165.243558] ? __switch_to_asm+0x34/0x70
[71165.243561] ? dynamic_emit_prefix+0x29/0x220
[71165.243602] ? smb2_queryfs+0xd9/0x1c0 [cifs]
[71165.243644] smb2_queryfs+0xd9/0x1c0 [cifs]
[71165.243688] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
[71165.243764] ? vfs_mknod+0xc0/0x320
[71165.243770] ? map_id_up+0x12f/0x1d0
[71165.243774] ? make_kprojid+0x20/0x20
[71165.243777] ? _raw_spin_lock+0x7a/0xd0
[71165.243818] cifs_statfs+0x13a/0x420 [cifs]
[71165.243825] statfs_by_dentry+0xa5/0xf0
[71165.243829] vfs_statfs+0x28/0x110
[71165.243835] user_statfs+0x91/0xf0
[71165.243838] ? vfs_statfs+0x110/0x110
[71165.243844] ? __schedule+0x562/0xac0
[71165.243847] __do_sys_statfs+0x64/0xc0
[71165.243854] ? user_statfs+0xf0/0xf0
[71165.243859] do_syscall_64+0x73/0x190
[71165.243863] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71165.243866] RIP: 0033:0x4a5c20
[71165.243873] Code: Bad RIP value.
[71165.243876] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[71165.243881] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX: 00000000004a5c20
[71165.243883] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI: 000000c0001c43c0
[71165.243885] RBP: 000000c0003414f0 R08: 0000000000000000 R09: 0000000000000000
[71165.243886] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[71165.243887] R13: 000000000000001f R14: 000000000000001e R15: 0000000000000100
[71165.243905] INFO: task cifsd:9136 blocked for more than 241 seconds.
[71165.245022] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71165.246156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71165.247272] cifsd D 0 9136 2 0x80004000
[71165.247274] Call Trace:
[71165.247281] ? __schedule+0x540/0xac0
[71165.247284] ? firmware_map_remove+0xe9/0xe9
[71165.247287] ? _raw_spin_lock+0x7a/0xd0
[71165.247289] schedule+0x5e/0x100
[71165.247291] schedule_preempt_disabled+0xa/0x10
[71165.247293] __mutex_lock.isra.4+0x484/0x820
[71165.247296] ? mutex_trylock+0x90/0x90
[71165.247300] ? irq_work_claim+0x2e/0x50
[71165.247303] ? vprintk_emit+0x11d/0x2e0
[71165.247305] ? mutex_lock+0xce/0xe0
[71165.247306] mutex_lock+0xce/0xe0
[71165.247308] ? __mutex_lock_slowpath+0x10/0x10
[71165.247358] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[71165.247398] cifs_reconnect+0x18c/0x1510 [cifs]
[71165.247402] ? ___ratelimit+0x106/0x190
[71165.247444] ? smb2_calc_size+0x15c/0x250 [cifs]
[71165.247484] ? extract_hostname+0xa0/0xa0 [cifs]
[71165.247486] ? _raw_spin_trylock+0x91/0xe0
[71165.247487] ? _raw_spin_trylock_bh+0x100/0x100
[71165.247490] ? ___ratelimit+0x106/0x190
[71165.247533] cifs_handle_standard+0x252/0x270 [cifs]
[71165.247575] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[71165.247616] ? cifs_handle_standard+0x270/0x270 [cifs]
[71165.247618] ? __switch_to_asm+0x40/0x70
[71165.247620] ? __switch_to_asm+0x34/0x70
[71165.247621] ? __switch_to_asm+0x40/0x70
[71165.247623] ? __switch_to_asm+0x34/0x70
[71165.247624] ? __switch_to_asm+0x40/0x70
[71165.247626] ? __switch_to_asm+0x34/0x70
[71165.247627] ? __switch_to_asm+0x40/0x70
[71165.247629] ? __switch_to_asm+0x34/0x70
[71165.247630] ? __switch_to_asm+0x40/0x70
[71165.247632] ? __switch_to_asm+0x34/0x70
[71165.247633] ? __switch_to_asm+0x40/0x70
[71165.247635] ? __switch_to_asm+0x34/0x70
[71165.247637] ? __switch_to_asm+0x40/0x70
[71165.247638] ? __switch_to_asm+0x34/0x70
[71165.247640] ? __switch_to_asm+0x40/0x70
[71165.247641] ? __switch_to_asm+0x34/0x70
[71165.247643] ? __switch_to_asm+0x40/0x70
[71165.247644] ? __switch_to_asm+0x34/0x70
[71165.247646] ? __switch_to_asm+0x40/0x70
[71165.247647] ? __switch_to_asm+0x34/0x70
[71165.247649] ? __switch_to_asm+0x40/0x70
[71165.247651] ? __switch_to_asm+0x34/0x70
[71165.247653] ? __switch_to_asm+0x40/0x70
[71165.247659] ? __switch_to_asm+0x34/0x70
[71165.247678] ? __switch_to_asm+0x40/0x70
[71165.247680] ? __switch_to_asm+0x34/0x70
[71165.247683] ? finish_task_switch+0xf6/0x370
[71165.247685] ? __switch_to+0x2ec/0x5e0
[71165.247688] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71165.247690] ? _raw_write_lock_bh+0xe0/0xe0
[71165.247731] ? cifs_handle_standard+0x270/0x270 [cifs]
[71165.247735] kthread+0x192/0x1e0
[71165.247737] ? kthread_create_worker_on_cpu+0xc0/0xc0
[71165.247739] ret_from_fork+0x35/0x40
[71185.717885] fs/cifs/smb2pdu.c: In echo request
[71185.717900] fs/cifs/smb2pdu.c: Echo request failed: -11
[71185.717906] fs/cifs/connect.c: Unable to send echo request to server: DC02
[71200.086057] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13363 with uid: 999
[71200.086149] fs/cifs/transport.c: Sending smb: smb_len=372
[71200.086716] fs/cifs/connect.c: RFC1002 header 0x1b8
[71200.086725] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71200.086726] fs/cifs/smb2misc.c: SMB2 len 208
[71200.086740] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71200.086741] fs/cifs/smb2misc.c: SMB2 len 104
[71200.086751] fs/cifs/smb2misc.c: SMB2 len 124
[71200.086752] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4078
[71200.086755] fs/cifs/smb2ops.c: add 3 credits total=512
[71200.086770] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4076 state=4
[71200.086775] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4077 state=4
[71200.086777] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4078 state=4
[71200.086781] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71200.086786] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71200.086791] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71200.086824] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13363) rc = 0
[71200.086841] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13364 with uid: 999
[71200.086938] fs/cifs/transport.c: Sending smb: smb_len=348
[71200.087336] fs/cifs/connect.c: RFC1002 header 0x1b8
[71200.087342] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71200.087343] fs/cifs/smb2misc.c: SMB2 len 208
[71200.087352] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71200.087353] fs/cifs/smb2misc.c: SMB2 len 104
[71200.087362] fs/cifs/smb2misc.c: SMB2 len 124
[71200.087364] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 86
[71200.087366] fs/cifs/smb2ops.c: add 30 credits total=1005
[71200.087379] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=84 state=4
[71200.087381] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=85 state=4
[71200.087383] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=86 state=4
[71200.087384] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71200.087391] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71200.087395] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71200.087428] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13364) rc = 0
[71247.152290] fs/cifs/smb2pdu.c: In echo request
[71247.152306] fs/cifs/smb2pdu.c: Echo request failed: -11
[71247.152313] fs/cifs/connect.c: Unable to send echo request to server: DC02
[71260.080476] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13365 with uid: 999
[71260.080564] fs/cifs/transport.c: Sending smb: smb_len=372
[71260.080923] fs/cifs/connect.c: RFC1002 header 0x1b8
[71260.080931] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71260.080932] fs/cifs/smb2misc.c: SMB2 len 208
[71260.080946] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71260.080947] fs/cifs/smb2misc.c: SMB2 len 104
[71260.080958] fs/cifs/smb2misc.c: SMB2 len 124
[71260.080959] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4081
[71260.080962] fs/cifs/smb2ops.c: add 3 credits total=512
[71260.080975] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4079 state=4
[71260.080979] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4080 state=4
[71260.080982] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4081 state=4
[71260.080985] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71260.080991] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71260.080996] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71260.081032] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13365) rc = 0
[71260.081163] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13366 with uid: 999
[71260.081228] fs/cifs/transport.c: Sending smb: smb_len=348
[71260.081798] fs/cifs/connect.c: RFC1002 header 0x1b8
[71260.081804] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71260.081805] fs/cifs/smb2misc.c: SMB2 len 208
[71260.081814] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71260.081814] fs/cifs/smb2misc.c: SMB2 len 104
[71260.081824] fs/cifs/smb2misc.c: SMB2 len 124
[71260.081826] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 89
[71260.081828] fs/cifs/smb2ops.c: add 30 credits total=1032
[71260.081841] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=87 state=4
[71260.081843] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=88 state=4
[71260.081845] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=89 state=4
[71260.081847] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71260.081854] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71260.081904] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71260.081936] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13366) rc = 0
[71286.060785] INFO: task node_exporter:7214 blocked for more than 362 seconds.
[71286.061989] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71286.063131] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71286.064284] node_exporter D 0 7214 1 0x00000000
[71286.064287] Call Trace:
[71286.064294] ? __schedule+0x540/0xac0
[71286.064297] ? firmware_map_remove+0xe9/0xe9
[71286.064301] ? vsnprintf+0x32c/0x870
[71286.064304] ? _raw_spin_lock+0x7a/0xd0
[71286.064306] schedule+0x5e/0x100
[71286.064308] schedule_preempt_disabled+0xa/0x10
[71286.064310] __mutex_lock.isra.4+0x484/0x820
[71286.064312] ? mutex_trylock+0x90/0x90
[71286.064315] ? string_nocheck+0xb0/0xd0
[71286.064317] ? pointer+0x387/0x460
[71286.064319] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71286.064321] ? _raw_write_lock_bh+0xe0/0xe0
[71286.064324] ? mutex_lock+0xce/0xe0
[71286.064325] mutex_lock+0xce/0xe0
[71286.064327] ? __mutex_lock_slowpath+0x10/0x10
[71286.064331] ? find_nls+0x7d/0xa0
[71286.064381] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
[71286.064386] ? deref_stack_reg+0x88/0xd0
[71286.064392] ? 0xffffffffa1e00000
[71286.064434] ? SMB2_tcon+0xab0/0xab0 [cifs]
[71286.064436] ? unwind_next_frame+0x90a/0x980
[71286.064438] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71286.064442] ? __module_text_address+0x11/0xa0
[71286.064444] ? __is_insn_slot_addr+0x1b/0x70
[71286.064447] ? is_bpf_text_address+0xa/0x20
[71286.064450] ? kernel_text_address+0xe2/0xf0
[71286.064451] ? __kernel_text_address+0xe/0x30
[71286.064454] ? unwind_get_return_address+0x2f/0x50
[71286.064457] ? create_prof_cpu_mask+0x20/0x20
[71286.064460] ? arch_stack_walk+0x92/0xe0
[71286.064463] ? stack_trace_save+0x8a/0xb0
[71286.064505] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
[71286.064548] SMB2_open_init+0x143/0x12b0 [cifs]
[71286.064586] ? cifs_statfs+0x13a/0x420 [cifs]
[71286.064590] ? statfs_by_dentry+0xa5/0xf0
[71286.064592] ? vfs_statfs+0x28/0x110
[71286.064594] ? __do_sys_statfs+0x64/0xc0
[71286.064636] ? smb2_parse_contexts+0x270/0x270 [cifs]
[71286.064638] ? _raw_write_trylock+0xe0/0xe0
[71286.064640] ? _raw_spin_lock+0x7a/0xd0
[71286.064641] ? _raw_write_trylock+0xe0/0xe0
[71286.064643] ? memset+0x1f/0x40
[71286.064646] ? stack_access_ok+0x35/0x90
[71286.064648] ? deref_stack_reg+0x88/0xd0
[71286.064650] ? stack_access_ok+0x35/0x90
[71286.064652] ? deref_stack_reg+0x88/0xd0
[71286.064654] ? __read_once_size_nocheck.constprop.6+0x10/0x10
[71286.064656] ? put_dec_trunc8+0x73/0x110
[71286.064658] ? number+0x356/0x4b0
[71286.064660] ? widen_string+0x23/0xf0
[71286.064705] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
[71286.064770] smb2_query_info_compound+0x242/0x4d0 [cifs]
[71286.064817] ? smb2_query_symlink+0xc70/0xc70 [cifs]
[71286.064821] ? pointer+0x460/0x460
[71286.064827] ? kernel_text_address+0xe2/0xf0
[71286.064835] ? va_format.isra.12+0xee/0x100
[71286.064841] ? vsnprintf+0x870/0x870
[71286.064854] ? string_nocheck+0xb0/0xd0
[71286.064869] ? pointer+0x387/0x460
[71286.064875] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71286.064886] ? _raw_write_lock_bh+0xe0/0xe0
[71286.064888] ? _raw_spin_lock+0x7a/0xd0
[71286.064889] ? _raw_write_trylock+0xe0/0xe0
[71286.064892] ? up+0x32/0x70
[71286.064896] ? __switch_to_asm+0x34/0x70
[71286.064898] ? dynamic_emit_prefix+0x29/0x220
[71286.064940] ? smb2_queryfs+0xd9/0x1c0 [cifs]
[71286.064982] smb2_queryfs+0xd9/0x1c0 [cifs]
[71286.065029] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
[71286.065035] ? vfs_mknod+0xc0/0x320
[71286.065038] ? map_id_up+0x12f/0x1d0
[71286.065040] ? make_kprojid+0x20/0x20
[71286.065042] ? _raw_spin_lock+0x7a/0xd0
[71286.065081] cifs_statfs+0x13a/0x420 [cifs]
[71286.065088] statfs_by_dentry+0xa5/0xf0
[71286.065091] vfs_statfs+0x28/0x110
[71286.065093] user_statfs+0x91/0xf0
[71286.065095] ? vfs_statfs+0x110/0x110
[71286.065099] ? __schedule+0x562/0xac0
[71286.065103] __do_sys_statfs+0x64/0xc0
[71286.065105] ? user_statfs+0xf0/0xf0
[71286.065109] do_syscall_64+0x73/0x190
[71286.065112] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71286.065116] RIP: 0033:0x4a5c20
[71286.065121] Code: Bad RIP value.
[71286.065122] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[71286.065125] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX: 00000000004a5c20
[71286.065127] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI: 000000c0001c43c0
[71286.065131] RBP: 000000c0003414f0 R08: 0000000000000000 R09: 0000000000000000
[71286.065133] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[71286.065134] R13: 000000000000001f R14: 000000000000001e R15: 0000000000000100
[71286.065156] INFO: task cifsd:9136 blocked for more than 362 seconds.
[71286.066326] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71286.068159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71286.069395] cifsd D 0 9136 2 0x80004000
[71286.069399] Call Trace:
[71286.069405] ? __schedule+0x540/0xac0
[71286.069408] ? firmware_map_remove+0xe9/0xe9
[71286.069411] ? _raw_spin_lock+0x7a/0xd0
[71286.069413] schedule+0x5e/0x100
[71286.069416] schedule_preempt_disabled+0xa/0x10
[71286.069417] __mutex_lock.isra.4+0x484/0x820
[71286.069420] ? mutex_trylock+0x90/0x90
[71286.069424] ? irq_work_claim+0x2e/0x50
[71286.069427] ? vprintk_emit+0x11d/0x2e0
[71286.069430] ? mutex_lock+0xce/0xe0
[71286.069431] mutex_lock+0xce/0xe0
[71286.069433] ? __mutex_lock_slowpath+0x10/0x10
[71286.069485] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[71286.069526] cifs_reconnect+0x18c/0x1510 [cifs]
[71286.069530] ? ___ratelimit+0x106/0x190
[71286.069572] ? smb2_calc_size+0x15c/0x250 [cifs]
[71286.069611] ? extract_hostname+0xa0/0xa0 [cifs]
[71286.069613] ? _raw_spin_trylock+0x91/0xe0
[71286.069615] ? _raw_spin_trylock_bh+0x100/0x100
[71286.069617] ? ___ratelimit+0x106/0x190
[71286.069657] cifs_handle_standard+0x252/0x270 [cifs]
[71286.069697] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[71286.069737] ? cifs_handle_standard+0x270/0x270 [cifs]
[71286.069739] ? __switch_to_asm+0x40/0x70
[71286.069741] ? __switch_to_asm+0x34/0x70
[71286.069742] ? __switch_to_asm+0x40/0x70
[71286.069744] ? __switch_to_asm+0x34/0x70
[71286.069746] ? __switch_to_asm+0x40/0x70
[71286.069748] ? __switch_to_asm+0x34/0x70
[71286.069750] ? __switch_to_asm+0x40/0x70
[71286.069752] ? __switch_to_asm+0x34/0x70
[71286.069753] ? __switch_to_asm+0x40/0x70
[71286.069755] ? __switch_to_asm+0x34/0x70
[71286.069756] ? __switch_to_asm+0x40/0x70
[71286.069758] ? __switch_to_asm+0x34/0x70
[71286.069759] ? __switch_to_asm+0x40/0x70
[71286.069761] ? __switch_to_asm+0x34/0x70
[71286.069762] ? __switch_to_asm+0x40/0x70
[71286.069764] ? __switch_to_asm+0x34/0x70
[71286.069765] ? __switch_to_asm+0x40/0x70
[71286.069767] ? __switch_to_asm+0x34/0x70
[71286.069768] ? __switch_to_asm+0x40/0x70
[71286.069770] ? __switch_to_asm+0x34/0x70
[71286.069772] ? __switch_to_asm+0x40/0x70
[71286.069773] ? __switch_to_asm+0x34/0x70
[71286.069775] ? __switch_to_asm+0x40/0x70
[71286.069776] ? __switch_to_asm+0x34/0x70
[71286.069778] ? __switch_to_asm+0x40/0x70
[71286.069779] ? __switch_to_asm+0x34/0x70
[71286.069782] ? finish_task_switch+0xf6/0x370
[71286.069784] ? __switch_to+0x2ec/0x5e0
[71286.069786] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71286.069788] ? _raw_write_lock_bh+0xe0/0xe0
[71286.069828] ? cifs_handle_standard+0x270/0x270 [cifs]
[71286.069830] kthread+0x192/0x1e0
[71286.069832] ? kthread_create_worker_on_cpu+0xc0/0xc0
[71286.069834] ret_from_fork+0x35/0x40
[71286.069842] INFO: task kworker/0:0:16069 blocked for more than 120 seconds.
[71286.071059] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71286.072279] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71286.073534] kworker/0:0 D 0 16069 2 0x80004000
[71286.073585] Workqueue: cifsiod refresh_cache_worker [cifs]
[71286.073586] Call Trace:
[71286.073590] ? __schedule+0x540/0xac0
[71286.073592] ? firmware_map_remove+0xe9/0xe9
[71286.073595] ? _raw_read_lock_irq+0x40/0x40
[71286.073597] ? _raw_spin_lock+0x7a/0xd0
[71286.073599] schedule+0x5e/0x100
[71286.073601] schedule_preempt_disabled+0xa/0x10
[71286.073603] __mutex_lock.isra.4+0x484/0x820
[71286.073606] ? mutex_trylock+0x90/0x90
[71286.073609] ? dynamic_emit_prefix+0x29/0x220
[71286.073610] ? __dynamic_pr_debug+0xf8/0x140
[71286.073612] ? dynamic_emit_prefix+0x220/0x220
[71286.073618] ? update_dl_rq_load_avg+0x2c3/0x4d0
[71286.073620] ? mutex_lock+0xce/0xe0
[71286.073621] mutex_lock+0xce/0xe0
[71286.073623] ? __mutex_lock_slowpath+0x10/0x10
[71286.073665] refresh_cache_worker+0x48f/0x14a0 [cifs]
[71286.073668] ? __switch_to_asm+0x40/0x70
[71286.073670] ? __switch_to_asm+0x40/0x70
[71286.073672] ? __switch_to_asm+0x34/0x70
[71286.073673] ? __switch_to_asm+0x40/0x70
[71286.073675] ? __switch_to_asm+0x34/0x70
[71286.073676] ? __switch_to_asm+0x40/0x70
[71286.073718] ? find_root_ses.isra.9+0x320/0x320 [cifs]
[71286.073720] ? __switch_to_asm+0x40/0x70
[71286.073722] ? __switch_to_asm+0x40/0x70
[71286.073723] ? __switch_to_asm+0x34/0x70
[71286.073725] ? __switch_to_asm+0x40/0x70
[71286.073728] ? __switch_to_asm+0x34/0x70
[71286.073729] ? __switch_to_asm+0x40/0x70
[71286.073731] ? __switch_to_asm+0x34/0x70
[71286.073732] ? __switch_to_asm+0x40/0x70
[71286.073734] ? __switch_to_asm+0x40/0x70
[71286.073736] ? __switch_to_asm+0x34/0x70
[71286.073738] ? finish_task_switch+0xf6/0x370
[71286.073741] ? __switch_to+0x2ec/0x5e0
[71286.073742] ? __schedule+0x562/0xac0
[71286.073745] ? read_word_at_a_time+0xe/0x20
[71286.073747] ? strscpy+0xca/0x1d0
[71286.073753] process_one_work+0x373/0x6e0
[71286.073756] worker_thread+0x78/0x5b0
[71286.073759] ? rescuer_thread+0x5e0/0x5e0
[71286.073760] kthread+0x192/0x1e0
[71286.073762] ? kthread_create_worker_on_cpu+0xc0/0xc0
[71286.073764] ret_from_fork+0x35/0x40
[71308.590706] fs/cifs/smb2pdu.c: In echo request
[71308.590730] fs/cifs/smb2pdu.c: Echo request failed: -11
[71308.590743] fs/cifs/connect.c: Unable to send echo request to server: DC02
[71320.544886] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13367 with uid: 999
[71320.544989] fs/cifs/transport.c: Sending smb: smb_len=372
[71320.545462] fs/cifs/connect.c: RFC1002 header 0x1b8
[71320.545477] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71320.545479] fs/cifs/smb2misc.c: SMB2 len 208
[71320.545495] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71320.545496] fs/cifs/smb2misc.c: SMB2 len 104
[71320.545512] fs/cifs/smb2misc.c: SMB2 len 124
[71320.545515] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4084
[71320.545517] fs/cifs/smb2ops.c: add 3 credits total=512
[71320.545533] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4082 state=4
[71320.545539] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4083 state=4
[71320.545542] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4084 state=4
[71320.545548] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71320.545555] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71320.545559] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71320.545620] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13367) rc = 0
[71320.548354] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13368 with uid: 999
[71320.548525] fs/cifs/transport.c: Sending smb: smb_len=348
[71320.548816] fs/cifs/connect.c: RFC1002 header 0x1b8
[71320.548826] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71320.548828] fs/cifs/smb2misc.c: SMB2 len 208
[71320.548844] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71320.548846] fs/cifs/smb2misc.c: SMB2 len 104
[71320.548864] fs/cifs/smb2misc.c: SMB2 len 124
[71320.548867] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 92
[71320.548870] fs/cifs/smb2ops.c: add 30 credits total=1059
[71320.548887] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=90 state=4
[71320.548890] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=91 state=4
[71320.548891] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=92 state=4
[71320.548893] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71320.548900] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71320.548904] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71320.548937] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13368) rc = 0
[71370.021141] fs/cifs/smb2pdu.c: In echo request
[71370.021164] fs/cifs/smb2pdu.c: Echo request failed: -11
[71370.021172] fs/cifs/connect.c: Unable to send echo request to server: DC02
[71380.065094] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13369 with uid: 999
[71380.065186] fs/cifs/transport.c: Sending smb: smb_len=372
[71380.065747] fs/cifs/connect.c: RFC1002 header 0x1b8
[71380.065757] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71380.065758] fs/cifs/smb2misc.c: SMB2 len 208
[71380.065771] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71380.065773] fs/cifs/smb2misc.c: SMB2 len 104
[71380.065786] fs/cifs/smb2misc.c: SMB2 len 124
[71380.065789] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 4087
[71380.065792] fs/cifs/smb2ops.c: add 3 credits total=512
[71380.065826] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=4085 state=4
[71380.065831] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=4086 state=4
[71380.065834] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=4087 state=4
[71380.065837] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71380.065843] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71380.065848] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71380.065881] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13369) rc = 0
[71380.065908] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13370 with uid: 999
[71380.065966] fs/cifs/transport.c: Sending smb: smb_len=348
[71380.066494] fs/cifs/connect.c: RFC1002 header 0x1b8
[71380.066501] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
[71380.066503] fs/cifs/smb2misc.c: SMB2 len 208
[71380.066516] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
[71380.066517] fs/cifs/smb2misc.c: SMB2 len 104
[71380.066532] fs/cifs/smb2misc.c: SMB2 len 124
[71380.066534] fs/cifs/smb2misc.c: Calculated size 124 length 128 mismatch mid 95
[71380.066536] fs/cifs/smb2ops.c: add 30 credits total=1086
[71380.067649] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=93 state=4
[71380.067652] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=94 state=4
[71380.067656] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=95 state=4
[71380.067658] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71380.067667] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71380.067672] fs/cifs/misc.c: Null buffer passed to cifs_small_buf_release
[71380.067707] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid = 13370) rc = 0
[71406.881866] INFO: task node_exporter:7214 blocked for more than 483 seconds.
[71406.883327] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71406.884700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71406.886064] node_exporter D 0 7214 1 0x00000000
[71406.886068] Call Trace:
[71406.886075] ? __schedule+0x540/0xac0
[71406.886080] ? firmware_map_remove+0xe9/0xe9
[71406.886087] ? vsnprintf+0x32c/0x870
[71406.886091] ? _raw_spin_lock+0x7a/0xd0
[71406.886095] schedule+0x5e/0x100
[71406.886098] schedule_preempt_disabled+0xa/0x10
[71406.886104] __mutex_lock.isra.4+0x484/0x820
[71406.886106] ? mutex_trylock+0x90/0x90
[71406.886108] ? string_nocheck+0xb0/0xd0
[71406.886111] ? pointer+0x387/0x460
[71406.886113] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71406.886115] ? _raw_write_lock_bh+0xe0/0xe0
[71406.886121] ? mutex_lock+0xce/0xe0
[71406.886122] mutex_lock+0xce/0xe0
[71406.886124] ? __mutex_lock_slowpath+0x10/0x10
[71406.886126] ? find_nls+0x7d/0xa0
[71406.886194] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
[71406.886206] ? deref_stack_reg+0x88/0xd0
[71406.886210] ? 0xffffffffa1e00000
[71406.886254] ? SMB2_tcon+0xab0/0xab0 [cifs]
[71406.886258] ? unwind_next_frame+0x90a/0x980
[71406.886260] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71406.886263] ? __module_text_address+0x11/0xa0
[71406.886265] ? __is_insn_slot_addr+0x1b/0x70
[71406.886272] ? is_bpf_text_address+0xa/0x20
[71406.886274] ? kernel_text_address+0xe2/0xf0
[71406.886275] ? __kernel_text_address+0xe/0x30
[71406.886278] ? unwind_get_return_address+0x2f/0x50
[71406.886280] ? create_prof_cpu_mask+0x20/0x20
[71406.886288] ? arch_stack_walk+0x92/0xe0
[71406.886291] ? stack_trace_save+0x8a/0xb0
[71406.886337] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
[71406.886381] SMB2_open_init+0x143/0x12b0 [cifs]
[71406.886420] ? cifs_statfs+0x13a/0x420 [cifs]
[71406.886424] ? statfs_by_dentry+0xa5/0xf0
[71406.886426] ? vfs_statfs+0x28/0x110
[71406.886428] ? __do_sys_statfs+0x64/0xc0
[71406.886470] ? smb2_parse_contexts+0x270/0x270 [cifs]
[71406.886475] ? _raw_write_trylock+0xe0/0xe0
[71406.886476] ? _raw_spin_lock+0x7a/0xd0
[71406.886478] ? _raw_write_trylock+0xe0/0xe0
[71406.886480] ? memset+0x1f/0x40
[71406.886482] ? stack_access_ok+0x35/0x90
[71406.886484] ? deref_stack_reg+0x88/0xd0
[71406.886486] ? stack_access_ok+0x35/0x90
[71406.886488] ? deref_stack_reg+0x88/0xd0
[71406.886491] ? __read_once_size_nocheck.constprop.6+0x10/0x10
[71406.886493] ? put_dec_trunc8+0x73/0x110
[71406.886495] ? number+0x356/0x4b0
[71406.886497] ? widen_string+0x23/0xf0
[71406.886538] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
[71406.886580] smb2_query_info_compound+0x242/0x4d0 [cifs]
[71406.886622] ? smb2_query_symlink+0xc70/0xc70 [cifs]
[71406.886624] ? pointer+0x460/0x460
[71406.886626] ? kernel_text_address+0xe2/0xf0
[71406.886628] ? va_format.isra.12+0xee/0x100
[71406.886630] ? vsnprintf+0x870/0x870
[71406.886632] ? string_nocheck+0xb0/0xd0
[71406.886634] ? pointer+0x387/0x460
[71406.886636] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71406.886638] ? _raw_write_lock_bh+0xe0/0xe0
[71406.886640] ? _raw_spin_lock+0x7a/0xd0
[71406.886641] ? _raw_write_trylock+0xe0/0xe0
[71406.886644] ? up+0x32/0x70
[71406.886647] ? __switch_to_asm+0x34/0x70
[71406.886649] ? dynamic_emit_prefix+0x29/0x220
[71406.886691] ? smb2_queryfs+0xd9/0x1c0 [cifs]
[71406.886744] smb2_queryfs+0xd9/0x1c0 [cifs]
[71406.886789] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
[71406.886792] ? vfs_mknod+0xc0/0x320
[71406.886795] ? map_id_up+0x12f/0x1d0
[71406.886797] ? make_kprojid+0x20/0x20
[71406.886799] ? _raw_spin_lock+0x7a/0xd0
[71406.886836] cifs_statfs+0x13a/0x420 [cifs]
[71406.886839] statfs_by_dentry+0xa5/0xf0
[71406.886842] vfs_statfs+0x28/0x110
[71406.886844] user_statfs+0x91/0xf0
[71406.886846] ? vfs_statfs+0x110/0x110
[71406.886848] ? __schedule+0x562/0xac0
[71406.886850] __do_sys_statfs+0x64/0xc0
[71406.886852] ? user_statfs+0xf0/0xf0
[71406.886856] do_syscall_64+0x73/0x190
[71406.886858] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[71406.886860] RIP: 0033:0x4a5c20
[71406.886864] Code: Bad RIP value.
[71406.886865] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX: 0000000000000089
[71406.886867] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX: 00000000004a5c20
[71406.886868] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI: 000000c0001c43c0
[71406.886869] RBP: 000000c0003414f0 R08: 0000000000000000 R09: 0000000000000000
[71406.886871] R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
[71406.886871] R13: 000000000000001f R14: 000000000000001e R15: 0000000000000100
[71406.886897] INFO: task cifsd:9136 blocked for more than 483 seconds.
[71406.888219] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71406.889666] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71406.891125] cifsd D 0 9136 2 0x80004000
[71406.891130] Call Trace:
[71406.891138] ? __schedule+0x540/0xac0
[71406.891147] ? firmware_map_remove+0xe9/0xe9
[71406.891156] ? _raw_spin_lock+0x7a/0xd0
[71406.891161] schedule+0x5e/0x100
[71406.891168] schedule_preempt_disabled+0xa/0x10
[71406.891173] __mutex_lock.isra.4+0x484/0x820
[71406.891179] ? mutex_trylock+0x90/0x90
[71406.891192] ? irq_work_claim+0x2e/0x50
[71406.891207] ? vprintk_emit+0x11d/0x2e0
[71406.891216] ? mutex_lock+0xce/0xe0
[71406.891228] mutex_lock+0xce/0xe0
[71406.891232] ? __mutex_lock_slowpath+0x10/0x10
[71406.891317] dfs_cache_noreq_find+0xa7/0x190 [cifs]
[71406.891378] cifs_reconnect+0x18c/0x1510 [cifs]
[71406.891383] ? ___ratelimit+0x106/0x190
[71406.891439] ? smb2_calc_size+0x15c/0x250 [cifs]
[71406.891505] ? extract_hostname+0xa0/0xa0 [cifs]
[71406.891510] ? _raw_spin_trylock+0x91/0xe0
[71406.891513] ? _raw_spin_trylock_bh+0x100/0x100
[71406.891516] ? ___ratelimit+0x106/0x190
[71406.891574] cifs_handle_standard+0x252/0x270 [cifs]
[71406.891631] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
[71406.891692] ? cifs_handle_standard+0x270/0x270 [cifs]
[71406.891696] ? __switch_to_asm+0x40/0x70
[71406.891698] ? __switch_to_asm+0x34/0x70
[71406.891701] ? __switch_to_asm+0x40/0x70
[71406.891703] ? __switch_to_asm+0x34/0x70
[71406.891706] ? __switch_to_asm+0x40/0x70
[71406.891709] ? __switch_to_asm+0x34/0x70
[71406.891712] ? __switch_to_asm+0x40/0x70
[71406.891715] ? __switch_to_asm+0x34/0x70
[71406.891718] ? __switch_to_asm+0x40/0x70
[71406.891720] ? __switch_to_asm+0x34/0x70
[71406.891723] ? __switch_to_asm+0x40/0x70
[71406.891725] ? __switch_to_asm+0x34/0x70
[71406.891727] ? __switch_to_asm+0x40/0x70
[71406.891729] ? __switch_to_asm+0x34/0x70
[71406.891732] ? __switch_to_asm+0x40/0x70
[71406.891734] ? __switch_to_asm+0x34/0x70
[71406.891737] ? __switch_to_asm+0x40/0x70
[71406.891739] ? __switch_to_asm+0x34/0x70
[71406.891741] ? __switch_to_asm+0x40/0x70
[71406.891744] ? __switch_to_asm+0x34/0x70
[71406.891746] ? __switch_to_asm+0x40/0x70
[71406.891749] ? __switch_to_asm+0x34/0x70
[71406.891751] ? __switch_to_asm+0x40/0x70
[71406.891753] ? __switch_to_asm+0x34/0x70
[71406.891756] ? __switch_to_asm+0x40/0x70
[71406.891758] ? __switch_to_asm+0x34/0x70
[71406.891763] ? finish_task_switch+0xf6/0x370
[71406.891767] ? __switch_to+0x2ec/0x5e0
[71406.891772] ? _raw_spin_lock_irqsave+0x8d/0xf0
[71406.891774] ? _raw_write_lock_bh+0xe0/0xe0
[71406.891946] ? cifs_handle_standard+0x270/0x270 [cifs]
[71406.891951] kthread+0x192/0x1e0
[71406.891955] ? kthread_create_worker_on_cpu+0xc0/0xc0
[71406.891958] ret_from_fork+0x35/0x40
[71406.891970] INFO: task kworker/0:0:16069 blocked for more than 241 seconds.
[71406.894262] Tainted: G E 5.3.11-pd-5.3.y #20191029
[71406.896328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[71406.898113] kworker/0:0 D 0 16069 2 0x80004000
[71406.898203] Workqueue: cifsiod refresh_cache_worker [cifs]
[71406.898211] Call Trace:
[71406.898220] ? __schedule+0x540/0xac0
[71406.898224] ? firmware_map_remove+0xe9/0xe9
[71406.898228] ? _raw_read_lock_irq+0x40/0x40
[71406.898231] ? _raw_spin_lock+0x7a/0xd0
[71406.898234] schedule+0x5e/0x100
[71406.898239] schedule_preempt_disabled+0xa/0x10
[71406.898242] __mutex_lock.isra.4+0x484/0x820
[71406.898247] ? mutex_trylock+0x90/0x90
[71406.898251] ? dynamic_emit_prefix+0x29/0x220
[71406.898255] ? __dynamic_pr_debug+0xf8/0x140
[71406.898260] ? dynamic_emit_prefix+0x220/0x220
[71406.898271] ? update_dl_rq_load_avg+0x2c3/0x4d0
[71406.898277] ? mutex_lock+0xce/0xe0
[71406.898280] mutex_lock+0xce/0xe0
[71406.898283] ? __mutex_lock_slowpath+0x10/0x10
[71406.898362] refresh_cache_worker+0x48f/0x14a0 [cifs]
[71406.898370] ? __switch_to_asm+0x40/0x70
[71406.898373] ? __switch_to_asm+0x40/0x70
[71406.898376] ? __switch_to_asm+0x34/0x70
[71406.898378] ? __switch_to_asm+0x40/0x70
[71406.898383] ? __switch_to_asm+0x34/0x70
[71406.898389] ? __switch_to_asm+0x40/0x70
[71406.898467] ? find_root_ses.isra.9+0x320/0x320 [cifs]
[71406.898472] ? __switch_to_asm+0x40/0x70
[71406.898476] ? __switch_to_asm+0x40/0x70
[71406.898479] ? __switch_to_asm+0x34/0x70
[71406.898484] ? __switch_to_asm+0x40/0x70
[71406.898488] ? __switch_to_asm+0x34/0x70
[71406.898490] ? __switch_to_asm+0x40/0x70
[71406.898499] ? __switch_to_asm+0x34/0x70
[71406.898502] ? __switch_to_asm+0x40/0x70
[71406.898505] ? __switch_to_asm+0x40/0x70
[71406.898508] ? __switch_to_asm+0x34/0x70
[71406.898516] ? finish_task_switch+0xf6/0x370
[71406.898519] ? __switch_to+0x2ec/0x5e0
[71406.898523] ? __schedule+0x562/0xac0
[71406.898530] ? read_word_at_a_time+0xe/0x20
[71406.898533] ? strscpy+0xca/0x1d0
[71406.898539] process_one_work+0x373/0x6e0
[71406.898547] worker_thread+0x78/0x5b0
[71406.898552] ? rescuer_thread+0x5e0/0x5e0
[71406.898555] kthread+0x192/0x1e0
[71406.898559] ? kthread_create_worker_on_cpu+0xc0/0xc0
[71406.898563] ret_from_fork+0x35/0x40
[71417.962152] fs/cifs/inode.c: CIFS VFS: in cifs_revalidate_dentry_attr as Xid: 13371 with uid: 11025
[71417.962171] fs/cifs/dir.c: name: \SOMEDIRX
[71417.962177] fs/cifs/inode.c: Update attributes: \SOMEDIRX inode 0x000000009f726d03 count 1 dentry: 0x00000000d22e461a d_time 4312598279 jiffies 4312748242
[71417.962180] fs/cifs/inode.c: Getting info on \SOMEDIRX
[71417.962367] fs/cifs/transport.c: Sending smb: smb_len=388
Gr, Martijn
--
Martijn de Gouw
Designer
Prodrive Technologies
Mobile: +31 63 17 76 161
Phone: +31 40 26 76 200
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Kernel hangs in cifs_reconnect
2019-10-30 19:17 ` Martijn de Gouw
@ 2019-10-30 19:33 ` Paulo Alcantara
0 siblings, 0 replies; 9+ messages in thread
From: Paulo Alcantara @ 2019-10-30 19:33 UTC (permalink / raw)
To: Martijn de Gouw, Aurélien Aptel, linux-cifs
On October 30, 2019 7:17:25 PM UTC, Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> wrote:
>Hi Paulo,
>
>On 29-10-2019 15:49, Paulo Alcantara wrote:
>> Hi Martijn,
>>
>> Martijn de Gouw <martijn.de.gouw@prodrive-technologies.com> writes:
>>
>>> Anybody any idea on what goes wrong here?
>>
>> Looks like an use-after-free bug in cifs_reconnect(). cifs superblock
>> gets freed due to automount expiration and then we dereference it in
>> dfs_cache_noreq_find().
>>
>>> Is any of the recently posted patches related to my issue, because
>I'm
>>> more that willing to test out patches if needed.
>>
>> Could you please test it again with below patch?
>
>Again cifs stopped working, it looks more like a live/dead lock to me
>than a use-after-free.
>I cut the dmesg from the moment the echo to DC02 starts failing.
>
>[70939.980125] fs/cifs/smb2pdu.c: In echo request
>[70939.980137] fs/cifs/smb2pdu.c: Echo request failed: -11
>[70939.980143] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[70960.106599] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13353
>with uid: 999
>[70960.106697] fs/cifs/transport.c: Sending smb: smb_len=372
>[70960.116162] fs/cifs/connect.c: RFC1002 header 0x1b8
>[70960.116170] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[70960.116171] fs/cifs/smb2misc.c: SMB2 len 208
>[70960.116185] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[70960.116186] fs/cifs/smb2misc.c: SMB2 len 104
>[70960.116196] fs/cifs/smb2misc.c: SMB2 len 124
>[70960.116197] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4066
>[70960.116200] fs/cifs/smb2ops.c: add 3 credits total=512
>[70960.116218] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4064 state=4
>[70960.116227] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4065 state=4
>[70960.116230] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4066 state=4
>[70960.116234] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[70960.116242] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[70960.116247] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[70960.116290] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13353) rc = 0
>[70960.116319] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13354
>with uid: 999
>[70960.116429] fs/cifs/transport.c: Sending smb: smb_len=348
>[70960.116829] fs/cifs/connect.c: RFC1002 header 0x1b8
>[70960.116834] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[70960.116835] fs/cifs/smb2misc.c: SMB2 len 208
>[70960.116842] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[70960.116843] fs/cifs/smb2misc.c: SMB2 len 104
>[70960.116856] fs/cifs/smb2misc.c: SMB2 len 124
>[70960.116857] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 74
>[70960.116859] fs/cifs/smb2ops.c: add 30 credits total=897
>[70960.116872] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=72
>state=4
>[70960.116874] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=73
>state=4
>[70960.116876] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=74
>state=4
>[70960.116877] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[70960.116882] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[70960.116887] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[70960.116954] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13354) rc = 0
>[71001.418549] fs/cifs/smb2pdu.c: In echo request
>[71001.418565] fs/cifs/smb2pdu.c: Echo request failed: -11
>[71001.418570] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[71020.097059] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13355
>with uid: 999
>[71020.097165] fs/cifs/transport.c: Sending smb: smb_len=372
>[71020.097760] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71020.097767] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71020.097768] fs/cifs/smb2misc.c: SMB2 len 208
>[71020.097777] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71020.097778] fs/cifs/smb2misc.c: SMB2 len 104
>[71020.097788] fs/cifs/smb2misc.c: SMB2 len 124
>[71020.097789] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4069
>[71020.097796] fs/cifs/smb2ops.c: add 3 credits total=512
>[71020.098387] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4067 state=4
>[71020.098395] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4068 state=4
>[71020.098398] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4069 state=4
>[71020.098405] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71020.098414] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71020.098419] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71020.098461] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13355) rc = 0
>[71020.098488] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13356
>with uid: 999
>[71020.098556] fs/cifs/transport.c: Sending smb: smb_len=348
>[71020.098915] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71020.098921] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71020.098922] fs/cifs/smb2misc.c: SMB2 len 208
>[71020.098932] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71020.098932] fs/cifs/smb2misc.c: SMB2 len 104
>[71020.098942] fs/cifs/smb2misc.c: SMB2 len 124
>[71020.098944] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 77
>[71020.098946] fs/cifs/smb2ops.c: add 30 credits total=924
>[71020.098956] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=75
>state=4
>[71020.098958] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=76
>state=4
>[71020.098959] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=77
>state=4
>[71020.098961] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71020.098968] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71020.098973] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71020.099011] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13356) rc = 0
>[71044.418723] INFO: task node_exporter:7214 blocked for more than 120
>seconds.
>[71044.420037] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71044.421491] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71044.422762] node_exporter D 0 7214 1 0x00000000
>[71044.422781] Call Trace:
>[71044.422806] ? __schedule+0x540/0xac0
>[71044.422813] ? firmware_map_remove+0xe9/0xe9
>[71044.422821] ? vsnprintf+0x32c/0x870
>[71044.422826] ? _raw_spin_lock+0x7a/0xd0
>[71044.422829] schedule+0x5e/0x100
>[71044.422837] schedule_preempt_disabled+0xa/0x10
>[71044.422840] __mutex_lock.isra.4+0x484/0x820
>[71044.422844] ? mutex_trylock+0x90/0x90
>[71044.422848] ? string_nocheck+0xb0/0xd0
>[71044.422854] ? pointer+0x387/0x460
>[71044.422859] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71044.422863] ? _raw_write_lock_bh+0xe0/0xe0
>[71044.422868] ? mutex_lock+0xce/0xe0
>[71044.422871] mutex_lock+0xce/0xe0
>[71044.422875] ? __mutex_lock_slowpath+0x10/0x10
>[71044.422892] ? find_nls+0x7d/0xa0
>[71044.422976] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
>[71044.422992] ? deref_stack_reg+0x88/0xd0
>[71044.422995] ? 0xffffffffa1e00000
>[71044.423065] ? SMB2_tcon+0xab0/0xab0 [cifs]
>[71044.423067] ? unwind_next_frame+0x90a/0x980
>[71044.423070] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71044.423081] ? __module_text_address+0x11/0xa0
>[71044.423087] ? __is_insn_slot_addr+0x1b/0x70
>[71044.423094] ? is_bpf_text_address+0xa/0x20
>[71044.423101] ? kernel_text_address+0xe2/0xf0
>[71044.423104] ? __kernel_text_address+0xe/0x30
>[71044.423108] ? unwind_get_return_address+0x2f/0x50
>[71044.423114] ? create_prof_cpu_mask+0x20/0x20
>[71044.423120] ? arch_stack_walk+0x92/0xe0
>[71044.423125] ? stack_trace_save+0x8a/0xb0
>[71044.423195] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
>[71044.423272] SMB2_open_init+0x143/0x12b0 [cifs]
>[71044.423339] ? cifs_statfs+0x13a/0x420 [cifs]
>[71044.423352] ? statfs_by_dentry+0xa5/0xf0
>[71044.423354] ? vfs_statfs+0x28/0x110
>[71044.423357] ? __do_sys_statfs+0x64/0xc0
>[71044.423413] ? smb2_parse_contexts+0x270/0x270 [cifs]
>[71044.423416] ? _raw_write_trylock+0xe0/0xe0
>[71044.423419] ? _raw_spin_lock+0x7a/0xd0
>[71044.423421] ? _raw_write_trylock+0xe0/0xe0
>[71044.423429] ? memset+0x1f/0x40
>[71044.423433] ? stack_access_ok+0x35/0x90
>[71044.423437] ? deref_stack_reg+0x88/0xd0
>[71044.423439] ? stack_access_ok+0x35/0x90
>[71044.423442] ? deref_stack_reg+0x88/0xd0
>[71044.423446] ? __read_once_size_nocheck.constprop.6+0x10/0x10
>[71044.423450] ? put_dec_trunc8+0x73/0x110
>[71044.423453] ? number+0x356/0x4b0
>[71044.423456] ? widen_string+0x23/0xf0
>[71044.423517] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71044.423558] smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71044.423601] ? smb2_query_symlink+0xc70/0xc70 [cifs]
>[71044.423604] ? pointer+0x460/0x460
>[71044.423606] ? kernel_text_address+0xe2/0xf0
>[71044.423608] ? va_format.isra.12+0xee/0x100
>[71044.423612] ? vsnprintf+0x870/0x870
>[71044.423614] ? string_nocheck+0xb0/0xd0
>[71044.423616] ? pointer+0x387/0x460
>[71044.423619] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71044.423620] ? _raw_write_lock_bh+0xe0/0xe0
>[71044.423622] ? _raw_spin_lock+0x7a/0xd0
>[71044.423624] ? _raw_write_trylock+0xe0/0xe0
>[71044.423626] ? up+0x32/0x70
>[71044.423630] ? __switch_to_asm+0x34/0x70
>[71044.423636] ? dynamic_emit_prefix+0x29/0x220
>[71044.423684] ? smb2_queryfs+0xd9/0x1c0 [cifs]
>[71044.423735] smb2_queryfs+0xd9/0x1c0 [cifs]
>[71044.423776] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
>[71044.423780] ? vfs_mknod+0xc0/0x320
>[71044.423785] ? map_id_up+0x12f/0x1d0
>[71044.423787] ? make_kprojid+0x20/0x20
>[71044.423789] ? _raw_spin_lock+0x7a/0xd0
>[71044.423837] cifs_statfs+0x13a/0x420 [cifs]
>[71044.423844] statfs_by_dentry+0xa5/0xf0
>[71044.423848] vfs_statfs+0x28/0x110
>[71044.423851] user_statfs+0x91/0xf0
>[71044.423853] ? vfs_statfs+0x110/0x110
>[71044.423855] ? __schedule+0x562/0xac0
>[71044.423857] __do_sys_statfs+0x64/0xc0
>[71044.423859] ? user_statfs+0xf0/0xf0
>[71044.423865] do_syscall_64+0x73/0x190
>[71044.423868] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71044.423875] RIP: 0033:0x4a5c20
>[71044.423880] Code: Bad RIP value.
>[71044.423882] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX:
>0000000000000089
>[71044.423888] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX:
>00000000004a5c20
>[71044.423890] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI:
>000000c0001c43c0
>[71044.423892] RBP: 000000c0003414f0 R08: 0000000000000000 R09:
>0000000000000000
>[71044.423893] R10: 0000000000000000 R11: 0000000000000206 R12:
>ffffffffffffffff
>[71044.423894] R13: 000000000000001f R14: 000000000000001e R15:
>0000000000000100
>[71044.423926] INFO: task cifsd:9136 blocked for more than 120 seconds.
>[71044.425076] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71044.426421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71044.427718] cifsd D 0 9136 2 0x80004000
>[71044.427721] Call Trace:
>[71044.427738] ? __schedule+0x540/0xac0
>[71044.427745] ? firmware_map_remove+0xe9/0xe9
>[71044.427752] ? _raw_spin_lock+0x7a/0xd0
>[71044.427757] schedule+0x5e/0x100
>[71044.427765] schedule_preempt_disabled+0xa/0x10
>[71044.427775] __mutex_lock.isra.4+0x484/0x820
>[71044.427781] ? mutex_trylock+0x90/0x90
>[71044.427787] ? irq_work_claim+0x2e/0x50
>[71044.427792] ? vprintk_emit+0x11d/0x2e0
>[71044.427795] ? mutex_lock+0xce/0xe0
>[71044.427797] mutex_lock+0xce/0xe0
>[71044.427800] ? __mutex_lock_slowpath+0x10/0x10
>[71044.427877] dfs_cache_noreq_find+0xa7/0x190 [cifs]
>[71044.427951] cifs_reconnect+0x18c/0x1510 [cifs]
>[71044.427970] ? ___ratelimit+0x106/0x190
>[71044.428050] ? smb2_calc_size+0x15c/0x250 [cifs]
>[71044.428107] ? extract_hostname+0xa0/0xa0 [cifs]
>[71044.428121] ? _raw_spin_trylock+0x91/0xe0
>[71044.428129] ? _raw_spin_trylock_bh+0x100/0x100
>[71044.428134] ? ___ratelimit+0x106/0x190
>[71044.428179] cifs_handle_standard+0x252/0x270 [cifs]
>[71044.428224] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
>[71044.428267] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71044.428272] ? __switch_to_asm+0x40/0x70
>[71044.428276] ? __switch_to_asm+0x34/0x70
>[71044.428278] ? __switch_to_asm+0x40/0x70
>[71044.428280] ? __switch_to_asm+0x34/0x70
>[71044.428281] ? __switch_to_asm+0x40/0x70
>[71044.428283] ? __switch_to_asm+0x34/0x70
>[71044.428284] ? __switch_to_asm+0x40/0x70
>[71044.428286] ? __switch_to_asm+0x34/0x70
>[71044.428287] ? __switch_to_asm+0x40/0x70
>[71044.428289] ? __switch_to_asm+0x34/0x70
>[71044.428290] ? __switch_to_asm+0x40/0x70
>[71044.428292] ? __switch_to_asm+0x34/0x70
>[71044.428293] ? __switch_to_asm+0x40/0x70
>[71044.428295] ? __switch_to_asm+0x34/0x70
>[71044.428296] ? __switch_to_asm+0x40/0x70
>[71044.428298] ? __switch_to_asm+0x34/0x70
>[71044.428299] ? __switch_to_asm+0x40/0x70
>[71044.428301] ? __switch_to_asm+0x34/0x70
>[71044.428303] ? __switch_to_asm+0x40/0x70
>[71044.428304] ? __switch_to_asm+0x34/0x70
>[71044.428306] ? __switch_to_asm+0x40/0x70
>[71044.428307] ? __switch_to_asm+0x34/0x70
>[71044.428309] ? __switch_to_asm+0x40/0x70
>[71044.428310] ? __switch_to_asm+0x34/0x70
>[71044.428312] ? __switch_to_asm+0x40/0x70
>[71044.428313] ? __switch_to_asm+0x34/0x70
>[71044.428317] ? finish_task_switch+0xf6/0x370
>[71044.428319] ? __switch_to+0x2ec/0x5e0
>[71044.428321] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71044.428323] ? _raw_write_lock_bh+0xe0/0xe0
>[71044.428375] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71044.428378] kthread+0x192/0x1e0
>[71044.428381] ? kthread_create_worker_on_cpu+0xc0/0xc0
>[71044.428384] ret_from_fork+0x35/0x40
>[71062.848999] fs/cifs/smb2pdu.c: In echo request
>[71062.849014] fs/cifs/smb2pdu.c: Echo request failed: -11
>[71062.849025] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[71080.092803] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13357
>with uid: 999
>[71080.092915] fs/cifs/transport.c: Sending smb: smb_len=372
>[71080.097568] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71080.097580] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71080.097581] fs/cifs/smb2misc.c: SMB2 len 208
>[71080.097591] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71080.097592] fs/cifs/smb2misc.c: SMB2 len 104
>[71080.097601] fs/cifs/smb2misc.c: SMB2 len 124
>[71080.097603] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4072
>[71080.097605] fs/cifs/smb2ops.c: add 3 credits total=512
>[71080.097741] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4070 state=4
>[71080.097749] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4071 state=4
>[71080.097752] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4072 state=4
>[71080.097758] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71080.097767] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71080.097771] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71080.097812] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13357) rc = 0
>[71080.097992] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13358
>with uid: 999
>[71080.098070] fs/cifs/transport.c: Sending smb: smb_len=348
>[71080.098487] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71080.098492] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71080.098493] fs/cifs/smb2misc.c: SMB2 len 208
>[71080.098500] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71080.098501] fs/cifs/smb2misc.c: SMB2 len 104
>[71080.098510] fs/cifs/smb2misc.c: SMB2 len 124
>[71080.098512] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 80
>[71080.098514] fs/cifs/smb2ops.c: add 30 credits total=951
>[71080.098609] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=78
>state=4
>[71080.098610] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=79
>state=4
>[71080.098612] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=80
>state=4
>[71080.098613] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71080.098619] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71080.098624] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71080.098655] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13358) rc = 0
>[71105.853107] fs/cifs/connect.c: Existing tcp session with server
>found
>[71105.853114] fs/cifs/dfs_cache.c: CIFS VFS: in do_refresh_tcon as
>Xid: 13359 with uid: 0
>[71116.587766] fs/cifs/inode.c: CIFS VFS: in
>cifs_revalidate_dentry_attr as Xid: 13360 with uid: 11025
>[71116.587834] fs/cifs/dir.c: name: \SOMEDIRX
>[71116.587842] fs/cifs/inode.c: Update attributes: \SOMEDIRX inode
>0x000000009f726d03 count 1 dentry: 0x00000000d22e461a d_time 4312598279
>jiffies 4312672891
>[71116.587847] fs/cifs/inode.c: Getting info on \SOMEDIRX
>[71116.588001] fs/cifs/transport.c: Sending smb: smb_len=388
>[71124.283416] fs/cifs/smb2pdu.c: In echo request
>[71124.283434] fs/cifs/smb2pdu.c: Echo request failed: -11
>[71124.283442] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[71140.087648] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13361
>with uid: 999
>[71140.087743] fs/cifs/transport.c: Sending smb: smb_len=372
>[71140.088088] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71140.088095] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71140.088096] fs/cifs/smb2misc.c: SMB2 len 208
>[71140.088106] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71140.088107] fs/cifs/smb2misc.c: SMB2 len 104
>[71140.088116] fs/cifs/smb2misc.c: SMB2 len 124
>[71140.088118] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4075
>[71140.088120] fs/cifs/smb2ops.c: add 3 credits total=512
>[71140.088137] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4073 state=4
>[71140.088141] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4074 state=4
>[71140.088144] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4075 state=4
>[71140.088147] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71140.088153] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71140.088158] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71140.088190] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13361) rc = 0
>[71140.088218] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13362
>with uid: 999
>[71140.088304] fs/cifs/transport.c: Sending smb: smb_len=348
>[71140.088643] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71140.088647] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71140.088648] fs/cifs/smb2misc.c: SMB2 len 208
>[71140.088655] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71140.088656] fs/cifs/smb2misc.c: SMB2 len 104
>[71140.088664] fs/cifs/smb2misc.c: SMB2 len 124
>[71140.088665] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 83
>[71140.088667] fs/cifs/smb2ops.c: add 30 credits total=978
>[71140.088680] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=81
>state=4
>[71140.088682] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=82
>state=4
>[71140.088684] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=83
>state=4
>[71140.088686] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71140.088692] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71140.088696] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71140.088728] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13362) rc = 0
>[71165.239724] INFO: task node_exporter:7214 blocked for more than 241
>seconds.
>[71165.240866] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71165.241935] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71165.243018] node_exporter D 0 7214 1 0x00000000
>[71165.243022] Call Trace:
>[71165.243028] ? __schedule+0x540/0xac0
>[71165.243031] ? firmware_map_remove+0xe9/0xe9
>[71165.243035] ? vsnprintf+0x32c/0x870
>[71165.243038] ? _raw_spin_lock+0x7a/0xd0
>[71165.243040] schedule+0x5e/0x100
>[71165.243042] schedule_preempt_disabled+0xa/0x10
>[71165.243044] __mutex_lock.isra.4+0x484/0x820
>[71165.243047] ? mutex_trylock+0x90/0x90
>[71165.243049] ? string_nocheck+0xb0/0xd0
>[71165.243051] ? pointer+0x387/0x460
>[71165.243054] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71165.243056] ? _raw_write_lock_bh+0xe0/0xe0
>[71165.243059] ? mutex_lock+0xce/0xe0
>[71165.243060] mutex_lock+0xce/0xe0
>[71165.243062] ? __mutex_lock_slowpath+0x10/0x10
>[71165.243065] ? find_nls+0x7d/0xa0
>[71165.243133] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
>[71165.243137] ? deref_stack_reg+0x88/0xd0
>[71165.243139] ? 0xffffffffa1e00000
>[71165.243180] ? SMB2_tcon+0xab0/0xab0 [cifs]
>[71165.243181] ? unwind_next_frame+0x90a/0x980
>[71165.243183] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71165.243187] ? __module_text_address+0x11/0xa0
>[71165.243189] ? __is_insn_slot_addr+0x1b/0x70
>[71165.243193] ? is_bpf_text_address+0xa/0x20
>[71165.243195] ? kernel_text_address+0xe2/0xf0
>[71165.243197] ? __kernel_text_address+0xe/0x30
>[71165.243200] ? unwind_get_return_address+0x2f/0x50
>[71165.243203] ? create_prof_cpu_mask+0x20/0x20
>[71165.243207] ? arch_stack_walk+0x92/0xe0
>[71165.243210] ? stack_trace_save+0x8a/0xb0
>[71165.243251] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
>[71165.243294] SMB2_open_init+0x143/0x12b0 [cifs]
>[71165.243332] ? cifs_statfs+0x13a/0x420 [cifs]
>[71165.243336] ? statfs_by_dentry+0xa5/0xf0
>[71165.243338] ? vfs_statfs+0x28/0x110
>[71165.243340] ? __do_sys_statfs+0x64/0xc0
>[71165.243382] ? smb2_parse_contexts+0x270/0x270 [cifs]
>[71165.243384] ? _raw_write_trylock+0xe0/0xe0
>[71165.243385] ? _raw_spin_lock+0x7a/0xd0
>[71165.243387] ? _raw_write_trylock+0xe0/0xe0
>[71165.243389] ? memset+0x1f/0x40
>[71165.243391] ? stack_access_ok+0x35/0x90
>[71165.243393] ? deref_stack_reg+0x88/0xd0
>[71165.243396] ? stack_access_ok+0x35/0x90
>[71165.243399] ? deref_stack_reg+0x88/0xd0
>[71165.243402] ? __read_once_size_nocheck.constprop.6+0x10/0x10
>[71165.243404] ? put_dec_trunc8+0x73/0x110
>[71165.243406] ? number+0x356/0x4b0
>[71165.243408] ? widen_string+0x23/0xf0
>[71165.243449] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71165.243490] smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71165.243533] ? smb2_query_symlink+0xc70/0xc70 [cifs]
>[71165.243535] ? pointer+0x460/0x460
>[71165.243537] ? kernel_text_address+0xe2/0xf0
>[71165.243539] ? va_format.isra.12+0xee/0x100
>[71165.243541] ? vsnprintf+0x870/0x870
>[71165.243543] ? string_nocheck+0xb0/0xd0
>[71165.243545] ? pointer+0x387/0x460
>[71165.243547] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71165.243549] ? _raw_write_lock_bh+0xe0/0xe0
>[71165.243551] ? _raw_spin_lock+0x7a/0xd0
>[71165.243552] ? _raw_write_trylock+0xe0/0xe0
>[71165.243555] ? up+0x32/0x70
>[71165.243558] ? __switch_to_asm+0x34/0x70
>[71165.243561] ? dynamic_emit_prefix+0x29/0x220
>[71165.243602] ? smb2_queryfs+0xd9/0x1c0 [cifs]
>[71165.243644] smb2_queryfs+0xd9/0x1c0 [cifs]
>[71165.243688] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
>[71165.243764] ? vfs_mknod+0xc0/0x320
>[71165.243770] ? map_id_up+0x12f/0x1d0
>[71165.243774] ? make_kprojid+0x20/0x20
>[71165.243777] ? _raw_spin_lock+0x7a/0xd0
>[71165.243818] cifs_statfs+0x13a/0x420 [cifs]
>[71165.243825] statfs_by_dentry+0xa5/0xf0
>[71165.243829] vfs_statfs+0x28/0x110
>[71165.243835] user_statfs+0x91/0xf0
>[71165.243838] ? vfs_statfs+0x110/0x110
>[71165.243844] ? __schedule+0x562/0xac0
>[71165.243847] __do_sys_statfs+0x64/0xc0
>[71165.243854] ? user_statfs+0xf0/0xf0
>[71165.243859] do_syscall_64+0x73/0x190
>[71165.243863] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71165.243866] RIP: 0033:0x4a5c20
>[71165.243873] Code: Bad RIP value.
>[71165.243876] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX:
>0000000000000089
>[71165.243881] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX:
>00000000004a5c20
>[71165.243883] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI:
>000000c0001c43c0
>[71165.243885] RBP: 000000c0003414f0 R08: 0000000000000000 R09:
>0000000000000000
>[71165.243886] R10: 0000000000000000 R11: 0000000000000206 R12:
>ffffffffffffffff
>[71165.243887] R13: 000000000000001f R14: 000000000000001e R15:
>0000000000000100
>[71165.243905] INFO: task cifsd:9136 blocked for more than 241 seconds.
>[71165.245022] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71165.246156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71165.247272] cifsd D 0 9136 2 0x80004000
>[71165.247274] Call Trace:
>[71165.247281] ? __schedule+0x540/0xac0
>[71165.247284] ? firmware_map_remove+0xe9/0xe9
>[71165.247287] ? _raw_spin_lock+0x7a/0xd0
>[71165.247289] schedule+0x5e/0x100
>[71165.247291] schedule_preempt_disabled+0xa/0x10
>[71165.247293] __mutex_lock.isra.4+0x484/0x820
>[71165.247296] ? mutex_trylock+0x90/0x90
>[71165.247300] ? irq_work_claim+0x2e/0x50
>[71165.247303] ? vprintk_emit+0x11d/0x2e0
>[71165.247305] ? mutex_lock+0xce/0xe0
>[71165.247306] mutex_lock+0xce/0xe0
>[71165.247308] ? __mutex_lock_slowpath+0x10/0x10
>[71165.247358] dfs_cache_noreq_find+0xa7/0x190 [cifs]
>[71165.247398] cifs_reconnect+0x18c/0x1510 [cifs]
>[71165.247402] ? ___ratelimit+0x106/0x190
>[71165.247444] ? smb2_calc_size+0x15c/0x250 [cifs]
>[71165.247484] ? extract_hostname+0xa0/0xa0 [cifs]
>[71165.247486] ? _raw_spin_trylock+0x91/0xe0
>[71165.247487] ? _raw_spin_trylock_bh+0x100/0x100
>[71165.247490] ? ___ratelimit+0x106/0x190
>[71165.247533] cifs_handle_standard+0x252/0x270 [cifs]
>[71165.247575] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
>[71165.247616] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71165.247618] ? __switch_to_asm+0x40/0x70
>[71165.247620] ? __switch_to_asm+0x34/0x70
>[71165.247621] ? __switch_to_asm+0x40/0x70
>[71165.247623] ? __switch_to_asm+0x34/0x70
>[71165.247624] ? __switch_to_asm+0x40/0x70
>[71165.247626] ? __switch_to_asm+0x34/0x70
>[71165.247627] ? __switch_to_asm+0x40/0x70
>[71165.247629] ? __switch_to_asm+0x34/0x70
>[71165.247630] ? __switch_to_asm+0x40/0x70
>[71165.247632] ? __switch_to_asm+0x34/0x70
>[71165.247633] ? __switch_to_asm+0x40/0x70
>[71165.247635] ? __switch_to_asm+0x34/0x70
>[71165.247637] ? __switch_to_asm+0x40/0x70
>[71165.247638] ? __switch_to_asm+0x34/0x70
>[71165.247640] ? __switch_to_asm+0x40/0x70
>[71165.247641] ? __switch_to_asm+0x34/0x70
>[71165.247643] ? __switch_to_asm+0x40/0x70
>[71165.247644] ? __switch_to_asm+0x34/0x70
>[71165.247646] ? __switch_to_asm+0x40/0x70
>[71165.247647] ? __switch_to_asm+0x34/0x70
>[71165.247649] ? __switch_to_asm+0x40/0x70
>[71165.247651] ? __switch_to_asm+0x34/0x70
>[71165.247653] ? __switch_to_asm+0x40/0x70
>[71165.247659] ? __switch_to_asm+0x34/0x70
>[71165.247678] ? __switch_to_asm+0x40/0x70
>[71165.247680] ? __switch_to_asm+0x34/0x70
>[71165.247683] ? finish_task_switch+0xf6/0x370
>[71165.247685] ? __switch_to+0x2ec/0x5e0
>[71165.247688] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71165.247690] ? _raw_write_lock_bh+0xe0/0xe0
>[71165.247731] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71165.247735] kthread+0x192/0x1e0
>[71165.247737] ? kthread_create_worker_on_cpu+0xc0/0xc0
>[71165.247739] ret_from_fork+0x35/0x40
>[71185.717885] fs/cifs/smb2pdu.c: In echo request
>[71185.717900] fs/cifs/smb2pdu.c: Echo request failed: -11
>[71185.717906] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[71200.086057] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13363
>with uid: 999
>[71200.086149] fs/cifs/transport.c: Sending smb: smb_len=372
>[71200.086716] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71200.086725] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71200.086726] fs/cifs/smb2misc.c: SMB2 len 208
>[71200.086740] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71200.086741] fs/cifs/smb2misc.c: SMB2 len 104
>[71200.086751] fs/cifs/smb2misc.c: SMB2 len 124
>[71200.086752] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4078
>[71200.086755] fs/cifs/smb2ops.c: add 3 credits total=512
>[71200.086770] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4076 state=4
>[71200.086775] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4077 state=4
>[71200.086777] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4078 state=4
>[71200.086781] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71200.086786] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71200.086791] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71200.086824] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13363) rc = 0
>[71200.086841] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13364
>with uid: 999
>[71200.086938] fs/cifs/transport.c: Sending smb: smb_len=348
>[71200.087336] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71200.087342] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71200.087343] fs/cifs/smb2misc.c: SMB2 len 208
>[71200.087352] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71200.087353] fs/cifs/smb2misc.c: SMB2 len 104
>[71200.087362] fs/cifs/smb2misc.c: SMB2 len 124
>[71200.087364] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 86
>[71200.087366] fs/cifs/smb2ops.c: add 30 credits total=1005
>[71200.087379] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=84
>state=4
>[71200.087381] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=85
>state=4
>[71200.087383] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=86
>state=4
>[71200.087384] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71200.087391] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71200.087395] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71200.087428] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13364) rc = 0
>[71247.152290] fs/cifs/smb2pdu.c: In echo request
>[71247.152306] fs/cifs/smb2pdu.c: Echo request failed: -11
>[71247.152313] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[71260.080476] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13365
>with uid: 999
>[71260.080564] fs/cifs/transport.c: Sending smb: smb_len=372
>[71260.080923] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71260.080931] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71260.080932] fs/cifs/smb2misc.c: SMB2 len 208
>[71260.080946] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71260.080947] fs/cifs/smb2misc.c: SMB2 len 104
>[71260.080958] fs/cifs/smb2misc.c: SMB2 len 124
>[71260.080959] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4081
>[71260.080962] fs/cifs/smb2ops.c: add 3 credits total=512
>[71260.080975] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4079 state=4
>[71260.080979] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4080 state=4
>[71260.080982] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4081 state=4
>[71260.080985] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71260.080991] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71260.080996] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71260.081032] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13365) rc = 0
>[71260.081163] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13366
>with uid: 999
>[71260.081228] fs/cifs/transport.c: Sending smb: smb_len=348
>[71260.081798] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71260.081804] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71260.081805] fs/cifs/smb2misc.c: SMB2 len 208
>[71260.081814] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71260.081814] fs/cifs/smb2misc.c: SMB2 len 104
>[71260.081824] fs/cifs/smb2misc.c: SMB2 len 124
>[71260.081826] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 89
>[71260.081828] fs/cifs/smb2ops.c: add 30 credits total=1032
>[71260.081841] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=87
>state=4
>[71260.081843] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=88
>state=4
>[71260.081845] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=89
>state=4
>[71260.081847] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71260.081854] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71260.081904] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71260.081936] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13366) rc = 0
>[71286.060785] INFO: task node_exporter:7214 blocked for more than 362
>seconds.
>[71286.061989] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71286.063131] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71286.064284] node_exporter D 0 7214 1 0x00000000
>[71286.064287] Call Trace:
>[71286.064294] ? __schedule+0x540/0xac0
>[71286.064297] ? firmware_map_remove+0xe9/0xe9
>[71286.064301] ? vsnprintf+0x32c/0x870
>[71286.064304] ? _raw_spin_lock+0x7a/0xd0
>[71286.064306] schedule+0x5e/0x100
>[71286.064308] schedule_preempt_disabled+0xa/0x10
>[71286.064310] __mutex_lock.isra.4+0x484/0x820
>[71286.064312] ? mutex_trylock+0x90/0x90
>[71286.064315] ? string_nocheck+0xb0/0xd0
>[71286.064317] ? pointer+0x387/0x460
>[71286.064319] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71286.064321] ? _raw_write_lock_bh+0xe0/0xe0
>[71286.064324] ? mutex_lock+0xce/0xe0
>[71286.064325] mutex_lock+0xce/0xe0
>[71286.064327] ? __mutex_lock_slowpath+0x10/0x10
>[71286.064331] ? find_nls+0x7d/0xa0
>[71286.064381] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
>[71286.064386] ? deref_stack_reg+0x88/0xd0
>[71286.064392] ? 0xffffffffa1e00000
>[71286.064434] ? SMB2_tcon+0xab0/0xab0 [cifs]
>[71286.064436] ? unwind_next_frame+0x90a/0x980
>[71286.064438] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71286.064442] ? __module_text_address+0x11/0xa0
>[71286.064444] ? __is_insn_slot_addr+0x1b/0x70
>[71286.064447] ? is_bpf_text_address+0xa/0x20
>[71286.064450] ? kernel_text_address+0xe2/0xf0
>[71286.064451] ? __kernel_text_address+0xe/0x30
>[71286.064454] ? unwind_get_return_address+0x2f/0x50
>[71286.064457] ? create_prof_cpu_mask+0x20/0x20
>[71286.064460] ? arch_stack_walk+0x92/0xe0
>[71286.064463] ? stack_trace_save+0x8a/0xb0
>[71286.064505] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
>[71286.064548] SMB2_open_init+0x143/0x12b0 [cifs]
>[71286.064586] ? cifs_statfs+0x13a/0x420 [cifs]
>[71286.064590] ? statfs_by_dentry+0xa5/0xf0
>[71286.064592] ? vfs_statfs+0x28/0x110
>[71286.064594] ? __do_sys_statfs+0x64/0xc0
>[71286.064636] ? smb2_parse_contexts+0x270/0x270 [cifs]
>[71286.064638] ? _raw_write_trylock+0xe0/0xe0
>[71286.064640] ? _raw_spin_lock+0x7a/0xd0
>[71286.064641] ? _raw_write_trylock+0xe0/0xe0
>[71286.064643] ? memset+0x1f/0x40
>[71286.064646] ? stack_access_ok+0x35/0x90
>[71286.064648] ? deref_stack_reg+0x88/0xd0
>[71286.064650] ? stack_access_ok+0x35/0x90
>[71286.064652] ? deref_stack_reg+0x88/0xd0
>[71286.064654] ? __read_once_size_nocheck.constprop.6+0x10/0x10
>[71286.064656] ? put_dec_trunc8+0x73/0x110
>[71286.064658] ? number+0x356/0x4b0
>[71286.064660] ? widen_string+0x23/0xf0
>[71286.064705] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71286.064770] smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71286.064817] ? smb2_query_symlink+0xc70/0xc70 [cifs]
>[71286.064821] ? pointer+0x460/0x460
>[71286.064827] ? kernel_text_address+0xe2/0xf0
>[71286.064835] ? va_format.isra.12+0xee/0x100
>[71286.064841] ? vsnprintf+0x870/0x870
>[71286.064854] ? string_nocheck+0xb0/0xd0
>[71286.064869] ? pointer+0x387/0x460
>[71286.064875] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71286.064886] ? _raw_write_lock_bh+0xe0/0xe0
>[71286.064888] ? _raw_spin_lock+0x7a/0xd0
>[71286.064889] ? _raw_write_trylock+0xe0/0xe0
>[71286.064892] ? up+0x32/0x70
>[71286.064896] ? __switch_to_asm+0x34/0x70
>[71286.064898] ? dynamic_emit_prefix+0x29/0x220
>[71286.064940] ? smb2_queryfs+0xd9/0x1c0 [cifs]
>[71286.064982] smb2_queryfs+0xd9/0x1c0 [cifs]
>[71286.065029] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
>[71286.065035] ? vfs_mknod+0xc0/0x320
>[71286.065038] ? map_id_up+0x12f/0x1d0
>[71286.065040] ? make_kprojid+0x20/0x20
>[71286.065042] ? _raw_spin_lock+0x7a/0xd0
>[71286.065081] cifs_statfs+0x13a/0x420 [cifs]
>[71286.065088] statfs_by_dentry+0xa5/0xf0
>[71286.065091] vfs_statfs+0x28/0x110
>[71286.065093] user_statfs+0x91/0xf0
>[71286.065095] ? vfs_statfs+0x110/0x110
>[71286.065099] ? __schedule+0x562/0xac0
>[71286.065103] __do_sys_statfs+0x64/0xc0
>[71286.065105] ? user_statfs+0xf0/0xf0
>[71286.065109] do_syscall_64+0x73/0x190
>[71286.065112] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71286.065116] RIP: 0033:0x4a5c20
>[71286.065121] Code: Bad RIP value.
>[71286.065122] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX:
>0000000000000089
>[71286.065125] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX:
>00000000004a5c20
>[71286.065127] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI:
>000000c0001c43c0
>[71286.065131] RBP: 000000c0003414f0 R08: 0000000000000000 R09:
>0000000000000000
>[71286.065133] R10: 0000000000000000 R11: 0000000000000206 R12:
>ffffffffffffffff
>[71286.065134] R13: 000000000000001f R14: 000000000000001e R15:
>0000000000000100
>[71286.065156] INFO: task cifsd:9136 blocked for more than 362 seconds.
>[71286.066326] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71286.068159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71286.069395] cifsd D 0 9136 2 0x80004000
>[71286.069399] Call Trace:
>[71286.069405] ? __schedule+0x540/0xac0
>[71286.069408] ? firmware_map_remove+0xe9/0xe9
>[71286.069411] ? _raw_spin_lock+0x7a/0xd0
>[71286.069413] schedule+0x5e/0x100
>[71286.069416] schedule_preempt_disabled+0xa/0x10
>[71286.069417] __mutex_lock.isra.4+0x484/0x820
>[71286.069420] ? mutex_trylock+0x90/0x90
>[71286.069424] ? irq_work_claim+0x2e/0x50
>[71286.069427] ? vprintk_emit+0x11d/0x2e0
>[71286.069430] ? mutex_lock+0xce/0xe0
>[71286.069431] mutex_lock+0xce/0xe0
>[71286.069433] ? __mutex_lock_slowpath+0x10/0x10
>[71286.069485] dfs_cache_noreq_find+0xa7/0x190 [cifs]
>[71286.069526] cifs_reconnect+0x18c/0x1510 [cifs]
>[71286.069530] ? ___ratelimit+0x106/0x190
>[71286.069572] ? smb2_calc_size+0x15c/0x250 [cifs]
>[71286.069611] ? extract_hostname+0xa0/0xa0 [cifs]
>[71286.069613] ? _raw_spin_trylock+0x91/0xe0
>[71286.069615] ? _raw_spin_trylock_bh+0x100/0x100
>[71286.069617] ? ___ratelimit+0x106/0x190
>[71286.069657] cifs_handle_standard+0x252/0x270 [cifs]
>[71286.069697] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
>[71286.069737] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71286.069739] ? __switch_to_asm+0x40/0x70
>[71286.069741] ? __switch_to_asm+0x34/0x70
>[71286.069742] ? __switch_to_asm+0x40/0x70
>[71286.069744] ? __switch_to_asm+0x34/0x70
>[71286.069746] ? __switch_to_asm+0x40/0x70
>[71286.069748] ? __switch_to_asm+0x34/0x70
>[71286.069750] ? __switch_to_asm+0x40/0x70
>[71286.069752] ? __switch_to_asm+0x34/0x70
>[71286.069753] ? __switch_to_asm+0x40/0x70
>[71286.069755] ? __switch_to_asm+0x34/0x70
>[71286.069756] ? __switch_to_asm+0x40/0x70
>[71286.069758] ? __switch_to_asm+0x34/0x70
>[71286.069759] ? __switch_to_asm+0x40/0x70
>[71286.069761] ? __switch_to_asm+0x34/0x70
>[71286.069762] ? __switch_to_asm+0x40/0x70
>[71286.069764] ? __switch_to_asm+0x34/0x70
>[71286.069765] ? __switch_to_asm+0x40/0x70
>[71286.069767] ? __switch_to_asm+0x34/0x70
>[71286.069768] ? __switch_to_asm+0x40/0x70
>[71286.069770] ? __switch_to_asm+0x34/0x70
>[71286.069772] ? __switch_to_asm+0x40/0x70
>[71286.069773] ? __switch_to_asm+0x34/0x70
>[71286.069775] ? __switch_to_asm+0x40/0x70
>[71286.069776] ? __switch_to_asm+0x34/0x70
>[71286.069778] ? __switch_to_asm+0x40/0x70
>[71286.069779] ? __switch_to_asm+0x34/0x70
>[71286.069782] ? finish_task_switch+0xf6/0x370
>[71286.069784] ? __switch_to+0x2ec/0x5e0
>[71286.069786] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71286.069788] ? _raw_write_lock_bh+0xe0/0xe0
>[71286.069828] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71286.069830] kthread+0x192/0x1e0
>[71286.069832] ? kthread_create_worker_on_cpu+0xc0/0xc0
>[71286.069834] ret_from_fork+0x35/0x40
>[71286.069842] INFO: task kworker/0:0:16069 blocked for more than 120
>seconds.
>[71286.071059] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71286.072279] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71286.073534] kworker/0:0 D 0 16069 2 0x80004000
>[71286.073585] Workqueue: cifsiod refresh_cache_worker [cifs]
>[71286.073586] Call Trace:
>[71286.073590] ? __schedule+0x540/0xac0
>[71286.073592] ? firmware_map_remove+0xe9/0xe9
>[71286.073595] ? _raw_read_lock_irq+0x40/0x40
>[71286.073597] ? _raw_spin_lock+0x7a/0xd0
>[71286.073599] schedule+0x5e/0x100
>[71286.073601] schedule_preempt_disabled+0xa/0x10
>[71286.073603] __mutex_lock.isra.4+0x484/0x820
>[71286.073606] ? mutex_trylock+0x90/0x90
>[71286.073609] ? dynamic_emit_prefix+0x29/0x220
>[71286.073610] ? __dynamic_pr_debug+0xf8/0x140
>[71286.073612] ? dynamic_emit_prefix+0x220/0x220
>[71286.073618] ? update_dl_rq_load_avg+0x2c3/0x4d0
>[71286.073620] ? mutex_lock+0xce/0xe0
>[71286.073621] mutex_lock+0xce/0xe0
>[71286.073623] ? __mutex_lock_slowpath+0x10/0x10
>[71286.073665] refresh_cache_worker+0x48f/0x14a0 [cifs]
>[71286.073668] ? __switch_to_asm+0x40/0x70
>[71286.073670] ? __switch_to_asm+0x40/0x70
>[71286.073672] ? __switch_to_asm+0x34/0x70
>[71286.073673] ? __switch_to_asm+0x40/0x70
>[71286.073675] ? __switch_to_asm+0x34/0x70
>[71286.073676] ? __switch_to_asm+0x40/0x70
>[71286.073718] ? find_root_ses.isra.9+0x320/0x320 [cifs]
>[71286.073720] ? __switch_to_asm+0x40/0x70
>[71286.073722] ? __switch_to_asm+0x40/0x70
>[71286.073723] ? __switch_to_asm+0x34/0x70
>[71286.073725] ? __switch_to_asm+0x40/0x70
>[71286.073728] ? __switch_to_asm+0x34/0x70
>[71286.073729] ? __switch_to_asm+0x40/0x70
>[71286.073731] ? __switch_to_asm+0x34/0x70
>[71286.073732] ? __switch_to_asm+0x40/0x70
>[71286.073734] ? __switch_to_asm+0x40/0x70
>[71286.073736] ? __switch_to_asm+0x34/0x70
>[71286.073738] ? finish_task_switch+0xf6/0x370
>[71286.073741] ? __switch_to+0x2ec/0x5e0
>[71286.073742] ? __schedule+0x562/0xac0
>[71286.073745] ? read_word_at_a_time+0xe/0x20
>[71286.073747] ? strscpy+0xca/0x1d0
>[71286.073753] process_one_work+0x373/0x6e0
>[71286.073756] worker_thread+0x78/0x5b0
>[71286.073759] ? rescuer_thread+0x5e0/0x5e0
>[71286.073760] kthread+0x192/0x1e0
>[71286.073762] ? kthread_create_worker_on_cpu+0xc0/0xc0
>[71286.073764] ret_from_fork+0x35/0x40
>[71308.590706] fs/cifs/smb2pdu.c: In echo request
>[71308.590730] fs/cifs/smb2pdu.c: Echo request failed: -11
>[71308.590743] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[71320.544886] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13367
>with uid: 999
>[71320.544989] fs/cifs/transport.c: Sending smb: smb_len=372
>[71320.545462] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71320.545477] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71320.545479] fs/cifs/smb2misc.c: SMB2 len 208
>[71320.545495] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71320.545496] fs/cifs/smb2misc.c: SMB2 len 104
>[71320.545512] fs/cifs/smb2misc.c: SMB2 len 124
>[71320.545515] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4084
>[71320.545517] fs/cifs/smb2ops.c: add 3 credits total=512
>[71320.545533] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4082 state=4
>[71320.545539] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4083 state=4
>[71320.545542] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4084 state=4
>[71320.545548] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71320.545555] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71320.545559] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71320.545620] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13367) rc = 0
>[71320.548354] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13368
>with uid: 999
>[71320.548525] fs/cifs/transport.c: Sending smb: smb_len=348
>[71320.548816] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71320.548826] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71320.548828] fs/cifs/smb2misc.c: SMB2 len 208
>[71320.548844] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71320.548846] fs/cifs/smb2misc.c: SMB2 len 104
>[71320.548864] fs/cifs/smb2misc.c: SMB2 len 124
>[71320.548867] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 92
>[71320.548870] fs/cifs/smb2ops.c: add 30 credits total=1059
>[71320.548887] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=90
>state=4
>[71320.548890] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=91
>state=4
>[71320.548891] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=92
>state=4
>[71320.548893] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71320.548900] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71320.548904] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71320.548937] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13368) rc = 0
>[71370.021141] fs/cifs/smb2pdu.c: In echo request
>[71370.021164] fs/cifs/smb2pdu.c: Echo request failed: -11
>[71370.021172] fs/cifs/connect.c: Unable to send echo request to
>server: DC02
>[71380.065094] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13369
>with uid: 999
>[71380.065186] fs/cifs/transport.c: Sending smb: smb_len=372
>[71380.065747] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71380.065757] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71380.065758] fs/cifs/smb2misc.c: SMB2 len 208
>[71380.065771] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71380.065773] fs/cifs/smb2misc.c: SMB2 len 104
>[71380.065786] fs/cifs/smb2misc.c: SMB2 len 124
>[71380.065789] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 4087
>[71380.065792] fs/cifs/smb2ops.c: add 3 credits total=512
>[71380.065826] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5
>mid=4085 state=4
>[71380.065831] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16
>mid=4086 state=4
>[71380.065834] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6
>mid=4087 state=4
>[71380.065837] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71380.065843] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71380.065848] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71380.065881] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13369) rc = 0
>[71380.065908] fs/cifs/cifsfs.c: CIFS VFS: in cifs_statfs as Xid: 13370
>with uid: 999
>[71380.065966] fs/cifs/transport.c: Sending smb: smb_len=348
>[71380.066494] fs/cifs/connect.c: RFC1002 header 0x1b8
>[71380.066501] fs/cifs/smb2misc.c: SMB2 data length 56 offset 152
>[71380.066503] fs/cifs/smb2misc.c: SMB2 len 208
>[71380.066516] fs/cifs/smb2misc.c: SMB2 data length 32 offset 72
>[71380.066517] fs/cifs/smb2misc.c: SMB2 len 104
>[71380.066532] fs/cifs/smb2misc.c: SMB2 len 124
>[71380.066534] fs/cifs/smb2misc.c: Calculated size 124 length 128
>mismatch mid 95
>[71380.066536] fs/cifs/smb2ops.c: add 30 credits total=1086
>[71380.067649] fs/cifs/transport.c: cifs_sync_mid_result: cmd=5 mid=93
>state=4
>[71380.067652] fs/cifs/transport.c: cifs_sync_mid_result: cmd=16 mid=94
>state=4
>[71380.067656] fs/cifs/transport.c: cifs_sync_mid_result: cmd=6 mid=95
>state=4
>[71380.067658] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71380.067667] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71380.067672] fs/cifs/misc.c: Null buffer passed to
>cifs_small_buf_release
>[71380.067707] fs/cifs/cifsfs.c: CIFS VFS: leaving cifs_statfs (xid =
>13370) rc = 0
>[71406.881866] INFO: task node_exporter:7214 blocked for more than 483
>seconds.
>[71406.883327] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71406.884700] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71406.886064] node_exporter D 0 7214 1 0x00000000
>[71406.886068] Call Trace:
>[71406.886075] ? __schedule+0x540/0xac0
>[71406.886080] ? firmware_map_remove+0xe9/0xe9
>[71406.886087] ? vsnprintf+0x32c/0x870
>[71406.886091] ? _raw_spin_lock+0x7a/0xd0
>[71406.886095] schedule+0x5e/0x100
>[71406.886098] schedule_preempt_disabled+0xa/0x10
>[71406.886104] __mutex_lock.isra.4+0x484/0x820
>[71406.886106] ? mutex_trylock+0x90/0x90
>[71406.886108] ? string_nocheck+0xb0/0xd0
>[71406.886111] ? pointer+0x387/0x460
>[71406.886113] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71406.886115] ? _raw_write_lock_bh+0xe0/0xe0
>[71406.886121] ? mutex_lock+0xce/0xe0
>[71406.886122] mutex_lock+0xce/0xe0
>[71406.886124] ? __mutex_lock_slowpath+0x10/0x10
>[71406.886126] ? find_nls+0x7d/0xa0
>[71406.886194] smb2_reconnect.part.21+0x218/0xbd0 [cifs]
>[71406.886206] ? deref_stack_reg+0x88/0xd0
>[71406.886210] ? 0xffffffffa1e00000
>[71406.886254] ? SMB2_tcon+0xab0/0xab0 [cifs]
>[71406.886258] ? unwind_next_frame+0x90a/0x980
>[71406.886260] ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71406.886263] ? __module_text_address+0x11/0xa0
>[71406.886265] ? __is_insn_slot_addr+0x1b/0x70
>[71406.886272] ? is_bpf_text_address+0xa/0x20
>[71406.886274] ? kernel_text_address+0xe2/0xf0
>[71406.886275] ? __kernel_text_address+0xe/0x30
>[71406.886278] ? unwind_get_return_address+0x2f/0x50
>[71406.886280] ? create_prof_cpu_mask+0x20/0x20
>[71406.886288] ? arch_stack_walk+0x92/0xe0
>[71406.886291] ? stack_trace_save+0x8a/0xb0
>[71406.886337] smb2_plain_req_init+0x2fe/0x3d0 [cifs]
>[71406.886381] SMB2_open_init+0x143/0x12b0 [cifs]
>[71406.886420] ? cifs_statfs+0x13a/0x420 [cifs]
>[71406.886424] ? statfs_by_dentry+0xa5/0xf0
>[71406.886426] ? vfs_statfs+0x28/0x110
>[71406.886428] ? __do_sys_statfs+0x64/0xc0
>[71406.886470] ? smb2_parse_contexts+0x270/0x270 [cifs]
>[71406.886475] ? _raw_write_trylock+0xe0/0xe0
>[71406.886476] ? _raw_spin_lock+0x7a/0xd0
>[71406.886478] ? _raw_write_trylock+0xe0/0xe0
>[71406.886480] ? memset+0x1f/0x40
>[71406.886482] ? stack_access_ok+0x35/0x90
>[71406.886484] ? deref_stack_reg+0x88/0xd0
>[71406.886486] ? stack_access_ok+0x35/0x90
>[71406.886488] ? deref_stack_reg+0x88/0xd0
>[71406.886491] ? __read_once_size_nocheck.constprop.6+0x10/0x10
>[71406.886493] ? put_dec_trunc8+0x73/0x110
>[71406.886495] ? number+0x356/0x4b0
>[71406.886497] ? widen_string+0x23/0xf0
>[71406.886538] ? smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71406.886580] smb2_query_info_compound+0x242/0x4d0 [cifs]
>[71406.886622] ? smb2_query_symlink+0xc70/0xc70 [cifs]
>[71406.886624] ? pointer+0x460/0x460
>[71406.886626] ? kernel_text_address+0xe2/0xf0
>[71406.886628] ? va_format.isra.12+0xee/0x100
>[71406.886630] ? vsnprintf+0x870/0x870
>[71406.886632] ? string_nocheck+0xb0/0xd0
>[71406.886634] ? pointer+0x387/0x460
>[71406.886636] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71406.886638] ? _raw_write_lock_bh+0xe0/0xe0
>[71406.886640] ? _raw_spin_lock+0x7a/0xd0
>[71406.886641] ? _raw_write_trylock+0xe0/0xe0
>[71406.886644] ? up+0x32/0x70
>[71406.886647] ? __switch_to_asm+0x34/0x70
>[71406.886649] ? dynamic_emit_prefix+0x29/0x220
>[71406.886691] ? smb2_queryfs+0xd9/0x1c0 [cifs]
>[71406.886744] smb2_queryfs+0xd9/0x1c0 [cifs]
>[71406.886789] ? smb2_query_eas+0x4f0/0x4f0 [cifs]
>[71406.886792] ? vfs_mknod+0xc0/0x320
>[71406.886795] ? map_id_up+0x12f/0x1d0
>[71406.886797] ? make_kprojid+0x20/0x20
>[71406.886799] ? _raw_spin_lock+0x7a/0xd0
>[71406.886836] cifs_statfs+0x13a/0x420 [cifs]
>[71406.886839] statfs_by_dentry+0xa5/0xf0
>[71406.886842] vfs_statfs+0x28/0x110
>[71406.886844] user_statfs+0x91/0xf0
>[71406.886846] ? vfs_statfs+0x110/0x110
>[71406.886848] ? __schedule+0x562/0xac0
>[71406.886850] __do_sys_statfs+0x64/0xc0
>[71406.886852] ? user_statfs+0xf0/0xf0
>[71406.886856] do_syscall_64+0x73/0x190
>[71406.886858] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>[71406.886860] RIP: 0033:0x4a5c20
>[71406.886864] Code: Bad RIP value.
>[71406.886865] RSP: 002b:000000c000341490 EFLAGS: 00000206 ORIG_RAX:
>0000000000000089
>[71406.886867] RAX: ffffffffffffffda RBX: 000000c00002ea00 RCX:
>00000000004a5c20
>[71406.886868] RDX: 0000000000000000 RSI: 000000c0003415c0 RDI:
>000000c0001c43c0
>[71406.886869] RBP: 000000c0003414f0 R08: 0000000000000000 R09:
>0000000000000000
>[71406.886871] R10: 0000000000000000 R11: 0000000000000206 R12:
>ffffffffffffffff
>[71406.886871] R13: 000000000000001f R14: 000000000000001e R15:
>0000000000000100
>[71406.886897] INFO: task cifsd:9136 blocked for more than 483 seconds.
>[71406.888219] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71406.889666] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71406.891125] cifsd D 0 9136 2 0x80004000
>[71406.891130] Call Trace:
>[71406.891138] ? __schedule+0x540/0xac0
>[71406.891147] ? firmware_map_remove+0xe9/0xe9
>[71406.891156] ? _raw_spin_lock+0x7a/0xd0
>[71406.891161] schedule+0x5e/0x100
>[71406.891168] schedule_preempt_disabled+0xa/0x10
>[71406.891173] __mutex_lock.isra.4+0x484/0x820
>[71406.891179] ? mutex_trylock+0x90/0x90
>[71406.891192] ? irq_work_claim+0x2e/0x50
>[71406.891207] ? vprintk_emit+0x11d/0x2e0
>[71406.891216] ? mutex_lock+0xce/0xe0
>[71406.891228] mutex_lock+0xce/0xe0
>[71406.891232] ? __mutex_lock_slowpath+0x10/0x10
>[71406.891317] dfs_cache_noreq_find+0xa7/0x190 [cifs]
>[71406.891378] cifs_reconnect+0x18c/0x1510 [cifs]
>[71406.891383] ? ___ratelimit+0x106/0x190
>[71406.891439] ? smb2_calc_size+0x15c/0x250 [cifs]
>[71406.891505] ? extract_hostname+0xa0/0xa0 [cifs]
>[71406.891510] ? _raw_spin_trylock+0x91/0xe0
>[71406.891513] ? _raw_spin_trylock_bh+0x100/0x100
>[71406.891516] ? ___ratelimit+0x106/0x190
>[71406.891574] cifs_handle_standard+0x252/0x270 [cifs]
>[71406.891631] cifs_demultiplex_thread+0x124a/0x13e0 [cifs]
>[71406.891692] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71406.891696] ? __switch_to_asm+0x40/0x70
>[71406.891698] ? __switch_to_asm+0x34/0x70
>[71406.891701] ? __switch_to_asm+0x40/0x70
>[71406.891703] ? __switch_to_asm+0x34/0x70
>[71406.891706] ? __switch_to_asm+0x40/0x70
>[71406.891709] ? __switch_to_asm+0x34/0x70
>[71406.891712] ? __switch_to_asm+0x40/0x70
>[71406.891715] ? __switch_to_asm+0x34/0x70
>[71406.891718] ? __switch_to_asm+0x40/0x70
>[71406.891720] ? __switch_to_asm+0x34/0x70
>[71406.891723] ? __switch_to_asm+0x40/0x70
>[71406.891725] ? __switch_to_asm+0x34/0x70
>[71406.891727] ? __switch_to_asm+0x40/0x70
>[71406.891729] ? __switch_to_asm+0x34/0x70
>[71406.891732] ? __switch_to_asm+0x40/0x70
>[71406.891734] ? __switch_to_asm+0x34/0x70
>[71406.891737] ? __switch_to_asm+0x40/0x70
>[71406.891739] ? __switch_to_asm+0x34/0x70
>[71406.891741] ? __switch_to_asm+0x40/0x70
>[71406.891744] ? __switch_to_asm+0x34/0x70
>[71406.891746] ? __switch_to_asm+0x40/0x70
>[71406.891749] ? __switch_to_asm+0x34/0x70
>[71406.891751] ? __switch_to_asm+0x40/0x70
>[71406.891753] ? __switch_to_asm+0x34/0x70
>[71406.891756] ? __switch_to_asm+0x40/0x70
>[71406.891758] ? __switch_to_asm+0x34/0x70
>[71406.891763] ? finish_task_switch+0xf6/0x370
>[71406.891767] ? __switch_to+0x2ec/0x5e0
>[71406.891772] ? _raw_spin_lock_irqsave+0x8d/0xf0
>[71406.891774] ? _raw_write_lock_bh+0xe0/0xe0
>[71406.891946] ? cifs_handle_standard+0x270/0x270 [cifs]
>[71406.891951] kthread+0x192/0x1e0
>[71406.891955] ? kthread_create_worker_on_cpu+0xc0/0xc0
>[71406.891958] ret_from_fork+0x35/0x40
>[71406.891970] INFO: task kworker/0:0:16069 blocked for more than 241
>seconds.
>[71406.894262] Tainted: G E 5.3.11-pd-5.3.y
>#20191029
>[71406.896328] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>disables this message.
>[71406.898113] kworker/0:0 D 0 16069 2 0x80004000
>[71406.898203] Workqueue: cifsiod refresh_cache_worker [cifs]
>[71406.898211] Call Trace:
>[71406.898220] ? __schedule+0x540/0xac0
>[71406.898224] ? firmware_map_remove+0xe9/0xe9
>[71406.898228] ? _raw_read_lock_irq+0x40/0x40
>[71406.898231] ? _raw_spin_lock+0x7a/0xd0
>[71406.898234] schedule+0x5e/0x100
>[71406.898239] schedule_preempt_disabled+0xa/0x10
>[71406.898242] __mutex_lock.isra.4+0x484/0x820
>[71406.898247] ? mutex_trylock+0x90/0x90
>[71406.898251] ? dynamic_emit_prefix+0x29/0x220
>[71406.898255] ? __dynamic_pr_debug+0xf8/0x140
>[71406.898260] ? dynamic_emit_prefix+0x220/0x220
>[71406.898271] ? update_dl_rq_load_avg+0x2c3/0x4d0
>[71406.898277] ? mutex_lock+0xce/0xe0
>[71406.898280] mutex_lock+0xce/0xe0
>[71406.898283] ? __mutex_lock_slowpath+0x10/0x10
>[71406.898362] refresh_cache_worker+0x48f/0x14a0 [cifs]
>[71406.898370] ? __switch_to_asm+0x40/0x70
>[71406.898373] ? __switch_to_asm+0x40/0x70
>[71406.898376] ? __switch_to_asm+0x34/0x70
>[71406.898378] ? __switch_to_asm+0x40/0x70
>[71406.898383] ? __switch_to_asm+0x34/0x70
>[71406.898389] ? __switch_to_asm+0x40/0x70
>[71406.898467] ? find_root_ses.isra.9+0x320/0x320 [cifs]
>[71406.898472] ? __switch_to_asm+0x40/0x70
>[71406.898476] ? __switch_to_asm+0x40/0x70
>[71406.898479] ? __switch_to_asm+0x34/0x70
>[71406.898484] ? __switch_to_asm+0x40/0x70
>[71406.898488] ? __switch_to_asm+0x34/0x70
>[71406.898490] ? __switch_to_asm+0x40/0x70
>[71406.898499] ? __switch_to_asm+0x34/0x70
>[71406.898502] ? __switch_to_asm+0x40/0x70
>[71406.898505] ? __switch_to_asm+0x40/0x70
>[71406.898508] ? __switch_to_asm+0x34/0x70
>[71406.898516] ? finish_task_switch+0xf6/0x370
>[71406.898519] ? __switch_to+0x2ec/0x5e0
>[71406.898523] ? __schedule+0x562/0xac0
>[71406.898530] ? read_word_at_a_time+0xe/0x20
>[71406.898533] ? strscpy+0xca/0x1d0
>[71406.898539] process_one_work+0x373/0x6e0
>[71406.898547] worker_thread+0x78/0x5b0
>[71406.898552] ? rescuer_thread+0x5e0/0x5e0
>[71406.898555] kthread+0x192/0x1e0
>[71406.898559] ? kthread_create_worker_on_cpu+0xc0/0xc0
>[71406.898563] ret_from_fork+0x35/0x40
>[71417.962152] fs/cifs/inode.c: CIFS VFS: in
>cifs_revalidate_dentry_attr as Xid: 13371 with uid: 11025
>[71417.962171] fs/cifs/dir.c: name: \SOMEDIRX
>[71417.962177] fs/cifs/inode.c: Update attributes: \SOMEDIRX inode
>0x000000009f726d03 count 1 dentry: 0x00000000d22e461a d_time 4312598279
>jiffies 4312748242
>[71417.962180] fs/cifs/inode.c: Getting info on \SOMEDIRX
>[71417.962367] fs/cifs/transport.c: Sending smb: smb_len=388
>
>Gr, Martijn
Yeah, thats another issue indeed. Thanks for the report.
I will look into it and let you know.
Paulo
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-10-30 19:33 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-13 15:55 Kernel hangs in cifs_reconnect Martijn de Gouw
2019-10-15 15:33 ` Aurélien Aptel
2019-10-15 16:27 ` Paulo Alcantara
2019-10-20 8:13 ` Martijn de Gouw
2019-10-29 11:07 ` Martijn de Gouw
2019-10-29 14:49 ` Paulo Alcantara
2019-10-29 18:38 ` Martijn de Gouw
2019-10-30 19:17 ` Martijn de Gouw
2019-10-30 19:33 ` Paulo Alcantara
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).