* NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5
@ 2013-06-10 17:42 Nix
2013-06-11 3:15 ` Al Viro
2013-06-12 1:23 ` Al Viro
0 siblings, 2 replies; 7+ messages in thread
From: Nix @ 2013-06-10 17:42 UTC (permalink / raw)
To: linux-kernel
Yes, my shutdown scripts are panicking the kernel again! They're not
causing filesystem corruption this time, but it's still fs-related.
Here's the 3.9.5 panic, seen on an x86-32 NFS client using NFSv3: NFSv4
was compiled in but not used. This happened when processes whose
current directory was on one of those NFS-mounted filesystems were being
killed, after it had been lazy-umounted (so by this point its cwd was in
a disconnected mount point).
[ 251.246800] BUG: unable to handle kernel NULL pointer dereference at 00000004
[ 251.256556] IP: [<c01739f6>] path_init+0xc7/0x27f
[ 251.256556] *pde = 00000000
[ 251.256556] Oops: 0000 [#1]
[ 251.256556] Pid: 748, comm: su Not tainted 3.9.5+ #1
[ 251.256556] EIP: 0060:[<c01739f6>] EFLAGS: 00010246 CPU: 0
[ 251.256556] EIP is at path_init+0xc7/0x27f
[ 251.256556] EAX: df63da80 EBX: dd501d64 ECX: 00000000 EDX: 00001051
[ 251.256556] ESI: dd501d40 EDI: 00000040 EBP: df5f180e ESP: dd501cc8
[ 251.256556] DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
[ 251.256556] CR0: 8005003b CR2: 00000004 CR3: 1f7ee000 CR4: 00000090
[ 251.256556] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 251.256556] DR6: ffff0ff0 DR7: 00000400
[ 251.256556] Process su (pid: 748, ti=dd500000 task=df63da80 task.ti=dd500000)
[ 251.256556] Stack:
[ 251.256556] c03fe9ac 00000044 df1ac000 dd501d64 dd501d40 00000041 df5f180e c0174832
[ 251.256556] dd501d64 dd501cf8 000009c0 00000040 00000000 00000040 00000000 00000000
[ 251.256556] 00000000 00000001 ffffff9c dd501d40 dd501d64 00000001 c0174db5 dd501d64
[ 251.256556] Call Trace:
[ 251.256556] [<c0174832>] ? path_lookupat+0x2c/0x593
[ 251.256556] [<c0174db5>] ? filename_lookup.isra.33+0x1c/0x51
[ 251.256556] [<c0174e5d>] ? do_path_lookup+0x2f/0x36
[ 251.256556] [<c0174ffb>] ? kern_path+0x1b/0x31
[ 251.256556] [<c016b8d1>] ? __kmalloc_track_caller+0x9e/0xc3
[ 251.256556] [<c026d5aa>] ? __alloc_skb+0x5f/0x14c
[ 251.256556] [<c026d40d>] ? __kmalloc_reserve.isra.38+0x1a/0x52
[ 251.256556] [<c026d5b9>] ? __alloc_skb+0x6e/0x14c
[ 251.256556] [<c02ef6ea>] ? unix_find_other.isra.40+0x24/0x133
[ 251.256556] [<c02ef8da>] ? unix_stream_connect+0xe1/0x2f7
[ 251.256556] [<c026a14d>] ? kernel_connect+0x10/0x14
[ 251.256556] [<c031ecb1>] ? xs_local_connect+0x108/0x181
[ 251.256556] [<c031c83b>] ? xprt_connect+0xcd/0xd1
[ 251.256556] [<c031fd1b>] ? __rpc_execute+0x5b/0x156
[ 251.256556] [<c0128ac2>] ? wake_up_bit+0xb/0x19
[ 251.256556] [<c031b83d>] ? rpc_run_task+0x55/0x5a
[ 251.256556] [<c031b8bc>] ? rpc_call_sync+0x7a/0x8d
[ 251.256556] [<c0325127>] ? rpcb_register_call+0x11/0x20
[ 251.256556] [<c032548a>] ? rpcb_v4_register+0x87/0xf6
[ 251.256556] [<c0321187>] ? svc_unregister.isra.22+0x46/0x87
[ 251.256556] [<c03211d0>] ? svc_rpcb_cleanup+0x8/0x10
[ 251.256556] [<c03213df>] ? svc_shutdown_net+0x18/0x1b
[ 251.256556] [<c01cb1f3>] ? lockd_down+0x22/0x97
[ 251.256556] [<c01c89df>] ? nlmclnt_done+0xc/0x14
[ 251.256556] [<c01b9064>] ? nfs_free_server+0x7f/0xdb
[ 251.256556] [<c016e776>] ? deactivate_locked_super+0x16/0x3e
[ 251.256556] [<c0187e17>] ? free_fs_struct+0x13/0x20
[ 251.256556] [<c011a009>] ? do_exit+0x224/0x64f
[ 251.256556] [<c016d51f>] ? vfs_write+0x82/0x108
[ 251.256556] [<c011a492>] ? do_group_exit+0x3a/0x65
[ 251.256556] [<c011a4ce>] ? sys_exit_group+0x11/0x11
[ 251.256556] [<c0332b3d>] ? syscall_call+0x7/0xb
[ 251.256556] Code: 00 80 7d 00 2f 0f 85 8b 00 00 00 83 e7 40 74 4e b8 a0 b2 3e c0 e8 c0 91 fb ff 83 7b 14 00 75 66 a1 00 1e 3e c0 8b 88 54 02 00 00 <8b> 71 04 f7 c6 01 00 00 00 74 04 f3 90 eb f1 8b 51 14 8b 41 10
[ 251.256556] EIP: [<c01739f6>] path_init+0xc7/0x27f SS:ESP 0068:dd501cc8
[ 251.256556] CR2: 0000000000000004
I was seeing very similar problems in 3.9.2 on a quite differently
configured x86-64 box -- but still with NFSv4 configured in but not
used, and an NFSv3 mount, and not-yet-killed processes inside a
lazy-umounted NFS filesystem. I reboot this box a lot more than the
other one, so can confirm that it happens about 80% of the time, but not
always, perhaps due to differences in the speed of lazy-umounting:
[145348.012438] BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
[145348.013216] IP: [<ffffffff81167856>] path_init+0x11c/0x36f
[145348.013906] PGD 0
[145348.014571] Oops: 0000 [#1] PREEMPT SMP
[145348.015248] Modules linked in: [last unloaded: microcode]
[145348.015952] CPU 3
[145348.015963] Pid: 1137, comm: ssh Not tainted 3.9.2-05286-ge8a76db-dirty #1 System manufacturer System Product Name/P8H61-MX USB3
[145348.017367] RIP: 0010:[<ffffffff81167856>] [<ffffffff81167856>] path_init+0x11c/0x36f
[145348.018121] RSP: 0018:ffff88041c179538 EFLAGS: 00010246
[145348.018879] RAX: 0000000000000000 RBX: ffff88041c179688 RCX: 00000000000000c3
[145348.019654] RDX: 000000000000c3c3 RSI: ffff88041881501a RDI: ffffffff81c34910
[145348.020454] RBP: ffff88041c179588 R08: ffff88041c1795b8 R09: ffff88041c1797f4
[145348.021245] R10: 00000000ffffff9c R11: ffff88041c179688 R12: 0000000000000041
[145348.022063] R13: 0000000000000040 R14: ffff88041881501a R15: ffff88041c1797f4
[145348.022866] FS: 00007f8a2e262700(0000) GS:ffff88042fac0000(0000) knlGS:0000000000000000
[145348.023783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[145348.024629] CR2: 0000000000000008 CR3: 0000000001c0b000 CR4: 00000000001407e0
[145348.025502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[145348.026369] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[145348.027239] Process ssh (pid: 1137, threadinfo ffff88041c178000, task ffff88041c838000)
[145348.028127] Stack: [145348.029055] 0000000000000000 ffffffff8152043b ffffc900080a4000 0000000000000034
[145348.029978] ffff88041cc51098 ffff88041c179688 0000000000000041 ffff88041881501a
[145348.030913] ffff88041c179658 ffff88041c1797f4 ffff88041c179618 ffffffff81167adc
[145348.031855] Call Trace:
[145348.032786] [<ffffffff8152043b>] ? skb_checksum+0x4f/0x25b
[145348.033735] [<ffffffff81167adc>] path_lookupat+0x33/0x69b
[145348.034688] [<ffffffff8152e092>] ? dev_hard_start_xmit+0x2bf/0x4ee
[145348.035652] [<ffffffff8116816a>] filename_lookup.isra.27+0x26/0x5c
[145348.036618] [<ffffffff81168234>] do_path_lookup+0x33/0x35
[145348.037593] [<ffffffff81168462>] kern_path+0x2a/0x4d
[145348.038573] [<ffffffff8115697e>] ? __kmalloc_track_caller+0x4c/0x148
[145348.039563] [<ffffffff81522cb0>] ? __alloc_skb+0x75/0x186
[145348.040555] [<ffffffff81522444>] ? __kmalloc_reserve.isra.42+0x2d/0x6c
[145348.041559] [<ffffffff815894eb>] unix_find_other+0x38/0x1b9
[145348.042567] [<ffffffff8158b2e6>] unix_stream_connect+0x102/0x3ed
[145348.043586] [<ffffffff8151a737>] ? __sock_create+0x168/0x1c0
[145348.044610] [<ffffffff8151820b>] kernel_connect+0x10/0x12
[145348.045581] [<ffffffff815e3dbe>] xs_local_connect+0x142/0x1ca
[145348.046571] [<ffffffff815df3cc>] ? call_refreshresult+0x91/0x91
[145348.047553] [<ffffffff815e11d2>] xprt_connect+0x112/0x11b
[145348.048534] [<ffffffff815df405>] call_connect+0x39/0x3b
[145348.049523] [<ffffffff815e6276>] __rpc_execute+0xe8/0x313
[145348.050521] [<ffffffff815e6549>] rpc_execute+0x76/0x9d
[145348.051499] [<ffffffff815dfbd5>] rpc_run_task+0x78/0x80
[145348.052478] [<ffffffff815dfd13>] rpc_call_sync+0x88/0x9e
[145348.053455] [<ffffffff815ed019>] rpcb_register_call+0x1f/0x2e
[145348.054440] [<ffffffff815ed4e8>] rpcb_v4_register+0xb2/0x13a
[145348.055430] [<ffffffff8108cfe2>] ? call_timer_fn+0x15d/0x15d
[145348.056450] [<ffffffff815e8b08>] svc_unregister.isra.11+0x5a/0xcb
[145348.057457] [<ffffffff815e8b8d>] svc_rpcb_cleanup+0x14/0x21
[145348.058464] [<ffffffff815e83cb>] svc_shutdown_net+0x2b/0x30
[145348.059483] [<ffffffff81251609>] lockd_down_net+0x7f/0xa3
[145348.060508] [<ffffffff8125165e>] lockd_down+0x31/0xb4
[145348.061529] [<ffffffff8124e7bb>] nlmclnt_done+0x1f/0x23
[145348.062552] [<ffffffff8121a806>] ? nfs_start_lockd+0xc8/0xc8
[145348.063596] [<ffffffff8121a81d>] nfs_destroy_server+0x17/0x19
[145348.064618] [<ffffffff8121acda>] nfs_free_server+0xeb/0x15c
[145348.065647] [<ffffffff81221d23>] nfs_kill_super+0x1f/0x23
[145348.066663] [<ffffffff8115f44f>] deactivate_locked_super+0x26/0x52
[145348.067684] [<ffffffff81160162>] deactivate_super+0x42/0x47
[145348.068703] [<ffffffff8117633b>] mntput_no_expire+0x135/0x13d
[145348.069725] [<ffffffff81176370>] mntput+0x2d/0x2f
[145348.070834] [<ffffffff81165987>] path_put+0x20/0x24
[145348.071856] [<ffffffff8118586d>] free_fs_struct+0x20/0x33
[145348.072859] [<ffffffff811858ec>] exit_fs+0x6c/0x75
[145348.073849] [<ffffffff81084d9c>] do_exit+0x3bf/0x8fa
[145348.074847] [<ffffffff811659a0>] ? terminate_walk+0x15/0x3f
[145348.075828] [<ffffffff81166d4e>] ? link_path_walk+0x32a/0x7d7
[145348.076803] [<ffffffff8108f7a4>] ? __dequeue_signal+0x1b/0x119
[145348.077776] [<ffffffff81085471>] do_group_exit+0x6f/0xa2
[145348.078726] [<ffffffff81091df7>] get_signal_to_deliver+0x4ff/0x53d
[145348.079655] [<ffffffff81168107>] ? path_lookupat+0x65e/0x69b
[145348.080574] [<ffffffff81038d01>] do_signal+0x4d/0x4a4
[145348.081484] [<ffffffff8116682e>] ? final_putname+0x36/0x3b
[145348.082381] [<ffffffff811686ad>] ? do_unlinkat+0x45/0x1b8
[145348.083273] [<ffffffff81039184>] do_notify_resume+0x2c/0x6b
[145348.084192] [<ffffffff816126d8>] int_signal+0x12/0x17
[145348.085085] Code: c7 c7 10 49 c3 81 e8 25 bc f3 ff e8 1d 34 f3 ff 48 83 7b 20 00 0f 85 8d 00 00 00 65 48 8b 04 25 c0 b8 00 00 48 8b 80 58 05 00 00 <8b> 50 08 f6 c2 01 74 04 f3 90 eb f4 48 8b 48 18 48 89 4b 20 48
[145348.087176] RIP [<ffffffff81167856>] path_init+0x11c/0x36f
[145348.088159] RSP <ffff88041c179538>
[145348.089132] CR2: 0000000000000008
[145348.090136] ---[ end trace f005e3ca73eafb37 ]---
[145348.091112] Kernel panic - not syncing: Fatal exception
[145348.092115] drm_kms_helper: panic occurred, switching back to text console
The shutdown scripts are doing this horrible hack (because we want to
umount -l everything possible whether or not other mounts fail to
unmount, and last I tried it a straight umount -l of lots of filesystems
on one command line failed to do this: this may have changed with the
libmount-based umount):
umount_fsen()
{
LAZY=${1:-}
ONLY_TYPE=${2:-}
# List all mounts, deepest mount point first
LANG=C sort -r -k 2 /proc/mounts | \
(DIRS=""
while read DEV DIR TYPE REST; do
case "$DIR" in
/|/proc|/dev|/proc/*|/sys)
continue;; # Ignoring virtual file systems needed later
esac
if [[ -z $ONLY_TYPE ]]; then
case $TYPE in
proc|procfs|sysfs|usbfs|usbdevfs|devpts)
continue;; # Ignoring non-tmpfs virtual file systems
esac
else
[[ $TYPE != $ONLY_TYPE ]] && continue
fi
DIRS="$DIRS $DIR"
done
if [[ -z $LAZY ]]; then
umount -r -v $DIRS
else
for name in $DIRS; do
umount -l -v $name
done
fi)
}
umount_fsen -l nfs
killall5 -15
killall5 -9
So it's nothing mre than a bunch of umount -l's of NFS filesystems that
have running processes on them, followed by a kill of those processes.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5
2013-06-10 17:42 NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5 Nix
@ 2013-06-11 3:15 ` Al Viro
2013-06-11 11:11 ` Nix
2013-06-12 1:23 ` Al Viro
1 sibling, 1 reply; 7+ messages in thread
From: Al Viro @ 2013-06-11 3:15 UTC (permalink / raw)
To: Nix; +Cc: linux-kernel
On Mon, Jun 10, 2013 at 06:42:49PM +0100, Nix wrote:
> Yes, my shutdown scripts are panicking the kernel again! They're not
> causing filesystem corruption this time, but it's still fs-related.
>
> Here's the 3.9.5 panic, seen on an x86-32 NFS client using NFSv3: NFSv4
> was compiled in but not used. This happened when processes whose
> current directory was on one of those NFS-mounted filesystems were being
> killed, after it had been lazy-umounted (so by this point its cwd was in
> a disconnected mount point).
Hrm... Could you post (or drop on anonftp, etc.) disassembly of path_init() from
your kernel? The code quoted in oops looks very odd...
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5
2013-06-11 3:15 ` Al Viro
@ 2013-06-11 11:11 ` Nix
0 siblings, 0 replies; 7+ messages in thread
From: Nix @ 2013-06-11 11:11 UTC (permalink / raw)
To: Al Viro; +Cc: linux-kernel
On 11 Jun 2013, Al Viro spake thusly:
> On Mon, Jun 10, 2013 at 06:42:49PM +0100, Nix wrote:
>> Yes, my shutdown scripts are panicking the kernel again! They're not
>> causing filesystem corruption this time, but it's still fs-related.
>>
>> Here's the 3.9.5 panic, seen on an x86-32 NFS client using NFSv3: NFSv4
>> was compiled in but not used. This happened when processes whose
>> current directory was on one of those NFS-mounted filesystems were being
>> killed, after it had been lazy-umounted (so by this point its cwd was in
>> a disconnected mount point).
>
> Hrm... Could you post (or drop on anonftp, etc.) disassembly of path_init() from
> your kernel? The code quoted in oops looks very odd...
Sure! The compiler is GCC 4.7.3 20130310 (so not on a release branch: it
is possible this is an already-fixed compiler bug).
Both x86-32 and x86-64 disassemblies follow:
x86:
0000083f <path_init>:
83f: 55 push %ebp
840: 57 push %edi
841: 56 push %esi
842: 53 push %ebx
843: 83 ec 0c sub $0xc,%esp
846: 89 d5 mov %edx,%ebp
848: 89 ca mov %ecx,%edx
84a: 89 cf mov %ecx,%edi
84c: 8b 5c 24 20 mov 0x20(%esp),%ebx
850: 80 ce 10 or $0x10,%dh
853: f6 c5 20 test $0x20,%ch
856: c7 43 28 01 00 00 00 movl $0x1,0x28(%ebx)
85d: 89 53 20 mov %edx,0x20(%ebx)
860: c7 43 2c 00 00 00 00 movl $0x0,0x2c(%ebx)
867: 74 6c je 8d5 <path_init+0x96>
869: 8b 43 18 mov 0x18(%ebx),%eax
86c: 80 7d 00 00 cmpb $0x0,0x0(%ebp)
870: 8b 70 20 mov 0x20(%eax),%esi
873: 74 21 je 896 <path_init+0x57>
875: f6 46 02 02 testb $0x2,0x2(%esi)
879: 0f 85 12 02 00 00 jne a91 <path_init+0x252>
87f: 8b 46 18 mov 0x18(%esi),%eax
882: 83 38 00 cmpl $0x0,(%eax)
885: 75 05 jne 88c <path_init+0x4d>
887: e9 9a 01 00 00 jmp a26 <path_init+0x1e7>
88c: 66 83 4e 02 02 orw $0x2,0x2(%esi)
891: e9 fb 01 00 00 jmp a91 <path_init+0x252>
896: 8b 43 14 mov 0x14(%ebx),%eax
899: 8b 53 18 mov 0x18(%ebx),%edx
89c: 89 03 mov %eax,(%ebx)
89e: 89 53 04 mov %edx,0x4(%ebx)
8a1: 89 73 1c mov %esi,0x1c(%ebx)
8a4: 83 e7 40 and $0x40,%edi
8a7: 74 20 je 8c9 <path_init+0x8a>
8a9: b8 00 00 00 00 mov $0x0,%eax
8ae: e8 fc ff ff ff call 8af <path_init+0x70>
8b3: 8b 53 04 mov 0x4(%ebx),%edx
8b6: 8b 42 04 mov 0x4(%edx),%eax
8b9: a8 01 test $0x1,%al
8bb: 74 04 je 8c1 <path_init+0x82>
8bd: f3 90 pause
8bf: eb f5 jmp 8b6 <path_init+0x77>
8c1: 89 43 24 mov %eax,0x24(%ebx)
8c4: e9 bd 01 00 00 jmp a86 <path_init+0x247>
8c9: 89 d8 mov %ebx,%eax
8cb: e8 fc ff ff ff call 8cc <path_init+0x8d>
8d0: e9 b1 01 00 00 jmp a86 <path_init+0x247>
8d5: c7 43 14 00 00 00 00 movl $0x0,0x14(%ebx)
8dc: 80 7d 00 2f cmpb $0x2f,0x0(%ebp)
8e0: 0f 85 8b 00 00 00 jne 971 <path_init+0x132>
8e6: 83 e7 40 and $0x40,%edi
8e9: 74 4e je 939 <path_init+0xfa>
8eb: b8 00 00 00 00 mov $0x0,%eax
8f0: e8 fc ff ff ff call 8f1 <path_init+0xb2>
8f5: 83 7b 14 00 cmpl $0x0,0x14(%ebx)
8f9: 75 66 jne 961 <path_init+0x122>
8fb: a1 00 00 00 00 mov 0x0,%eax
900: 8b 88 54 02 00 00 mov 0x254(%eax),%ecx
906: 8b 71 04 mov 0x4(%ecx),%esi
909: f7 c6 01 00 00 00 test $0x1,%esi
90f: 74 04 je 915 <path_init+0xd6>
911: f3 90 pause
913: eb f1 jmp 906 <path_init+0xc7>
915: 8b 51 14 mov 0x14(%ecx),%edx
918: 8b 41 10 mov 0x10(%ecx),%eax
91b: 89 53 18 mov %edx,0x18(%ebx)
91e: 89 43 14 mov %eax,0x14(%ebx)
921: 8b 53 18 mov 0x18(%ebx),%edx
924: 8b 42 04 mov 0x4(%edx),%eax
927: a8 01 test $0x1,%al
929: 74 04 je 92f <path_init+0xf0>
92b: f3 90 pause
92d: eb f5 jmp 924 <path_init+0xe5>
92f: 89 43 24 mov %eax,0x24(%ebx)
932: 39 71 04 cmp %esi,0x4(%ecx)
935: 74 2a je 961 <path_init+0x122>
937: eb cd jmp 906 <path_init+0xc7>
939: a1 00 00 00 00 mov 0x0,%eax
93e: 8d 6b 14 lea 0x14(%ebx),%ebp
941: 8b 80 54 02 00 00 mov 0x254(%eax),%eax
947: 8b 70 10 mov 0x10(%eax),%esi
94a: 8b 78 14 mov 0x14(%eax),%edi
94d: 89 7b 18 mov %edi,0x18(%ebx)
950: 89 73 14 mov %esi,0x14(%ebx)
953: 89 e8 mov %ebp,%eax
955: e8 fc ff ff ff call 956 <path_init+0x117>
95a: 89 e8 mov %ebp,%eax
95c: e8 fc ff ff ff call 95d <path_init+0x11e>
961: 8b 43 14 mov 0x14(%ebx),%eax
964: 8b 53 18 mov 0x18(%ebx),%edx
967: 89 03 mov %eax,(%ebx)
969: 89 53 04 mov %edx,0x4(%ebx)
96c: e9 0c 01 00 00 jmp a7d <path_init+0x23e>
971: 83 f8 9c cmp $0xffffff9c,%eax
974: 75 6a jne 9e0 <path_init+0x1a1>
976: a1 00 00 00 00 mov 0x0,%eax
97b: 83 e7 40 and $0x40,%edi
97e: 74 43 je 9c3 <path_init+0x184>
980: 8b b0 54 02 00 00 mov 0x254(%eax),%esi
986: b8 00 00 00 00 mov $0x0,%eax
98b: e8 fc ff ff ff call 98c <path_init+0x14d>
990: 8b 4e 04 mov 0x4(%esi),%ecx
993: f6 c1 01 test $0x1,%cl
996: 74 04 je 99c <path_init+0x15d>
998: f3 90 pause
99a: eb f4 jmp 990 <path_init+0x151>
99c: 8b 56 1c mov 0x1c(%esi),%edx
99f: 8b 46 18 mov 0x18(%esi),%eax
9a2: 89 53 04 mov %edx,0x4(%ebx)
9a5: 89 03 mov %eax,(%ebx)
9a7: 8b 53 04 mov 0x4(%ebx),%edx
9aa: 8b 42 04 mov 0x4(%edx),%eax
9ad: a8 01 test $0x1,%al
9af: 74 04 je 9b5 <path_init+0x176>
9b1: f3 90 pause
9b3: eb f5 jmp 9aa <path_init+0x16b>
9b5: 89 43 24 mov %eax,0x24(%ebx)
9b8: 39 4e 04 cmp %ecx,0x4(%esi)
9bb: 0f 84 bc 00 00 00 je a7d <path_init+0x23e>
9c1: eb cd jmp 990 <path_init+0x151>
9c3: 8b 80 54 02 00 00 mov 0x254(%eax),%eax
9c9: 8b 50 1c mov 0x1c(%eax),%edx
9cc: 8b 40 18 mov 0x18(%eax),%eax
9cf: 89 03 mov %eax,(%ebx)
9d1: 89 53 04 mov %edx,0x4(%ebx)
9d4: 89 d8 mov %ebx,%eax
9d6: e8 fc ff ff ff call 9d7 <path_init+0x198>
9db: e9 87 00 00 00 jmp a67 <path_init+0x228>
9e0: 8d 54 24 08 lea 0x8(%esp),%edx
9e4: e8 fc ff ff ff call 9e5 <path_init+0x1a6>
9e9: 8b 54 24 08 mov 0x8(%esp),%edx
9ed: 89 c6 mov %eax,%esi
9ef: 89 14 24 mov %edx,(%esp)
9f2: 85 c0 test %eax,%eax
9f4: 0f 84 90 00 00 00 je a8a <path_init+0x24b>
9fa: 8b 40 0c mov 0xc(%eax),%eax
9fd: 80 7d 00 00 cmpb $0x0,0x0(%ebp)
a01: 74 2d je a30 <path_init+0x1f1>
a03: 8b 40 20 mov 0x20(%eax),%eax
a06: f6 40 02 02 testb $0x2,0x2(%eax)
a0a: 75 24 jne a30 <path_init+0x1f1>
a0c: 8b 48 18 mov 0x18(%eax),%ecx
a0f: 83 39 00 cmpl $0x0,(%ecx)
a12: 0f 84 8f 00 00 00 je aa7 <path_init+0x268>
a18: 66 83 48 02 02 orw $0x2,0x2(%eax)
a1d: eb 11 jmp a30 <path_init+0x1f1>
a1f: 89 f0 mov %esi,%eax
a21: e8 fc ff ff ff call a22 <path_init+0x1e3>
a26: b8 ec ff ff ff mov $0xffffffec,%eax
a2b: e9 86 00 00 00 jmp ab6 <path_init+0x277>
a30: 8b 46 08 mov 0x8(%esi),%eax
a33: 8b 56 0c mov 0xc(%esi),%edx
a36: 89 53 04 mov %edx,0x4(%ebx)
a39: 89 03 mov %eax,(%ebx)
a3b: 83 e7 40 and $0x40,%edi
a3e: 74 29 je a69 <path_init+0x22a>
a40: 83 3c 24 00 cmpl $0x0,(%esp)
a44: 74 06 je a4c <path_init+0x20d>
a46: 8b 44 24 24 mov 0x24(%esp),%eax
a4a: 89 30 mov %esi,(%eax)
a4c: 8b 53 04 mov 0x4(%ebx),%edx
a4f: 8b 42 04 mov 0x4(%edx),%eax
a52: a8 01 test $0x1,%al
a54: 74 04 je a5a <path_init+0x21b>
a56: f3 90 pause
a58: eb f5 jmp a4f <path_init+0x210>
a5a: 89 43 24 mov %eax,0x24(%ebx)
a5d: b8 00 00 00 00 mov $0x0,%eax
a62: e8 fc ff ff ff call a63 <path_init+0x224>
a67: eb 14 jmp a7d <path_init+0x23e>
a69: 89 d8 mov %ebx,%eax
a6b: e8 fc ff ff ff call a6c <path_init+0x22d>
a70: 83 3c 24 00 cmpl $0x0,(%esp)
a74: 74 07 je a7d <path_init+0x23e>
a76: 89 f0 mov %esi,%eax
a78: e8 fc ff ff ff call a79 <path_init+0x23a>
a7d: 8b 43 04 mov 0x4(%ebx),%eax
a80: 8b 40 20 mov 0x20(%eax),%eax
a83: 89 43 1c mov %eax,0x1c(%ebx)
a86: 31 c0 xor %eax,%eax
a88: eb 2c jmp ab6 <path_init+0x277>
a8a: b8 f7 ff ff ff mov $0xfffffff7,%eax
a8f: eb 25 jmp ab6 <path_init+0x277>
a91: ba 01 00 00 00 mov $0x1,%edx
a96: 89 f0 mov %esi,%eax
a98: e8 fc ff ff ff call a99 <path_init+0x25a>
a9d: 85 c0 test %eax,%eax
a9f: 0f 84 f1 fd ff ff je 896 <path_init+0x57>
aa5: eb 0f jmp ab6 <path_init+0x277>
aa7: 83 3c 24 00 cmpl $0x0,(%esp)
aab: 0f 84 75 ff ff ff je a26 <path_init+0x1e7>
ab1: e9 69 ff ff ff jmp a1f <path_init+0x1e0>
ab6: 83 c4 0c add $0xc,%esp
ab9: 5b pop %ebx
aba: 5e pop %esi
abb: 5f pop %edi
abc: 5d pop %ebp
abd: c3 ret
x86-64:
0000000000002406 <path_init>:
2406: e8 00 00 00 00 callq 240b <get_write_access+0x240b>
240b: 55 push %rbp
240c: 89 d0 mov %edx,%eax
240e: 48 89 e5 mov %rsp,%rbp
2411: 41 57 push %r15
2413: 80 cc 10 or $0x10,%ah
2416: 41 56 push %r14
2418: 49 89 f6 mov %rsi,%r14
241b: 41 55 push %r13
241d: 41 89 d5 mov %edx,%r13d
2420: 41 54 push %r12
2422: 53 push %rbx
2423: 48 89 cb mov %rcx,%rbx
2426: 48 83 ec 28 sub $0x28,%rsp
242a: f6 c6 20 test $0x20,%dh
242d: c7 41 40 01 00 00 00 movl $0x1,0x40(%rcx)
2434: 89 41 38 mov %eax,0x38(%rcx)
2437: c7 41 44 00 00 00 00 movl $0x0,0x44(%rcx)
243e: 0f 84 9b 00 00 00 je 24df <get_write_access+0x24df>
2444: 48 8b 41 28 mov 0x28(%rcx),%rax
2448: 80 3e 00 cmpb $0x0,(%rsi)
244b: 4c 8b 60 30 mov 0x30(%rax),%r12
244f: 74 40 je 2491 <get_write_access+0x2491>
2451: 41 f6 44 24 02 02 testb $0x2,0x2(%r12)
2457: 0f 85 e4 02 00 00 jne 2741 <get_write_access+0x2741>
245d: 49 8b 44 24 20 mov 0x20(%r12),%rax
2462: 48 83 38 00 cmpq $0x0,(%rax)
2466: 75 05 jne 246d <get_write_access+0x246d>
2468: e9 5a 02 00 00 jmpq 26c7 <get_write_access+0x26c7>
246d: 4d 8d b4 24 88 00 00 lea 0x88(%r12),%r14
2474: 00
2475: 4c 89 f7 mov %r14,%rdi
2478: e8 00 00 00 00 callq 247d <get_write_access+0x247d>
247d: 4c 89 f7 mov %r14,%rdi
2480: 66 41 83 4c 24 02 02 orw $0x2,0x2(%r12)
2487: e8 00 00 00 00 callq 248c <get_write_access+0x248c>
248c: e9 b0 02 00 00 jmpq 2741 <get_write_access+0x2741>
2491: 48 8b 43 20 mov 0x20(%rbx),%rax
2495: 41 80 e5 40 and $0x40,%r13b
2499: 4c 89 63 30 mov %r12,0x30(%rbx)
249d: 48 89 03 mov %rax,(%rbx)
24a0: 48 8b 43 28 mov 0x28(%rbx),%rax
24a4: 48 89 43 08 mov %rax,0x8(%rbx)
24a8: 74 28 je 24d2 <get_write_access+0x24d2>
24aa: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
24b1: e8 00 00 00 00 callq 24b6 <get_write_access+0x24b6>
24b6: e8 00 00 00 00 callq 24bb <get_write_access+0x24bb>
24bb: 48 8b 53 08 mov 0x8(%rbx),%rdx
24bf: 8b 42 04 mov 0x4(%rdx),%eax
24c2: a8 01 test $0x1,%al
24c4: 74 04 je 24ca <get_write_access+0x24ca>
24c6: f3 90 pause
24c8: eb f5 jmp 24bf <get_write_access+0x24bf>
24ca: 89 43 3c mov %eax,0x3c(%rbx)
24cd: e9 64 02 00 00 jmpq 2736 <get_write_access+0x2736>
24d2: 48 89 df mov %rbx,%rdi
24d5: e8 00 00 00 00 callq 24da <get_write_access+0x24da>
24da: e9 57 02 00 00 jmpq 2736 <get_write_access+0x2736>
24df: 48 c7 41 20 00 00 00 movq $0x0,0x20(%rcx)
24e6: 00
24e7: 80 3e 2f cmpb $0x2f,(%rsi)
24ea: 0f 85 c3 00 00 00 jne 25b3 <get_write_access+0x25b3>
24f0: 41 80 e5 40 and $0x40,%r13b
24f4: 74 5e je 2554 <get_write_access+0x2554>
24f6: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
24fd: e8 00 00 00 00 callq 2502 <get_write_access+0x2502>
2502: e8 00 00 00 00 callq 2507 <get_write_access+0x2507>
2507: 48 83 7b 20 00 cmpq $0x0,0x20(%rbx)
250c: 0f 85 8d 00 00 00 jne 259f <get_write_access+0x259f>
2512: 65 48 8b 04 25 00 00 mov %gs:0x0,%rax
2519: 00 00
251b: 48 8b 80 58 05 00 00 mov 0x558(%rax),%rax
2522: 8b 50 08 mov 0x8(%rax),%edx
2525: f6 c2 01 test $0x1,%dl
2528: 74 04 je 252e <get_write_access+0x252e>
252a: f3 90 pause
252c: eb f4 jmp 2522 <get_write_access+0x2522>
252e: 48 8b 48 18 mov 0x18(%rax),%rcx
2532: 48 89 4b 20 mov %rcx,0x20(%rbx)
2536: 48 8b 70 20 mov 0x20(%rax),%rsi
253a: 48 89 73 28 mov %rsi,0x28(%rbx)
253e: 8b 4e 04 mov 0x4(%rsi),%ecx
2541: f6 c1 01 test $0x1,%cl
2544: 74 04 je 254a <get_write_access+0x254a>
2546: f3 90 pause
2548: eb f4 jmp 253e <get_write_access+0x253e>
254a: 89 4b 3c mov %ecx,0x3c(%rbx)
254d: 39 50 08 cmp %edx,0x8(%rax)
2550: 74 4d je 259f <get_write_access+0x259f>
2552: eb ce jmp 2522 <get_write_access+0x2522>
2554: 65 48 8b 04 25 00 00 mov %gs:0x0,%rax
255b: 00 00
255d: 4c 8b a0 58 05 00 00 mov 0x558(%rax),%r12
2564: 4c 8d 69 20 lea 0x20(%rcx),%r13
2568: 4d 8d 74 24 04 lea 0x4(%r12),%r14
256d: 4c 89 f7 mov %r14,%rdi
2570: e8 00 00 00 00 callq 2575 <get_write_access+0x2575>
2575: 49 8b 44 24 18 mov 0x18(%r12),%rax
257a: 4c 89 ef mov %r13,%rdi
257d: 48 89 43 20 mov %rax,0x20(%rbx)
2581: 49 8b 44 24 20 mov 0x20(%r12),%rax
2586: 48 89 43 28 mov %rax,0x28(%rbx)
258a: e8 00 00 00 00 callq 258f <get_write_access+0x258f>
258f: 4c 89 f7 mov %r14,%rdi
2592: e8 00 00 00 00 callq 2597 <get_write_access+0x2597>
2597: 4c 89 ef mov %r13,%rdi
259a: e8 00 00 00 00 callq 259f <get_write_access+0x259f>
259f: 48 8b 43 20 mov 0x20(%rbx),%rax
25a3: 48 89 03 mov %rax,(%rbx)
25a6: 48 8b 43 28 mov 0x28(%rbx),%rax
25aa: 48 89 43 08 mov %rax,0x8(%rbx)
25ae: e9 77 01 00 00 jmpq 272a <get_write_access+0x272a>
25b3: 83 ff 9c cmp $0xffffff9c,%edi
25b6: 0f 85 94 00 00 00 jne 2650 <get_write_access+0x2650>
25bc: 65 48 8b 04 25 00 00 mov %gs:0x0,%rax
25c3: 00 00
25c5: 41 80 e5 40 and $0x40,%r13b
25c9: 4c 8b a0 58 05 00 00 mov 0x558(%rax),%r12
25d0: 74 4b je 261d <get_write_access+0x261d>
25d2: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
25d9: e8 00 00 00 00 callq 25de <get_write_access+0x25de>
25de: e8 00 00 00 00 callq 25e3 <get_write_access+0x25e3>
25e3: 41 8b 44 24 08 mov 0x8(%r12),%eax
25e8: a8 01 test $0x1,%al
25ea: 74 04 je 25f0 <get_write_access+0x25f0>
25ec: f3 90 pause
25ee: eb f3 jmp 25e3 <get_write_access+0x25e3>
25f0: 49 8b 54 24 28 mov 0x28(%r12),%rdx
25f5: 48 89 13 mov %rdx,(%rbx)
25f8: 49 8b 4c 24 30 mov 0x30(%r12),%rcx
25fd: 48 89 4b 08 mov %rcx,0x8(%rbx)
2601: 8b 51 04 mov 0x4(%rcx),%edx
2604: f6 c2 01 test $0x1,%dl
2607: 74 04 je 260d <get_write_access+0x260d>
2609: f3 90 pause
260b: eb f4 jmp 2601 <get_write_access+0x2601>
260d: 89 53 3c mov %edx,0x3c(%rbx)
2610: 41 39 44 24 08 cmp %eax,0x8(%r12)
2615: 0f 84 0f 01 00 00 je 272a <get_write_access+0x272a>
261b: eb c6 jmp 25e3 <get_write_access+0x25e3>
261d: 4d 8d 6c 24 04 lea 0x4(%r12),%r13
2622: 4c 89 ef mov %r13,%rdi
2625: e8 00 00 00 00 callq 262a <get_write_access+0x262a>
262a: 49 8b 44 24 28 mov 0x28(%r12),%rax
262f: 48 89 df mov %rbx,%rdi
2632: 48 89 03 mov %rax,(%rbx)
2635: 49 8b 44 24 30 mov 0x30(%r12),%rax
263a: 48 89 43 08 mov %rax,0x8(%rbx)
263e: e8 00 00 00 00 callq 2643 <get_write_access+0x2643>
2643: 4c 89 ef mov %r13,%rdi
2646: e8 00 00 00 00 callq 264b <get_write_access+0x264b>
264b: e9 da 00 00 00 jmpq 272a <get_write_access+0x272a>
2650: 48 8d 75 cc lea -0x34(%rbp),%rsi
2654: 4c 89 45 b8 mov %r8,-0x48(%rbp)
2658: e8 00 00 00 00 callq 265d <get_write_access+0x265d>
265d: 44 8b 7d cc mov -0x34(%rbp),%r15d
2661: 48 85 c0 test %rax,%rax
2664: 49 89 c4 mov %rax,%r12
2667: 4c 8b 45 b8 mov -0x48(%rbp),%r8
266b: 0f 84 c9 00 00 00 je 273a <get_write_access+0x273a>
2671: 41 80 3e 00 cmpb $0x0,(%r14)
2675: 48 8b 50 18 mov 0x18(%rax),%rdx
2679: 74 56 je 26d1 <get_write_access+0x26d1>
267b: 4c 8b 72 30 mov 0x30(%rdx),%r14
267f: 41 f6 46 02 02 testb $0x2,0x2(%r14)
2684: 75 4b jne 26d1 <get_write_access+0x26d1>
2686: 49 8b 56 20 mov 0x20(%r14),%rdx
268a: 48 83 3a 00 cmpq $0x0,(%rdx)
268e: 0f 84 c4 00 00 00 je 2758 <get_write_access+0x2758>
2694: 49 8d 96 88 00 00 00 lea 0x88(%r14),%rdx
269b: 48 89 d7 mov %rdx,%rdi
269e: 48 89 55 b0 mov %rdx,-0x50(%rbp)
26a2: e8 00 00 00 00 callq 26a7 <get_write_access+0x26a7>
26a7: 66 41 83 4e 02 02 orw $0x2,0x2(%r14)
26ad: 48 8b 55 b0 mov -0x50(%rbp),%rdx
26b1: 48 89 d7 mov %rdx,%rdi
26b4: e8 00 00 00 00 callq 26b9 <get_write_access+0x26b9>
26b9: 4c 8b 45 b8 mov -0x48(%rbp),%r8
26bd: eb 12 jmp 26d1 <get_write_access+0x26d1>
26bf: 4c 89 e7 mov %r12,%rdi
26c2: e8 00 00 00 00 callq 26c7 <get_write_access+0x26c7>
26c7: b8 ec ff ff ff mov $0xffffffec,%eax
26cc: e9 95 00 00 00 jmpq 2766 <get_write_access+0x2766>
26d1: 49 8b 54 24 10 mov 0x10(%r12),%rdx
26d6: 41 80 e5 40 and $0x40,%r13b
26da: 48 89 13 mov %rdx,(%rbx)
26dd: 49 8b 54 24 18 mov 0x18(%r12),%rdx
26e2: 48 89 53 08 mov %rdx,0x8(%rbx)
26e6: 74 2d je 2715 <get_write_access+0x2715>
26e8: 45 85 ff test %r15d,%r15d
26eb: 74 03 je 26f0 <get_write_access+0x26f0>
26ed: 4d 89 20 mov %r12,(%r8)
26f0: 48 8b 53 08 mov 0x8(%rbx),%rdx
26f4: 8b 42 04 mov 0x4(%rdx),%eax
26f7: a8 01 test $0x1,%al
26f9: 74 04 je 26ff <get_write_access+0x26ff>
26fb: f3 90 pause
26fd: eb f5 jmp 26f4 <get_write_access+0x26f4>
26ff: 89 43 3c mov %eax,0x3c(%rbx)
2702: 48 c7 c7 00 00 00 00 mov $0x0,%rdi
2709: e8 00 00 00 00 callq 270e <get_write_access+0x270e>
270e: e8 00 00 00 00 callq 2713 <get_write_access+0x2713>
2713: eb 15 jmp 272a <get_write_access+0x272a>
2715: 48 89 df mov %rbx,%rdi
2718: e8 00 00 00 00 callq 271d <get_write_access+0x271d>
271d: 45 85 ff test %r15d,%r15d
2720: 74 08 je 272a <get_write_access+0x272a>
2722: 4c 89 e7 mov %r12,%rdi
2725: e8 00 00 00 00 callq 272a <get_write_access+0x272a>
272a: 48 8b 43 08 mov 0x8(%rbx),%rax
272e: 48 8b 40 30 mov 0x30(%rax),%rax
2732: 48 89 43 30 mov %rax,0x30(%rbx)
2736: 31 c0 xor %eax,%eax
2738: eb 2c jmp 2766 <get_write_access+0x2766>
273a: b8 f7 ff ff ff mov $0xfffffff7,%eax
273f: eb 25 jmp 2766 <get_write_access+0x2766>
2741: be 01 00 00 00 mov $0x1,%esi
2746: 4c 89 e7 mov %r12,%rdi
2749: e8 00 00 00 00 callq 274e <get_write_access+0x274e>
274e: 85 c0 test %eax,%eax
2750: 0f 84 3b fd ff ff je 2491 <get_write_access+0x2491>
2756: eb 0e jmp 2766 <get_write_access+0x2766>
2758: 45 85 ff test %r15d,%r15d
275b: 0f 84 66 ff ff ff je 26c7 <get_write_access+0x26c7>
2761: e9 59 ff ff ff jmpq 26bf <get_write_access+0x26bf>
2766: 48 83 c4 28 add $0x28,%rsp
276a: 5b pop %rbx
276b: 41 5c pop %r12
276d: 41 5d pop %r13
276f: 41 5e pop %r14
2771: 41 5f pop %r15
2773: 5d pop %rbp
2774: c3 retq
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5
2013-06-10 17:42 NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5 Nix
2013-06-11 3:15 ` Al Viro
@ 2013-06-12 1:23 ` Al Viro
2013-06-12 12:08 ` Nix
1 sibling, 1 reply; 7+ messages in thread
From: Al Viro @ 2013-06-12 1:23 UTC (permalink / raw)
To: Nix; +Cc: linux-kernel
On Mon, Jun 10, 2013 at 06:42:49PM +0100, Nix wrote:
> Yes, my shutdown scripts are panicking the kernel again! They're not
> causing filesystem corruption this time, but it's still fs-related.
>
> Here's the 3.9.5 panic, seen on an x86-32 NFS client using NFSv3: NFSv4
> was compiled in but not used. This happened when processes whose
> current directory was on one of those NFS-mounted filesystems were being
> killed, after it had been lazy-umounted (so by this point its cwd was in
> a disconnected mount point).
>
> [ 251.246800] BUG: unable to handle kernel NULL pointer dereference at 00000004
> [ 251.256556] IP: [<c01739f6>] path_init+0xc7/0x27f
> [ 251.256556] *pde = 00000000
> [ 251.256556] Oops: 0000 [#1]
> [ 251.256556] Pid: 748, comm: su Not tainted 3.9.5+ #1
> [ 251.256556] EIP: 0060:[<c01739f6>] EFLAGS: 00010246 CPU: 0
> [ 251.256556] EIP is at path_init+0xc7/0x27f
Apparently that's set_root_rcu() with current->fs being NULL. Which comes from
AF_UNIX connect done by some twisted call chain in context of hell knows what.
> [ 251.256556] [<c02ef8da>] ? unix_stream_connect+0xe1/0x2f7
> [ 251.256556] [<c026a14d>] ? kernel_connect+0x10/0x14
> [ 251.256556] [<c031ecb1>] ? xs_local_connect+0x108/0x181
> [ 251.256556] [<c031c83b>] ? xprt_connect+0xcd/0xd1
> [ 251.256556] [<c031fd1b>] ? __rpc_execute+0x5b/0x156
> [ 251.256556] [<c0128ac2>] ? wake_up_bit+0xb/0x19
> [ 251.256556] [<c031b83d>] ? rpc_run_task+0x55/0x5a
> [ 251.256556] [<c031b8bc>] ? rpc_call_sync+0x7a/0x8d
> [ 251.256556] [<c0325127>] ? rpcb_register_call+0x11/0x20
> [ 251.256556] [<c032548a>] ? rpcb_v4_register+0x87/0xf6
> [ 251.256556] [<c0321187>] ? svc_unregister.isra.22+0x46/0x87
> [ 251.256556] [<c03211d0>] ? svc_rpcb_cleanup+0x8/0x10
> [ 251.256556] [<c03213df>] ? svc_shutdown_net+0x18/0x1b
> [ 251.256556] [<c01cb1f3>] ? lockd_down+0x22/0x97
> [ 251.256556] [<c01c89df>] ? nlmclnt_done+0xc/0x14
> [ 251.256556] [<c01b9064>] ? nfs_free_server+0x7f/0xdb
> [ 251.256556] [<c016e776>] ? deactivate_locked_super+0x16/0x3e
> [ 251.256556] [<c0187e17>] ? free_fs_struct+0x13/0x20
> [ 251.256556] [<c011a009>] ? do_exit+0x224/0x64f
> [ 251.256556] [<c016d51f>] ? vfs_write+0x82/0x108
> [ 251.256556] [<c011a492>] ? do_group_exit+0x3a/0x65
> [ 251.256556] [<c011a4ce>] ? sys_exit_group+0x11/0x11
> [ 251.256556] [<c0332b3d>] ? syscall_call+0x7/0xb
Why is it done in essentially random process context, anyway? There's such thing
as chroot, after all, which would screw that sucker as hard as NULL ->fs, but in
a less visible way...
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5
2013-06-12 1:23 ` Al Viro
@ 2013-06-12 12:08 ` Nix
2013-06-12 15:54 ` Al Viro
0 siblings, 1 reply; 7+ messages in thread
From: Nix @ 2013-06-12 12:08 UTC (permalink / raw)
To: Al Viro; +Cc: linux-kernel, NFS list
On 12 Jun 2013, Al Viro told this:
> On Mon, Jun 10, 2013 at 06:42:49PM +0100, Nix wrote:
>> Yes, my shutdown scripts are panicking the kernel again! They're not
>> causing filesystem corruption this time, but it's still fs-related.
>>
>> Here's the 3.9.5 panic, seen on an x86-32 NFS client using NFSv3: NFSv4
>> was compiled in but not used. This happened when processes whose
>> current directory was on one of those NFS-mounted filesystems were being
>> killed, after it had been lazy-umounted (so by this point its cwd was in
>> a disconnected mount point).
>>
>> [ 251.246800] BUG: unable to handle kernel NULL pointer dereference at 00000004
>> [ 251.256556] IP: [<c01739f6>] path_init+0xc7/0x27f
>> [ 251.256556] *pde = 00000000
>> [ 251.256556] Oops: 0000 [#1]
>> [ 251.256556] Pid: 748, comm: su Not tainted 3.9.5+ #1
>> [ 251.256556] EIP: 0060:[<c01739f6>] EFLAGS: 00010246 CPU: 0
>> [ 251.256556] EIP is at path_init+0xc7/0x27f
>
> Apparently that's set_root_rcu() with current->fs being NULL. Which comes from
> AF_UNIX connect done by some twisted call chain in context of hell knows what.
It's all NFS's fault!
>> [ 251.256556] [<c02ef8da>] ? unix_stream_connect+0xe1/0x2f7
>> [ 251.256556] [<c026a14d>] ? kernel_connect+0x10/0x14
>> [ 251.256556] [<c031ecb1>] ? xs_local_connect+0x108/0x181
>> [ 251.256556] [<c031c83b>] ? xprt_connect+0xcd/0xd1
At this point, we have a sibcall to call_connect() I think. The RPC task
of discourse happens to be local, and as the relevant comment says
* We want the AF_LOCAL connect to be resolved in the
* filesystem namespace of the process making the rpc
* call. Thus we connect synchronously.
Probably this should be doing this only if said namespace isn't
disconnected and going away...
>> [ 251.256556] [<c031fd1b>] ? __rpc_execute+0x5b/0x156
>> [ 251.256556] [<c0128ac2>] ? wake_up_bit+0xb/0x19
>> [ 251.256556] [<c031b83d>] ? rpc_run_task+0x55/0x5a
>> [ 251.256556] [<c031b8bc>] ? rpc_call_sync+0x7a/0x8d
>> [ 251.256556] [<c0325127>] ? rpcb_register_call+0x11/0x20
>> [ 251.256556] [<c032548a>] ? rpcb_v4_register+0x87/0xf6
This is happening because of this code in net/sunrpc/svc.c (and, indeed,
I am running rpcbind, like everyone should be these days):
/*
* If user space is running rpcbind, it should take the v4 UNSET
* and clear everything for this [program, version]. If user space
* is running portmap, it will reject the v4 UNSET, but won't have
* any "inet6" entries anyway. So a PMAP_UNSET should be sufficient
* in this case to clear all existing entries for [program, version].
*/
static void __svc_unregister(struct net *net, const u32 program, const u32 version,
const char *progname)
{
int error;
error = rpcb_v4_register(net, program, version, NULL, "");
/*
* User space didn't support rpcbind v4, so retry this
* request with the legacy rpcbind v2 protocol.
*/
if (error == -EPROTONOSUPPORT)
error = rpcb_register(net, program, version, 0, 0);
Ah yes, because what unregister should do is *register* something.
That's clear as mud :)
> Why is it done in essentially random process context, anyway? There's such thing
> as chroot, after all, which would screw that sucker as hard as NULL ->fs, but in
> a less visible way...
I don't think it is a random process context. It's all intentionally
done in the context of the process which is the last to close that
filesystem, as part of the process of tearing it down -- but it looks
like the NFS svcrpc connection code isn't expecting to be called in that
situation.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5
2013-06-12 12:08 ` Nix
@ 2013-06-12 15:54 ` Al Viro
2013-06-12 21:27 ` Nix
0 siblings, 1 reply; 7+ messages in thread
From: Al Viro @ 2013-06-12 15:54 UTC (permalink / raw)
To: Nix; +Cc: linux-kernel, NFS list
On Wed, Jun 12, 2013 at 01:08:26PM +0100, Nix wrote:
> At this point, we have a sibcall to call_connect() I think. The RPC task
> of discourse happens to be local, and as the relevant comment says
>
> * We want the AF_LOCAL connect to be resolved in the
> * filesystem namespace of the process making the rpc
> * call. Thus we connect synchronously.
>
> Probably this should be doing this only if said namespace isn't
> disconnected and going away...
Namespace, shnamespace... In this case the namespace is alive and well,
it's just that the process is getting killed and it's already past the
point where it has discarded all references to root/cwd.
> > Why is it done in essentially random process context, anyway? There's such thing
> > as chroot, after all, which would screw that sucker as hard as NULL ->fs, but in
> > a less visible way...
>
> I don't think it is a random process context. It's all intentionally
> done in the context of the process which is the last to close that
> filesystem, as part of the process of tearing it down -- but it looks
> like the NFS svcrpc connection code isn't expecting to be called in that
> situation.
_What_? Suppose we have something mounted on /jail/net/foo/bar; will the
effect of process chrooted into /jail doing umount /net/foo/bar be different
from that of process outside of the jail doing umount /jail/net/foo/bar?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5
2013-06-12 15:54 ` Al Viro
@ 2013-06-12 21:27 ` Nix
0 siblings, 0 replies; 7+ messages in thread
From: Nix @ 2013-06-12 21:27 UTC (permalink / raw)
To: Al Viro; +Cc: linux-kernel, NFS list
On 12 Jun 2013, Al Viro outgrape:
> On Wed, Jun 12, 2013 at 01:08:26PM +0100, Nix wrote:
>
>> At this point, we have a sibcall to call_connect() I think. The RPC task
>> of discourse happens to be local, and as the relevant comment says
>>
>> * We want the AF_LOCAL connect to be resolved in the
>> * filesystem namespace of the process making the rpc
>> * call. Thus we connect synchronously.
>>
>> Probably this should be doing this only if said namespace isn't
>> disconnected and going away...
>
> Namespace, shnamespace... In this case the namespace is alive and well,
> it's just that the process is getting killed and it's already past the
> point where it has discarded all references to root/cwd.
Yeah.
>> > Why is it done in essentially random process context, anyway? There's such thing
>> > as chroot, after all, which would screw that sucker as hard as NULL ->fs, but in
>> > a less visible way...
>>
>> I don't think it is a random process context. It's all intentionally
>> done in the context of the process which is the last to close that
>> filesystem, as part of the process of tearing it down -- but it looks
>> like the NFS svcrpc connection code isn't expecting to be called in that
>> situation.
>
> _What_? Suppose we have something mounted on /jail/net/foo/bar; will the
> effect of process chrooted into /jail doing umount /net/foo/bar be different
> from that of process outside of the jail doing umount /jail/net/foo/bar?
Correction: that comment suggests that it was intentionally done. I
didn't write the comment and I make no judgement on whether it makes
sense or not (it looks like it would *normally* make sense, but I guess
nobody thought of the case of a connection being done as part of
disconnection after the cwd is gone).
I'm just the guy getting bitten by the resulting oops :)
--
NULL && (void)
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2013-06-12 21:27 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-06-10 17:42 NFS/lazy-umount/path-lookup-related panics at shutdown (at kill of processes on lazy-umounted filesystems) with 3.9.2 and 3.9.5 Nix
2013-06-11 3:15 ` Al Viro
2013-06-11 11:11 ` Nix
2013-06-12 1:23 ` Al Viro
2013-06-12 12:08 ` Nix
2013-06-12 15:54 ` Al Viro
2013-06-12 21:27 ` Nix
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).