OOPS: v4.19.171 kernfs_sop_show_options - Invalid "root->syscall_ops" ?

[Philipp Hahn <pmhahn@pmhahn.de>]

Hello,

Some time ago one of my VMs crashed running Debian-10-Buster using Linux 
kernel 4.19.171:

> [Feb16 06:50] general protection fault: 0000 [#1] SMP PTI
> [  +0,000065] CPU: 1 PID: 17450 Comm: nfsmounts Not tainted 4.19.0-14-amd64 #1 Debian 4.19.171-2
> [  +0,000052] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.2-1 04/01/2014
> [  +0,000067] RIP: 0010:kernfs_sop_show_options+0x2d/0x40
> [  +0,000035] Code: 00 00 48 8b 46 30 48 85 c0 74 07 48 8b 80 50 02 00 00 48 8b 50 08 48 85 d2 48 0f 44 d0 48 8b 72 50 48 8b 46 30 48 85 c0 74 0e <48> 8b 40 08 48 85 c0 74 05 e9 05 57 72 00 31 c0 c3 66 90 0f 1f 44
> [  +0,000106] RSP: 0018:ffff9e5f01fffe20 EFLAGS: 00010202
> [  +0,000032] RAX: 0105028102951075 RBX: ffff8b0c7af66680 RCX: 00000000656f6e2c
> [  +0,000041] RDX: ffff8b0c351de440 RSI: ffff8b0c34a74840 RDI: ffff8b0c7af66680
> [  +0,000040] RBP: ffff8b0c34286c20 R08: 6d6974616c65722c R09: 656d6974616c6572
> [  +0,000041] R10: ffff9e5f01fffddc R11: ffff8b0c75e123bc R12: ffff8b0c7cfec000
> [  +0,000040] R13: ffff8b0c77e50e80 R14: ffff8b0c7abb4900 R15: ffff8b0c7af66680
> [  +0,000042] FS:  00007ff4bb68e740(0000) GS:ffff8b0c7db00000(0000) knlGS:0000000000000000
> [  +0,000045] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  +0,000033] CR2: 00007ff4bb634e10 CR3: 000000007ac0c000 CR4: 00000000000006e0
> [  +0,000069] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  +0,000041] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [  +0,000041] Call Trace:
> [  +0,000053]  show_vfsmnt+0x124/0x170
> [  +0,000038]  seq_read+0x2e9/0x410
> [  +0,000026]  vfs_read+0x91/0x140
> [  +0,000023]  ksys_read+0x57/0xd0
> [  +0,000058]  do_syscall_64+0x53/0x110
> [  +0,000051]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [  +0,000053] RIP: 0033:0x7ff4bb77d461
> [  +0,000058] Code: fe ff ff 50 48 8d 3d fe d0 09 00 e8 e9 03 02 00 66 0f 1f 84 00 00 00 00 00 48 8d 05 99 62 0d 00 8b 00 85 c0 75 13 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 57 c3 66 0f 1f 44 00 00 41 54 49 89 d4 55 48
> [  +0,000135] RSP: 002b:00007ffc1c076e68 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
> [  +0,000067] RAX: ffffffffffffffda RBX: 000055f6fceab250 RCX: 00007ff4bb77d461
> [  +0,000070] RDX: 0000000000002000 RSI: 000055f6fd162e10 RDI: 0000000000000004
> [  +0,000096] RBP: 00007ff4bb84b2a0 R08: 0000000000000b40 R09: 000055f6fceab250
> [  +0,000089] R10: 000055f6fcda0010 R11: 0000000000000246 R12: 0000000000002000
> [  +0,000093] R13: 000055f6fd162e10 R14: 0000000000000d68 R15: 00007ff4bb84a760
> [  +0,000100] Modules linked in: ipt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_state xt_conntrack nft_compat nft_counter nft_chain_route_ipv6 nft_chain_rou
> [  +0,009114]  raid1 raid0 multipath linear md_mod dm_snapshot dm_bufio dm_mirror dm_region_hash dm_log dm_mod hid_generic usbhid hid sr_mod cdrom sd_mod virtio_gpu ttm drm_kms_helper ahci libahci drm virtio_net net_failover failover virtio_scsi xhci_pci libata xhci_hcd 
> [  +0,009397] ---[ end trace b63534c6f4770b73 ]---

Here's the dis-assembly of said code:

# scripts/decodecode
>   0:   INCOMPLETE
>   2:   48 8b 46 30             mov    0x30(%rsi),%rax
>   6:   48 85 c0                test   %rax,%rax
>   9:   74 07                   je     0x12
>   b:   48 8b 80 50 02 00 00    mov    0x250(%rax),%rax
>  12:   48 8b 50 08             mov    0x8(%rax),%rdx
>  16:   48 85 d2                test   %rdx,%rdx
>  19:   48 0f 44 d0             cmove  %rax,%rdx
>  1d:   48 8b 72 50             mov    0x50(%rdx),%rsi
>  21:   48 8b 46 30             mov    0x30(%rsi),%rax
>  25:   48 85 c0                test   %rax,%rax
>  28:   74 0e                   je     0x38
>  2a:*  48 8b 40 08             mov    0x8(%rax),%rax           <-- trapping instruction
>  2e:   48 85 c0                test   %rax,%rax
>  31:   74 05                   je     0x38
>  33:   e9 05 57 72 00          jmpq   0x72573d
>  38:   31 c0                   xor    %eax,%eax
>  3a:   c3                      retq   
>  3b:   STRIPPED

# objdump -S fs/kernfs/mount.o --start-address=0x30 --stop-address=0x60
> 0000000000000030 <kernfs_sop_show_options>:
> static int kernfs_sop_show_options(struct seq_file *sf, struct dentry *dentry)
> {
>   00:   e8 00 00 00 00          callq  35 <kernfs_sop_show_options+0x5>
> };
> #define kernfs_info(SB) ((struct kernfs_super_info *)(SB->s_fs_info))
> static inline struct kernfs_node *kernfs_dentry_node(struct dentry *dentry)
> {
>         if (d_really_is_negative(dentry))
>   05:   48 8b 46 30             mov    0x30(%rsi),%rax
>   09:   48 85 c0                test   %rax,%rax
>   0c:   74 07                   je     45 <kernfs_sop_show_options+0x15>
>                 return NULL;
>         return d_inode(dentry)->i_private;
>   0e:   48 8b 80 50 02 00 00    mov    0x250(%rax),%rax
>         if (kn->parent)
>   15:   48 8b 50 08             mov    0x8(%rax),%rdx
=== RDX: ffff8b0c351de440 -> kn->parent
>   19:   48 85 d2                test   %rdx,%rdx
>   1c:   48 0f 44 d0             cmove  %rax,%rdx
>         return kn->dir.root;
    20:   48 8b 72 50             mov    0x50(%rdx),%rsi
=== RSI: ffff8b0c34a74840 -> root
>         struct kernfs_root *root = kernfs_root(kernfs_dentry_node(dentry));
>         struct kernfs_syscall_ops *scops = root->syscall_ops;
>   24:   48 8b 46 30             mov    0x30(%rsi),%rax
=== RAX: ffffffffffffffda -> root->syscall_ops ?= -ENOSYS ???
>         if (scops && scops->show_options)
>   28:   48 85 c0                test   %rax,%rax
>   2b:   74 0e                   je     6b <kernfs_sop_show_options+0x3b>
>   2d:   48 8b 40 08             mov    0x8(%rax),%rax
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   31:   48 85 c0                test   %rax,%rax
>   34:   74 05                   je     6b <kernfs_sop_show_options+0x3b>
>                 return scops->show_options(sf, root);
>   36:   e9 00 00 00 00          jmpq   6b <kernfs_sop_show_options+0x3b>
>         return 0;
> }
>   3b:   31 c0                   xor    %eax,%eax
>   3d:   c3                      retq   


I don't know how to trigger that again, but here are my findings to far:

- The bug seems to be triggered by a Python script 
`/usr/lib/univention-directory-policy/nfsmounts` reading 
`/proc/self/mounts` AKA `/etc/mtab`.

- `syscall_ops` seems to be invalid, which is only ever initialized by 
fs/kernfs/dir.c:kernfs_create_root()

- This function seems to be used for every cgroup by 
kernel/cgroup/cgroup.c:cgroup_setup_root()


Does that ring any bells?

I had a look at some changes between v4.19.171...HEAD in that area and 
have found the following changes, which might be interesting - but I'm 
no Kernel guru
- 630faf81b3e6 cgroup: don't put ERR_PTR() into fc->root
- 35ac1184244f cgroup: saner refcounting for cgroup_root
- 399504e21a10 fix cgroup_do_mount() handling of failure exits
- 2fd60da46da7 kernfs: fix potential null pointer dereference

Debian and many other LTS users would probably like to get this fixed 
for stable/linux-4.19.y.
I'm willing to spend more time on this, but currently I'm out of ideads 
what else to do or where else to look.

Thank you in advance

Philipp Hahn