linux-fsdevel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Darrick J. Wong" <djwong@kernel.org>
To: xfs <linux-xfs@vger.kernel.org>
Cc: linux-fsdevel <linux-fsdevel@vger.kernel.org>,
	Kent Overstreet <kent.overstreet@linux.dev>
Subject: dentry UAF bugs crashing arm64 machines on 6.5/6.6?
Date: Tue, 12 Sep 2023 10:30:26 -0700	[thread overview]
Message-ID: <20230912173026.GA3389127@frogsfrogsfrogs> (raw)

Hi everyone,

Shortly after 6.5 was tagged, I started seeing the following stacktraces
when running xfs through fstests on arm64.  Curiously, x86_64 does not
seem affected.

At first I thought this might be caused by the bug fixes in my
development tree, so I started bisecting them.  Bisecting identified a
particular patchset[1] that didn't seem like it was the culprit.  A
couple of days later, one of my arm64 vms with that patch reverted
crashed in the same way.  So, clearly not the culprit.

[1] https://lore.kernel.org/linux-xfs/169335056933.3525521.6054773682023937525.stgit@frogsfrogsfrogs/

  run fstests generic/162 at 2023-09-11 22:06:42
  spectre-v4 mitigation disabled by command-line option
  XFS (sda2): EXPERIMENTAL metadata directory feature in use. Use at your own risk!
  XFS (sda2): EXPERIMENTAL realtime allocation group feature in use. Use at your own risk!
  XFS (sda2): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
  XFS (sda2): Mounting V5 Filesystem ec3b0e05-f8c2-4fe5-a6cf-d8c5819c6ee7
  XFS (sda2): Ending clean mount
  XFS (sda3): EXPERIMENTAL metadata directory feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL realtime allocation group feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
  XFS (sda3): Mounting V5 Filesystem 0c328923-fa81-458f-846c-45bb6828aa0f
  XFS (sda3): Ending clean mount
  XFS (sda3): Quotacheck needed: Please wait.
  XFS (sda3): Quotacheck: Done.
  XFS (sda3): Unmounting Filesystem 0c328923-fa81-458f-846c-45bb6828aa0f
  XFS (sda3): EXPERIMENTAL metadata directory feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL realtime allocation group feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
  XFS (sda3): Mounting V5 Filesystem 287e61f0-a509-4dd1-b788-70076dda0efd
  XFS (sda3): Ending clean mount
  XFS (sda3): Quotacheck needed: Please wait.
  XFS (sda3): Quotacheck: Done.
  XFS (sda3): Unmounting Filesystem 287e61f0-a509-4dd1-b788-70076dda0efd
  XFS (sda3): EXPERIMENTAL metadata directory feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL realtime allocation group feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
  XFS (sda3): Mounting V5 Filesystem 287e61f0-a509-4dd1-b788-70076dda0efd
  XFS (sda3): Ending clean mount
  ==================================================================
  BUG: KASAN: slab-use-after-free in d_alloc_parallel+0x850/0xa00
  Read of size 4 at addr fffffc0176c88360 by task xfs_io/1341106
  
  CPU: 0 PID: 1341106 Comm: xfs_io Not tainted 6.5.0-xfsa #12 7284e9be14b81c73627c489aa5df798cf1143960
  Hardware name: QEMU KVM Virtual Machine, BIOS 1.5.1 06/16/2021
  Call trace:
   dump_backtrace+0x9c/0x100
   show_stack+0x20/0x38
   dump_stack_lvl+0x48/0x60
   print_report+0xf4/0x5b0
   kasan_report+0xa4/0xf0
   __asan_report_load4_noabort+0x20/0x30
   d_alloc_parallel+0x850/0xa00
   __lookup_slow+0x11c/0x2e8
   walk_component+0x1f8/0x498
   link_path_walk.part.0.constprop.0+0x41c/0x960
   path_lookupat+0x70/0x590
   filename_lookup+0x144/0x368
   user_path_at_empty+0x54/0x88
   do_faccessat+0x3f0/0x7c0
   __arm64_sys_faccessat+0x78/0xb0
   do_el0_svc+0x124/0x318
   el0_svc+0x34/0xe8
   el0t_64_sync_handler+0x13c/0x158
   el0t_64_sync+0x190/0x198
  
  Allocated by task 1340914:
   kasan_save_stack+0x2c/0x58
   kasan_set_track+0x2c/0x40
   kasan_save_alloc_info+0x24/0x38
   __kasan_slab_alloc+0x74/0x90
   kmem_cache_alloc_lru+0x180/0x440
   __d_alloc+0x40/0x830
   d_alloc+0x3c/0x1c8
   d_alloc_parallel+0xe4/0xa00
   __lookup_slow+0x11c/0x2e8
   walk_component+0x1f8/0x498
   path_lookupat+0x10c/0x590
   filename_lookup+0x144/0x368
   user_path_at_empty+0x54/0x88
   do_readlinkat+0xcc/0x250
   __arm64_sys_readlinkat+0x90/0xe0
   do_el0_svc+0x124/0x318
   el0_svc+0x34/0xe8
   el0t_64_sync_handler+0x13c/0x158
   el0t_64_sync+0x190/0x198
  
  Last potentially related work creation:
   kasan_save_stack+0x2c/0x58
   __kasan_record_aux_stack+0x9c/0xc8
   kasan_record_aux_stack_noalloc+0x14/0x20
   __call_rcu_common.constprop.0+0x74/0x5b0
   call_rcu+0x18/0x30
   dentry_free+0x9c/0x158
   __dentry_kill+0x434/0x578
   dput+0x30c/0x6b8
   step_into+0xa50/0x1680
   walk_component+0xb0/0x498
   path_lookupat+0x10c/0x590
   filename_lookup+0x144/0x368
   user_path_at_empty+0x54/0x88
   do_readlinkat+0xcc/0x250
   __arm64_sys_readlinkat+0x90/0xe0
   do_el0_svc+0x124/0x318
   el0_svc+0x34/0xe8
   el0t_64_sync_handler+0x13c/0x158
   el0t_64_sync+0x190/0x198
  
  Second to last potentially related work creation:
   kasan_save_stack+0x2c/0x58
   __kasan_record_aux_stack+0x9c/0xc8
   kasan_record_aux_stack_noalloc+0x14/0x20
   __call_rcu_common.constprop.0+0x74/0x5b0
   call_rcu+0x18/0x30
   dentry_free+0x9c/0x158
   __dentry_kill+0x434/0x578
   dput+0x30c/0x6b8
   step_into+0xa50/0x1680
   walk_component+0xb0/0x498
   path_lookupat+0x10c/0x590
   filename_lookup+0x144/0x368
   user_path_at_empty+0x54/0x88
   do_readlinkat+0xcc/0x250
   __arm64_sys_readlinkat+0x90/0xe0
   do_el0_svc+0x124/0x318
   el0_svc+0x34/0xe8
   el0t_64_sync_handler+0x13c/0x158
   el0t_64_sync+0x190/0x198
  
  The buggy address belongs to the object at fffffc0176c88340
   which belongs to the cache dentry of size 216
  The buggy address is located 32 bytes inside of
   freed 216-byte region [fffffc0176c88340, fffffc0176c88418)
  
  The buggy address belongs to the physical page:
  page:ffffffff005db200 refcount:1 mapcount:0 mapping:0000000000000000 index:0xfffffc0176c86900 pfn:0x1b6c8
  memcg:fffffc004195a001
  flags: 0x1ffe000000000200(slab|node=0|zone=1|lastcpupid=0x7ff)
  page_type: 0xffffffff()
  raw: 1ffe000000000200 fffffc00e0016b80 ffffffff0052bb90 ffffffff00405350
  raw: fffffc0176c86900 0000000000ea0037 00000001ffffffff fffffc004195a001
  page dumped because: kasan: bad access detected
  
  Memory state around the buggy address:
   fffffc0176c88200: fc fc fc fc fc fb fb fb fb fb fb fb fb fb fb fb
   fffffc0176c88280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
  >fffffc0176c88300: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
                                                         ^
   fffffc0176c88380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
   fffffc0176c88400: fb fb fb fc fc fc fc fc fc fc fc fb fb fb fb fb
  ==================================================================
  Disabling lock debugging due to kernel taint
  XFS (sda2): Unmounting Filesystem ec3b0e05-f8c2-4fe5-a6cf-d8c5819c6ee7
  XFS (sda3): EXPERIMENTAL online scrub feature in use. Use at your own risk!
  XFS (sda3): Unmounting Filesystem 287e61f0-a509-4dd1-b788-70076dda0efd
  memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=1343704 'xfs_repair'
  XFS (sda3): EXPERIMENTAL metadata directory feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL realtime allocation group feature in use. Use at your own risk!
  XFS (sda3): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
  XFS (sda3): Mounting V5 Filesystem 287e61f0-a509-4dd1-b788-70076dda0efd
  XFS (sda3): Ending clean mount
  XFS (sda3): Unmounting Filesystem 287e61f0-a509-4dd1-b788-70076dda0efd
  run fstests generic/137 at 2023-09-11 22:06:59
  spectre-v4 mitigation disabled by command-line option
  XFS (sda2): EXPERIMENTAL metadata directory feature in use. Use at your own risk!
  XFS (sda2): EXPERIMENTAL realtime allocation group feature in use. Use at your own risk!
  XFS (sda2): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
  XFS (sda2): Mounting V5 Filesystem ec3b0e05-f8c2-4fe5-a6cf-d8c5819c6ee7
  XFS (sda2): Ending clean mount
  XFS (sda2): Unmounting Filesystem ec3b0e05-f8c2-4fe5-a6cf-d8c5819c6ee7
  XFS (sda2): EXPERIMENTAL metadata directory feature in use. Use at your own risk!
  XFS (sda2): EXPERIMENTAL realtime allocation group feature in use. Use at your own risk!
  XFS (sda2): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
  XFS (sda2): Mounting V5 Filesystem ec3b0e05-f8c2-4fe5-a6cf-d8c5819c6ee7
  XFS (sda2): Ending clean mount
  Unable to handle kernel paging request at virtual address e0c83a00c0029201
  KASAN: maybe wild-memory-access in range [0x0641e00600149008-0x0641e0060014900f]
  Mem abort info:
    ESR = 0x0000000096000004
    EC = 0x25: DABT (current EL), IL = 32 bits
    SET = 0, FnV = 0
    EA = 0, S1PTW = 0
    FSC = 0x04: level 0 translation fault
  Data abort info:
    ISV = 0, ISS = 0x00000004, ISS2 = 0x00000000
    CM = 0, WnR = 0, TnD = 0, TagAccess = 0
    GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
  [e0c83a00c0029201] address between user and kernel address ranges
  Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
  Dumping ftrace buffer:
     (ftrace buffer empty)
  Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison xfs ext2 mbcache dm_flakey dm_snapshot dm_bufio dm_zero nft_chain_nat xt_REDIRECT nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 rpcsec_gss_krb5 auth_rpcgss xt_tcpudp crct10dif_ce bfq ip_set_hash_ip ip_set_hash_net xt_set nft_compat ip_set_hash_mac nf_tables libcrc32c sch_fq_codel fuse configfs efivarfs ip_tables x_tables overlay nfsv4 [last unloaded: scsi_debug]
  CPU: 1 PID: 1348080 Comm: md5sum Tainted: G    B              6.5.0-xfsa #12 7284e9be14b81c73627c489aa5df798cf1143960
  Hardware name: QEMU KVM Virtual Machine, BIOS 1.5.1 06/16/2021
  pstate: 40401005 (nZcv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)
  pc : d_alloc_parallel+0x250/0xa00
  lr : d_alloc_parallel+0x85c/0xa00
  sp : fffffe008b8cf720
  x29: fffffe008b8cf720 x28: 00c83c00c0029201 x27: fffffc00fa39e340
  x26: 000000003c7f8e47 x25: dffffe0000000000 x24: fffffe0082ab9d90
  x23: 0000000000000003 x22: fffffc014dd9df20 x21: 0641e006001490b0
  x20: 0641e00600149008 x19: fffffe00825a55c0 x18: 0000000000000000
  x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
  x14: 0000000000000001 x13: 80808080807fffff x12: fffffdc0105573b3
  x11: 1fffffc0105573b2 x10: fffffdc0105573b2 x9 : dffffe0000000000
  x8 : 0000023fefaa8c4e x7 : fffffe0082ab9d97 x6 : 0000000000000001
  x5 : fffffe0082ab9d90 x4 : 0000000000000000 x3 : fffffe008070fad4
  x2 : 0000000000000000 x1 : fffffe0082a9e9d8 x0 : 0000000000000000
  Call trace:
   d_alloc_parallel+0x250/0xa00
   path_openat+0x1030/0x23d0
   do_filp_open+0x15c/0x338
   do_sys_openat2+0x12c/0x168
   __arm64_sys_openat+0x138/0x1d0
   do_el0_svc+0x124/0x318
   el0_svc+0x34/0xe8
   el0t_64_sync_handler+0x13c/0x158
   el0t_64_sync+0x190/0x198
  Code: f94002b5 b4001375 d102a2b4 d343fe9c (38f96b80) 
  ---[ end trace 0000000000000000 ]---
  note: md5sum[1348080] exited with preempt_count 1

I tried popping /all/ the bugfixes, and for 6.5 that seemed to make
fstests settle down.  But then 6.6-rc1 came out, and the crashes
returned.  In addition, there's a new crash:

  run fstests xfs/711 at 2023-09-11 20:25:35
  spectre-v4 mitigation disabled by command-line option
  XFS (sda3): Mounting V5 Filesystem add44358-2799-4361-b909-073e50305a53
  XFS (sda3): Ending clean mount
  XFS (sda3): Quotacheck needed: Please wait.
  XFS (sda3): Quotacheck: Done.
  XFS (sda3): EXPERIMENTAL online scrub feature in use. Use at your own risk!
  Unable to handle kernel execute from non-executable memory at virtual address fffffe0082e85070
  KASAN: probably user-memory-access in range [0x0000000417428380-0x0000000417428387]
  Mem abort info:
    ESR = 0x000000008600000f
    EC = 0x21: IABT (current EL), IL = 32 bits
    SET = 0, FnV = 0
    EA = 0, S1PTW = 0
    FSC = 0x0f: level 3 permission fault
  swapper pgtable: 64k pages, 42-bit VAs, pgdp=0000000041d40000
  [fffffe0082e85070] pgd=100000023fff0003, p4d=100000023fff0003, pud=100000023fff0003, pmd=100000023fff0003, pte=0068000042e80703
  Internal error: Oops: 000000008600000f [#1] PREEMPT SMP
  Dumping ftrace buffer:
     (ftrace buffer empty)
  Modules linked in: xfs nft_chain_nat xt_REDIRECT nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 rpcsec_gss_krb5 auth_rpcgss xt_tcpudp ip_set_hash_ip ip_set_hash_net xt_set nft_compat ip_set_hash_mac ip_set bfq nf_tables libcrc32c crct10dif_ce nfnetlink sch_fq_codel fuse configfs efivarfs ip_tables x_tables overlay nfsv4
  CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.6.0-rc1-djwa #rc1 31f6bcd5927f495a7ed31b02fb778f37562278b5
  Hardware name: QEMU KVM Virtual Machine, BIOS 1.5.1 06/16/2021
  pstate: 20401005 (nzCv daif +PAN -UAO -TCO -DIT +SSBS BTYPE=--)
  pc : in_lookup_hashtable+0x16b0/0x2020
  lr : rcu_core+0xadc/0x13a8
  sp : fffffe0082f2fdc0
  x29: fffffe0082f2fdc0 x28: fffffd801c1228a1 x27: fffffc00e0914500
  x26: 0000000000000003 x25: fffffc01bea6aa38 x24: fffffe0082f2fea0
  x23: 1fffffc0105e5fd0 x22: fffffe00826459f8 x21: dffffe0000000000
  x20: 0000000000000003 x19: fffffc01bea6a9c0 x18: 0000000000000000
  x17: fffffe013cac0000 x16: fffffe0082f20000 x15: 0000000000000000
  x14: 0000000000000000 x13: 1fffffc0102ff4f1 x12: fffffdc0104dafc9
  x11: 1fffffc0104dafc8 x10: fffffdc0104dafc8 x9 : fffffe00802188dc
  x8 : fffffd802037eb51 x7 : fffffe0082f2fab8 x6 : 0000000000000002
  x5 : 0000000000000050 x4 : fffffc01bd7f34e8 x3 : 0000000000000118
  x2 : fffffe0082e85070 x1 : fffffc0101bfa078 x0 : fffffc0101bfa078
  Call trace:
   in_lookup_hashtable+0x16b0/0x2020
   rcu_core_si+0x18/0x30
   __do_softirq+0x280/0xad4
   ____do_softirq+0x18/0x30
   call_on_irq_stack+0x24/0x58
   do_softirq_own_stack+0x24/0x38
   irq_exit_rcu+0x198/0x238
   el1_interrupt+0x38/0x58
   el1h_64_irq_handler+0x18/0x28
   el1h_64_irq+0x64/0x68
   arch_local_irq_enable+0x4/0x8
   do_idle+0x32c/0x450
   cpu_startup_entry+0x2c/0x40
   secondary_start_kernel+0x230/0x2e0
   __secondary_switched+0xb8/0xc0
  Code: 00000000 00000000 00000000 00000000 (01bfa078) 
  ---[ end trace 0000000000000000 ]---
  Kernel panic - not syncing: Oops: Fatal exception in interrupt
  SMP: stopping secondary CPUs
  Dumping ftrace buffer:
     (ftrace buffer empty)
  Kernel Offset: disabled
  CPU features: 0x00000000,70020043,1001700b
  Memory Limit: none
  ---[ end Kernel panic - not syncing: Oops: Fatal exception in interrupt ]---

IDGI.  Has anyone else seen this sort of crash?  They all seem to
revolve around UAF bugs with dentries that look like they've been
rcu-freed recently.

Kent said he's affected by some crash on arm64 too, so I cc'd him.

I also haven't any clue why this hasn't triggered at all on x86_64.

--D

             reply	other threads:[~2023-09-12 17:30 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-09-12 17:30 Darrick J. Wong [this message]
2023-09-12 17:48 ` dentry UAF bugs crashing arm64 machines on 6.5/6.6? Kent Overstreet
2023-09-18  1:37 ` Dave Chinner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230912173026.GA3389127@frogsfrogsfrogs \
    --to=djwong@kernel.org \
    --cc=kent.overstreet@linux.dev \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-xfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).