Re: Crash with PREEMPT_RT on aarch64 machine

[Pierre Gondois <pierre.gondois@arm.com>]

On 11/28/22 16:58, Sebastian Andrzej Siewior wrote:
> How about this?
> 
> - The fast path is easy…
> 
> - The slow path first sets the WAITER bits (mark_rt_mutex_waiters()) so
>    I made that one _acquire so that it is visible by the unlocker forcing
>    everyone into slow path.
> 
> - If the lock is acquired, then the owner is written via
>    rt_mutex_set_owner(). This happens under wait_lock so it is
>    serialized and so a WRITE_ONCE() is used to write the final owner. I
>    replaced it with a cmpxchg_acquire() to have the owner there.
>    Not sure if I shouldn't make this as you put it:
> |   e.g. by making use of dependency ordering where it already exists.
>    The other (locking) CPU needs to see the owner not only the WAITER
>    bit. I'm not sure if this could be replaced with smp_store_release().
> 
> - After the whole operation completes, fixup_rt_mutex_waiters() cleans
>    the WAITER bit and I kept the _acquire semantic here. Now looking at
>    it again, I don't think that needs to be done since that shouldn't be
>    set here.
> 
> - There is rtmutex_spin_on_owner() which (as the name implies) spins on
>    the owner as long as it active. It obtains it via READ_ONCE() and I'm
>    not sure if any memory barrier is needed. Worst case is that it will
>    spin while owner isn't set if it observers a stale value.
> 
> - The unlock path first clears the waiter bit if there are no waiters
>    recorded (via simple assignments under the wait_lock (every locker
>    will fail with the cmpxchg_acquire() and go for the wait_lock)) and
>    then finally drop it via rt_mutex_cmpxchg_release(,, NULL).
>    Should there be a wait, it will just store the WAITER bit with
>    smp_store_release() (setting the owner is NULL but the WAITER bit
>    forces everyone into the slow path).
> 
> - Added rt_mutex_set_owner_pi() which does simple assignment. This is
>    used from the futex code and here everything happens under a lock.
> 
> - I added a smp_load_acquire() to rt_mutex_base_is_locked() since I
>    *think* want to observe a real waiter and not something stale.
> 
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>


Hello,
Just to share some debug attempts, I tried Sebastian's patch and could not
reproduce the error. While trying to understand the solution, I could not
reproduce the error if I only took the changes made to
mark_rt_mutex_waiters(), or to rt_mutex_set_owner_pi(). I am not sure I
understand why this would be a rt-mutex issue.

Without Sebastian's patch, to try adding some synchronization around the
'i_wb_list', I did the following:

diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 443f83382b9b..42ce1f7f8aef 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -1271,10 +1271,10 @@ void sb_clear_inode_writeback(struct inode *inode)
         struct super_block *sb = inode->i_sb;
         unsigned long flags;
  
-       if (!list_empty(&inode->i_wb_list)) {
+       if (!list_empty_careful(&inode->i_wb_list)) {
                 spin_lock_irqsave(&sb->s_inode_wblist_lock, flags);
-               if (!list_empty(&inode->i_wb_list)) {
-                       list_del_init(&inode->i_wb_list);
+               if (!list_empty_careful(&inode->i_wb_list)) {
+                       list_del_init_careful(&inode->i_wb_list);
                         trace_sb_clear_inode_writeback(inode);
                 }
                 spin_unlock_irqrestore(&sb->s_inode_wblist_lock, flags);
diff --git a/fs/inode.c b/fs/inode.c
index b608528efd3a..fbe6b4fe5831 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -621,7 +621,7 @@ void clear_inode(struct inode *inode)
         BUG_ON(!list_empty(&inode->i_data.private_list));
         BUG_ON(!(inode->i_state & I_FREEING));
         BUG_ON(inode->i_state & I_CLEAR);
-       BUG_ON(!list_empty(&inode->i_wb_list));
+       BUG_ON(!list_empty_careful(&inode->i_wb_list));
         /* don't need i_lock here, no concurrent mods to i_state */
         inode->i_state = I_FREEING | I_CLEAR;
  }

I never stepped on the:
   BUG_ON(!list_empty_careful(&inode->i_wb_list))
statement again, but got the dump at [2]. I also regularly end-up
with the following endless logs when trying other things, when rebooting:

EXT4-fs (nvme0n1p3): sb orphan head is 2840597
sb_info orphan list:
   inode nvme0n1p3:3958579 at 00000000b5934dff: mode 100664, nlink 1, next 0
   inode nvme0n1p3:3958579 at 00000000b5934dff: mode 100664, nlink 1, next 0
   ...

Also, Jan said that the issue was reproducible on 'two different aarch64
machines', cf [1]. Would it be possible to know which one ?

Regards,
Pierre

[1]
https://lore.kernel.org/all/20221103115444.m2rjglbkubydidts@quack3/

[2]
EXT4-fs (nvme0n1p3): Inode 2834153 (0000000051c7b29b): orphan list check failed!
0000000051c7b29b: 0000f30a 00000004 00000000 00000000  ................
00000000a0792dde: 00000000 00000000 00000000 00000000  ................
0000000065a25e3d: 00000000 00000000 00000000 00000000  ................
00000000b2085d6d: 00000000 00000000 00000000 002b3fe8  .............?+.
0000000088f2c42f: 00000000 00000000 00000159 00000000  ........Y.......
00000000b9d22813: 00080000 00000040 80000000 00000000  ....@...........
0000000004f93ec7: 00000000 00000000 00000000 00000000  ................
00000000d8e00df6: 00000000 00000000 00000000 00000000  ................
00000000d29047af: ba813320 ffff07ff 0d822240 ffff4001   3......@"...@..
00000000e9708e5a: 0d822250 ffff4001 0d822250 ffff4001  P"...@..P"...@..
000000001f08ff8a: 0d822260 ffff4001 0d822260 ffff4001  `"...@..`"...@..
000000003231aba6: 00000000 00000000 00000000 00000000  ................
00000000df5b63ba: 00000000 00000000 00000000 00000000  ................
00000000367f58f3: 00000000 00000000 00000000 00000000  ................
000000008a1af872: 0d8222a0 ffff4001 0d8222a0 ffff4001  ."...@..."...@..
000000002a8dc95e: 00000000 00000000 00000000 00000000  ................
00000000e4c5bdb4: 00000000 00000000 00000000 00000000  ................
000000004f0a738d: 00000000 00000000 80000000 00000000  ................
0000000082a9e5b2: 00000000 00000000 00000000 00000000  ................
0000000064dce462: 00000000 00000000 00000000 00000000  ................
00000000fe106bb0: 000d8180 000040ab 000040ab 00000000  .....@...@......
00000000946bcd55: 00000000 00000000 00000000 00000000  ................
000000001d64e9fd: c1390a80 ffffdab5 87d6c800 ffff07ff  ..9.............
000000002afa83ff: 0d822488 ffff4001 54d25060 ffff4002  .$...@..`P.T.@..
0000000075bfe8c6: 002b3ee9 00000000 00000000 00000000  .>+.............
00000000580feb22: 00000000 00000000 63877e76 00000000  ........v~.c....
00000000cca606aa: 010d8f0e 00000000 63877e76 00000000  ........v~.c....
00000000d446eaef: 010d8f0e 00000000 63877e76 00000000  ........v~.c....
0000000072243536: 010d8f0e 00000000 00000000 00000000  ................
000000007cbeccb9: 00000000 00000000 00000000 00000000  ................
0000000026d5ad72: 00000000 00000000 000c0000 00000000  ................
00000000e28ac20a: 00000000 00000000 00000060 00000000  ........`.......
0000000076ed32fb: 80000000 00000000 00000000 00000000  ................
00000000cd183175: 00000000 00000000 00000000 00000000  ................
00000000ecafc825: 00000000 00000000 00000000 00000000  ................
00000000408fde6f: 00000000 00000000 00000000 00000000  ................
000000004d7c3704: 00000000 00000000 0d822408 ffff4001  .........$...@..
000000007a24c141: 0d822408 ffff4001 0d822418 ffff4001  .$...@...$...@..
000000007ce51788: 0d822418 ffff4001 0d822428 ffff4001  .$...@..($...@..
00000000efd9c162: 0d822428 ffff4001 0d822438 ffff4001  ($...@..8$...@..
0000000013f0626e: 0d822438 ffff4001 00000000 00000000  8$...@..........
00000000e8fc5904: 00000000 00000000 00000002 00000000  ................
000000006533e04b: 00000000 00000000 00000000 00000000  ................
000000009a33c9d5: 00000000 00000000 c1390b40 ffffdab5  ........@.9.....
00000000b743e93e: 00000000 00000000 0d822300 ffff4001  .........#...@..
0000000041c5a701: 00000000 00000000 00000000 00000000  ................
000000009f872e56: 00000000 00000000 00000000 00000000  ................
00000000f6ca0703: 00000021 00000000 00000000 00000000  !...............
00000000e79eacb9: 80000000 00000000 00000000 00000000  ................
000000003afb3989: 00000000 00000000 00000000 00000000  ................
00000000164436d4: 00000000 00000000 00100cca 00000000  ................
00000000d63a3021: 00000000 00000000 00000000 00000000  ................
00000000e0e1ace3: 80000000 00000000 00000000 00000000  ................
0000000043ac9c19: 00000000 00000000 00000000 00000000  ................
00000000f3870564: 00000000 00000000 00000000 00000000  ................
0000000082c87bf9: 00000000 00000000 c1391310 ffffdab5  ..........9.....
00000000f5524c75: 00000010 00000000 00000000 00000000  ................
00000000dfda4192: 00000000 00000000 00000000 00000000  ................
00000000863650dd: 00000000 00000000 00000000 00000000  ................
00000000b709ac61: 0d822570 ffff4001 0d822570 ffff4001  p%...@..p%...@..
000000009f316d71: 00000000 00000000 0d822588 ffff4001  .........%...@..
00000000f15fb4ed: 0d822588 ffff4001 00000000 00000000  .%...@..........
000000000027077a: 6401ffec 00000000 00000000 00000000  ...d............
00000000c828fe47: 00000000 00000000 00000000 00000000  ................
00000000b5f575af: 00000000 00000000 00000000 00000000  ................
000000001466bf98: 00000000 00000000 00000000 00000000  ................
0000000097025855: 00000000 00000000 00000000 00000000  ................
000000002557bcf0: 63877e76 00000000 010d8f0e 00000000  v~.c............
00000000b128f3c5: 00000000 00000000 0d822608 ffff4001  .........&...@..
000000005b473b40: 0d822608 ffff4001 00000000 00000000  .&...@..........
000000000913f445: 00000000 00000000 00000000 00000000  ................
000000003023853f: 00000000 00000000 00000000 00000000  ................
0000000025fcdffb: 00000000 00000000 80000000 00000000  ................
00000000334c5dc4: 00000000 00000000 00000000 00000000  ................
00000000f1ada795: 00000000 00000000 00000000 00000000  ................
0000000030e27dd3: 00000000 00000000 0d822678 ffff4001  ........x&...@..
0000000033a6a483: 0d822678 ffff4001 00000000 00000000  x&...@..........
00000000eb614a98: 00000000 ffffffff 00000000 00000000  ................
00000000df74e25f: 00000000 00000000 00000020 00000000  ........ .......
00000000efcf717a: 00000000 00000000 00000000 00000000  ................
00000000f84ffeba: 00000000 00000000 00000000 00000000  ................
00000000651071c7: 00000000 00000000 0d8226d8 ffff4001  .........&...@..
00000000a748241c: 0d8226d8 ffff4001 ffffffe0 0000000f  .&...@..........
000000004c1557b3: 0d8226f0 ffff4001 0d8226f0 ffff4001  .&...@...&...@..
00000000dccbb716: c08cf6b0 ffffdab5 00000000 00000000  ................
000000009d2fb057: 00000000 00000000 00000000 00000000  ................
00000000f6f284e6: 00000000 00000000 00000000 00000000  ................
000000003bb74f4c: 00414087 00414087 00000000 00000000  .@A..@A.........
00000000e0601ec8: 00000000 00000000 00000000 00000000  ................
000000006a017bfb: cef69528 00000000                    (.......
CPU: 125 PID: 3898 Comm: dbench Not tainted 6.0.5-rt14-[...]
Hardware name: WIWYNN Mt.Jade Server System B81.03001.0005/Mt.Jade Motherboard, BIOS 1.08.20220218 (SCP: 1.08.20220218) 2022/02/18
Call trace:
  [...]
  ext4_destroy_inode+0xc8/0xd0
  destroy_inode+0x48/0x80
  evict+0x148/0x190
  iput+0x184/0x250
  do_unlinkat+0x1d8/0x290
  __arm64_sys_unlinkat+0x48/0x90
  invoke_syscall+0x78/0x100
  el0_svc_common.constprop.0+0x54/0x194
  do_el0_svc+0x38/0xd0
  el0_svc+0x34/0x160
  el0t_64_sync_handler+0xbc/0x13c
  el0t_64_sync+0x1a0/0x1a4