Re: livelock with KASAN_SW_TAGS

From: Qian Cai <cai@lca.pw>
To: Will Deacon <will.deacon@arm.com>
Cc: Andrey Konovalov <andreyknvl@google.com>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>,
	kasan-dev <kasan-dev@googlegroups.com>,
	aryabinin@virtuozzo.com
Subject: Re: livelock with KASAN_SW_TAGS
Date: Thu, 14 Feb 2019 11:50:59 -0500	[thread overview]
Message-ID: <50ef4f07-af09-5498-2bca-26ced76d9736@lca.pw> (raw)
In-Reply-To: <20190214163536.GB1825@fuggles.cambridge.arm.com>

On 2/14/19 11:35 AM, Will Deacon wrote:
> Hi Qian Cai,
> 
> On Wed, Feb 13, 2019 at 10:32:11PM -0500, Qian Cai wrote:
>> Running LTP msgstress03 [1] triggers endless soft lockups below after a few
>> minutes on a ThunderX2 server. It works fine with KASAN_GENERIC and finish the
>> test in roughly 11 minutes.
> 
> I've not been able to reproduce this failure under KVM, however the test
> only takes around 18s to complete on the host and the guest, so it feels
> like something is amiss here. Please could you share more information about
> how you're triggering this problem? For example:
> 
>   - Kernel version and .config

Latest mainline at 1f947a7a01 ("Merge branch 'akpm' (patches from Andrew)") plus
a few KASAN_SW_TAGS patches in order to boot.

https://marc.info/?l=linux-mm&m=155006632110129&w=2 (all 5)
https://marc.info/?l=linux-mm&m=154968731424637&w=2
https://marc.info/?l=linux-mm&m=155010395725051&w=2

https://git.sr.ht/~cai/linux-debug/tree/master/config

>   - Clang version

clang-7.0.1

>   - LTP version and command-line

latest git at db57ddc1497e

/opt/ltp/testcases/bin/msgstress03

>   - Machine configuration (CPUs, memory, etc)

https://git.sr.ht/~cai/linux-debug/tree/master/dmesg

# lscpu
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              256
On-line CPU(s) list: 0-255
Thread(s) per core:  4
Core(s) per socket:  32
Socket(s):           2
NUMA node(s):        2
Vendor ID:           Cavium
Model:               1
Model name:          ThunderX2 99xx
Stepping:            0x1
BogoMIPS:            400.00
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            32768K
NUMA node0 CPU(s):   0-127
NUMA node1 CPU(s):   128-255
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid
asimdrdm

# free -m
              total        used        free      shared  buff/cache   available
Mem:          84871        8347       75942          55         581       69699
Swap:          4095           0        4095

> 
>> # ./scripts/faddr2line vmlinux _raw_spin_unlock_irqrestore+0x44/0xac
>> _raw_spin_unlock_irqrestore+0x44/0xac:
>> arch_local_irq_restore at arch/arm64/include/asm/irqflags.h:88
>> (inlined by) __raw_spin_unlock_irqrestore at include/linux/spinlock_api_smp.h:160
>> (inlined by) _raw_spin_unlock_irqrestore at kernel/locking/spinlock.c:184
>>
>> This code:
>>
>> 	asm volatile(
>> 		"msr	daif, %0
>> 	:
>> 	: "r" (flags)
>> 	: "memory");
> 
> This is just the instruction when we unmask interrupts, so that's when the
> soft watchdog fires. I suspect the real issue is that we're spending lots
> of time in:
> 
>> [  545.772027]  free_debug_processing+0x308/0x410
> 
> with irqs disabled. Does the machine eventually come back to life?
>

Never.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel