Re: KVM guest sometimes failed to boot because of kernel stack overflow if KPTI is enabled on a hisilicon ARM64 platform.

From: Will Deacon <will.deacon@arm.com>
To: Wei Xu <xuwei5@hisilicon.com>
Cc: Mark Rutland <mark.rutland@arm.com>,
	James Morse <james.morse@arm.com>,
	catalin.marinas@arm.com, suzuki.poulose@arm.com,
	dave.martin@arm.com, marc.zyngier@arm.com,
	linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org, Linuxarm <linuxarm@huawei.com>,
	Hanjun Guo <guohanjun@huawei.com>,
	xiexiuqi@huawei.com, huangdaode <huangdaode@hisilicon.com>,
	"Chenxin (Charles)" <charles.chenxin@huawei.com>,
	"Xiongfanggou (James)" <james.xiong@huawei.com>,
	"Liguozhu (Kenneth)" <liguozhu@hisilicon.com>,
	Zhangyi ac <zhangyi.ac@huawei.com>,
	jonathan.cameron@huawei.com,
	Shameerali Kolothum Thodi  <shameerali.kolothum.thodi@huawei.com>,
	John Garry <john.garry@huawei.com>,
	Salil Mehta <salil.mehta@huawei.com>,
	Shiju Jose <shiju.jose@huawei.com>,
	"Zhuangyuzeng (Yisen)" <yisen.zhuang@huawei.com>,
	"Wangzhou (B)" <wangzhou1@hisilicon.com>,
	"kongxinwei (A)" <kong.kongxinwei@hisilicon.com>,
	"Liyuan (Larry, Turing Solution)" <Larry.T@huawei.com>,
	libeijian@hisilicon.com, zhangbin011@hisilicon.com
Subject: Re: KVM guest sometimes failed to boot because of kernel stack overflow if KPTI is enabled on a hisilicon ARM64 platform.
Date: Fri, 22 Jun 2018 16:41:17 +0100	[thread overview]
Message-ID: <20180622154117.GC1802@arm.com> (raw)
In-Reply-To: <5B2D1595.6020000@hisilicon.com>

On Fri, Jun 22, 2018 at 11:28:21PM +0800, Wei Xu wrote:
> On 2018/6/22 22:28, Mark Rutland wrote:
> >On Fri, Jun 22, 2018 at 09:18:27PM +0800, Wei Xu wrote:
> >>     [    0.227507] Mem abort info:
> >>     [    0.230390]   ESR = 0x96000006
> >>     [    0.233517]   Exception class = DABT (current EL), IL = 32 bits
> >>     [    0.239428]   SET = 0, FnV = 0
> >>     [    0.242555]   EA = 0, S1PTW = 0
> >>     [    0.245797] Data abort info:
> >>     [    0.248795]   ISV = 0, ISS = 0x00000006
> >>     [    0.252652]   CM = 0, WnR = 0
> >>     [    0.255769] swapper pgtable: 4k pages, 48-bit VAs, pgdp =
> >>(ptrval)
> >>     [    0.262645] [ffff0000093abce0] pgd=00000000411f8803,
> >>pud=00000000411f9803, pmd=0000000000000000
> >... and here the PMD for the task stack is all zeroes, so evidently
> >that's getting corrupted somehow.
> >
> >It appears that the overflow stack (which IIRC is embedded within the
> >kernel's data segment, as part of the image mapping), is fine.
> >
> >I wonder if there's some existing weirdness in the page tables for the
> >vmalloc area that causes things to go wrong. Can you please:
> >
> >* enable ARM64_PTDUMP_DEBUGFS
> >
> >* boot with kpti=off (with Will's patch to make this work)
> >
> >* as root, cat /sys/kernel/debug/kernel_page_tables
> >
> >... and dump the result here?
> Thanks!
> Can I do this later since Will's new patch works?

Yes, you should probably go to bed now! Please note that my patch still
isn't the right thing for mainline, since it avoids setting PTE_NG for
tables and therefore won't solve the boot-time issue with KASAN enabled.

We also don't understand why clean+invalidate is causing the issue on your
CPU, whereas clean does not. It looks like clean+invalidate somehow results
in page table entries being zeroed.

Have a good weekend,

Will