xen:arm boot failures after commit 1c92a2aa*

xen:arm boot failures after commit 1c92a2aa*

[Suriyan Ramasami]

Hello,
    I just noticed that after commit
1c92a2aaf8c681efea9a3125e63d45b2d0083abf in master -  xen: arm:
support for up to 48-bit IPA addressing on arm64 - odroid xu fails to
boot. I am wondering if that effects Arndale too and other ARM boards.

The output on XEN console is as follows:

(XEN) P2M: 40-bit IPA
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80003518
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 0000000060000000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000080000000-0x000000a0000000 (512MB)
(XEN) Loading zImage from 0000000060000000 to 0000000087c00000-0000000087f10478
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Unable to map translate guest address
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)

Thanks
- Suriyan

Re: xen:arm boot failures after commit 1c92a2aa*

[Ian Campbell]

On Wed, 2014-09-24 at 05:17 -0700, Suriyan Ramasami wrote:
> Hello,
>     I just noticed that after commit
> 1c92a2aaf8c681efea9a3125e63d45b2d0083abf in master -  xen: arm:
> support for up to 48-bit IPA addressing on arm64 - odroid xu fails to
> boot. I am wondering if that effects Arndale too and other ARM boards.

It didn't affect Arndale in my tests yesterday.

Did you bisect to exactly that commit or did you update a bunch of stuff
to end up there?

> The output on XEN console is as follows:
> 
> (XEN) P2M: 40-bit IPA
> (XEN) P2M: 3 levels with order-1 root, VTCR 0x80003518
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Loading kernel from boot module @ 0000000060000000
> (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
> (XEN) BANK[0] 0x00000080000000-0x000000a0000000 (512MB)
> (XEN) Loading zImage from 0000000060000000 to 0000000087c00000-0000000087f10478
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Unable to map translate guest address

Can you turn that panic into a BUG_ON please so we can see the register
state.

> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
> 
> Thanks
> - Suriyan

Re: xen:arm boot failures after commit 1c92a2aa*

[Suriyan Ramasami]

On Wed, Sep 24, 2014 at 5:26 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2014-09-24 at 05:17 -0700, Suriyan Ramasami wrote:
>> Hello,
>>     I just noticed that after commit
>> 1c92a2aaf8c681efea9a3125e63d45b2d0083abf in master -  xen: arm:
>> support for up to 48-bit IPA addressing on arm64 - odroid xu fails to
>> boot. I am wondering if that effects Arndale too and other ARM boards.
>
> It didn't affect Arndale in my tests yesterday.
>
> Did you bisect to exactly that commit or did you update a bunch of stuff
> to end up there?
>

I reverted the commits from HEAD one by one till I could boot. I could
boot with commit "xen:arm support for up to 48 bit physical
addressing".
>From the next commit on = 1c92a2a* and later it gives the panic.

>> The output on XEN console is as follows:
>>
>> (XEN) P2M: 40-bit IPA
>> (XEN) P2M: 3 levels with order-1 root, VTCR 0x80003518
>> (XEN) *** LOADING DOMAIN 0 ***
>> (XEN) Loading kernel from boot module @ 0000000060000000
>> (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
>> (XEN) BANK[0] 0x00000080000000-0x000000a0000000 (512MB)
>> (XEN) Loading zImage from 0000000060000000 to 0000000087c00000-0000000087f10478
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) Unable to map translate guest address
>
> Can you turn that panic into a BUG_ON please so we can see the register
> state.

With BUG_ON this is the output:

(XEN) P2M: 40-bit IPA
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80003518
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading kernel from boot module @ 0000000060000000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000080000000-0x000000a0000000 (512MB)
(XEN) Loading zImage from 0000000060000000 to 0000000087c00000-0000000087f10478
(XEN) Xen BUG at kernel.c:181
(XEN) CPU0: Unexpected Trap: Undefined Instruction
(XEN) ----[ Xen-4.5-unstable  arm32  debug=y  Tainted:    C ]----
(XEN) CPU:    0
(XEN) PC:     002417a4 __bug+0x28/0x40
(XEN) CPSR:   2000005a MODE:Hypervisor
(XEN)      R0: 0026f6d4 R1: 00000000 R2: 00000000 R3: 00000fff
(XEN)      R4: 000000b5 R5: 00264518 R6: 00000000 R7: 00001000
(XEN)      R8: 00000001 R9: 00000000 R10:00000000 R11:002a7ac4 R12:00000001
(XEN) HYP: SP: 002a7abc LR: 002417a4
(XEN)
(XEN)   VTCR_EL2: 80003518
(XEN)  VTTBR_EL2: 00010000be8f6000
(XEN)
(XEN)  SCTLR_EL2: 30cd187f
(XEN)    HCR_EL2: 000000000038643f
(XEN)  TTBR0_EL2: 00000000be8e8000
(XEN)
(XEN)    ESR_EL2: 00000000
(XEN)  HPFAR_EL2: 0000000025806840
(XEN)      HDFAR: 4850d4e0
(XEN)      HIFAR: 9908898c
(XEN)
(XEN) Xen stack trace from sp=002a7abc:
(XEN)    00000001 002a7b1c 0024c6bc 87c00000 00000000 87f10478 00000000 00310478
(XEN)    00000000 60000000 00000000 87c00000 00000000 00000000 0026aa64 4005a000
(XEN)    002f0500 002a7ba0 40046000 00000000 0026aa64 4005a000 40037000 002a7b24
(XEN)    0024ca74 002a7eac 00248d5c a0000000 00000000 00000200 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    4ffbff58 00000000 00000000 40046000 4005a000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 4ff80000 00000000 00000000
(XEN)    00000000 00000001 00000000 80000000 00000000 20000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 87c00000 00000000 002904d0 00000000 88000000
(XEN)    00000000 88200000 00000000 0024c518 00000000 60000000 00000000 00310478
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN)    00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
(XEN) Xen call trace:
(XEN)    [<002417a4>] __bug+0x28/0x40 (PC)
(XEN)    [<002417a4>] __bug+0x28/0x40 (LR)
(XEN)    [<0024c6bc>] kernel_zimage_load+0x1a4/0x1ac
(XEN)    [<0024ca74>] kernel_load+0x10/0x14
(XEN)    [<00248d5c>] construct_dom0+0x684/0xaec
(XEN)    [<0027d310>] start_xen+0xc20/0xe40
(XEN)    [<8020011c>] 8020011c
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) CPU0: Unexpected Trap: Undefined Instruction
(XEN)
(XEN) ****************************************
(XEN)
(XEN) Manual reset required ('noreboot' specified)


>
>> (XEN) ****************************************
>> (XEN)
>> (XEN) Manual reset required ('noreboot' specified)
>>
>> Thanks
>> - Suriyan
>
>

Re: xen:arm boot failures after commit 1c92a2aa*

[Ian Campbell]

On Wed, 2014-09-24 at 05:36 -0700, Suriyan Ramasami wrote:
Thanks,

> (XEN)   VTCR_EL2: 80003518

THis has changed from 0x80003558 before this change (sorry, this was in
your original report too and I failed to spot it).

The missing bit changes the starting level of the PT from L1 to L2,
which ain't gonna work very well ;-).

I can't for the life of me figure out how/why this works on arndale, but
it does...

Anyway, does this fix it for you?

8<------------------

>From ada27d3a56ba89513721e24abe4a4bebee2ab9cf Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Wed, 24 Sep 2014 15:13:28 +0100
Subject: [PATCH] xen: arm: correct VTCR setting on arm32.

1c92a2aaf8c6 "xen: arm: support for up to 48-bit IPA addressing on
arm64" inadvertently changes the VTCR setting for 32-bit from
0x80003558 to 0x80003518, changing the SL0 setting from 0x1 (p2m
starts at L1) to 0x0 (p2m starts at L2).

For some (inexplicable) reason this doesn't cause any issue on
Arndale but it does on the OdroidXU.

Reported-by: Suriyan Ramasami <suriyan.r@gmail.com>
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/arch/arm/p2m.c |    1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 4dccf7b..70929fc 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -1163,6 +1163,7 @@ void __init setup_virt_paging(void)
 #ifdef CONFIG_ARM_32
     printk("P2M: 40-bit IPA\n");
     val |= VTCR_T0SZ(0x18); /* 40 bit IPA */
+    val |= VTCR_SL0(0x1); /* P2M starts at first level */
 #else /* CONFIG_ARM_64 */
     const struct {
         unsigned int pabits; /* Physical Address Size */
-- 
1.7.10.4

Re: xen:arm boot failures after commit 1c92a2aa*

[Julien Grall]

Hi Ian,

On 09/24/2014 03:16 PM, Ian Campbell wrote:
> On Wed, 2014-09-24 at 05:36 -0700, Suriyan Ramasami wrote:
> Thanks,
> 
>> (XEN)   VTCR_EL2: 80003518
> 
> THis has changed from 0x80003558 before this change (sorry, this was in
> your original report too and I failed to spot it).
> 
> The missing bit changes the starting level of the PT from L1 to L2,
> which ain't gonna work very well ;-).
> 
> I can't for the life of me figure out how/why this works on arndale, but
> it does...

It also works on Midway (without the patch below). I guess it depends on
the position of the RAM in the p2m.

> Anyway, does this fix it for you?
> 
> 8<------------------
> 
> From ada27d3a56ba89513721e24abe4a4bebee2ab9cf Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Wed, 24 Sep 2014 15:13:28 +0100
> Subject: [PATCH] xen: arm: correct VTCR setting on arm32.
> 
> 1c92a2aaf8c6 "xen: arm: support for up to 48-bit IPA addressing on
> arm64" inadvertently changes the VTCR setting for 32-bit from
> 0x80003558 to 0x80003518, changing the SL0 setting from 0x1 (p2m
> starts at L1) to 0x0 (p2m starts at L2).
> 
> For some (inexplicable) reason this doesn't cause any issue on
> Arndale but it does on the OdroidXU.
> 
> Reported-by: Suriyan Ramasami <suriyan.r@gmail.com>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

Reviewed-by: Julien Grall <julien.grall@linaro.org>

Regards,

-- 
Julien Grall

Re: xen:arm boot failures after commit 1c92a2aa*

[Suriyan Ramasami]

Hello Ian

On Wed, Sep 24, 2014 at 8:58 AM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Ian,
>
> On 09/24/2014 03:16 PM, Ian Campbell wrote:
>> On Wed, 2014-09-24 at 05:36 -0700, Suriyan Ramasami wrote:
>> Thanks,
>>
>>> (XEN)   VTCR_EL2: 80003518
>>
>> THis has changed from 0x80003558 before this change (sorry, this was in
>> your original report too and I failed to spot it).
>>
>> The missing bit changes the starting level of the PT from L1 to L2,
>> which ain't gonna work very well ;-).
>>
>> I can't for the life of me figure out how/why this works on arndale, but
>> it does...
>
> It also works on Midway (without the patch below). I guess it depends on
> the position of the RAM in the p2m.
>
>> Anyway, does this fix it for you?
>>

This does indeed fix it for the OdroidXU.
(XEN) P2M: 40-bit IPA
(XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558

It now boots up fine!

Thanks
- Suriyan

>> 8<------------------
>>
>> From ada27d3a56ba89513721e24abe4a4bebee2ab9cf Mon Sep 17 00:00:00 2001
>> From: Ian Campbell <ian.campbell@citrix.com>
>> Date: Wed, 24 Sep 2014 15:13:28 +0100
>> Subject: [PATCH] xen: arm: correct VTCR setting on arm32.
>>
>> 1c92a2aaf8c6 "xen: arm: support for up to 48-bit IPA addressing on
>> arm64" inadvertently changes the VTCR setting for 32-bit from
>> 0x80003558 to 0x80003518, changing the SL0 setting from 0x1 (p2m
>> starts at L1) to 0x0 (p2m starts at L2).
>>
>> For some (inexplicable) reason this doesn't cause any issue on
>> Arndale but it does on the OdroidXU.
>>
>> Reported-by: Suriyan Ramasami <suriyan.r@gmail.com>
>> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
>
> Reviewed-by: Julien Grall <julien.grall@linaro.org>
>
> Regards,
>
> --
> Julien Grall

Re: xen:arm boot failures after commit 1c92a2aa*

[Ian Campbell]

On Wed, 2014-09-24 at 09:11 -0700, Suriyan Ramasami wrote:
> Hello Ian
> 
> On Wed, Sep 24, 2014 at 8:58 AM, Julien Grall <julien.grall@linaro.org> wrote:
> > Hi Ian,
> >
> > On 09/24/2014 03:16 PM, Ian Campbell wrote:
> >> On Wed, 2014-09-24 at 05:36 -0700, Suriyan Ramasami wrote:
> >> Thanks,
> >>
> >>> (XEN)   VTCR_EL2: 80003518
> >>
> >> THis has changed from 0x80003558 before this change (sorry, this was in
> >> your original report too and I failed to spot it).
> >>
> >> The missing bit changes the starting level of the PT from L1 to L2,
> >> which ain't gonna work very well ;-).
> >>
> >> I can't for the life of me figure out how/why this works on arndale, but
> >> it does...
> >
> > It also works on Midway (without the patch below). I guess it depends on
> > the position of the RAM in the p2m.
> >
> >> Anyway, does this fix it for you?
> >>
> 
> This does indeed fix it for the OdroidXU.
> (XEN) P2M: 40-bit IPA
> (XEN) P2M: 3 levels with order-1 root, VTCR 0x80003558
> 
> It now boots up fine!

Thanks, I shall add your Tested-by: when I commit.

Ian.

Re: xen:arm boot failures after commit 1c92a2aa*

[Ian Campbell]

On Wed, 2014-09-24 at 16:58 +0100, Julien Grall wrote:
> Hi Ian,
> 
> On 09/24/2014 03:16 PM, Ian Campbell wrote:
> > On Wed, 2014-09-24 at 05:36 -0700, Suriyan Ramasami wrote:
> > Thanks,
> > 
> >> (XEN)   VTCR_EL2: 80003518
> > 
> > THis has changed from 0x80003558 before this change (sorry, this was in
> > your original report too and I failed to spot it).
> > 
> > The missing bit changes the starting level of the PT from L1 to L2,
> > which ain't gonna work very well ;-).
> > 
> > I can't for the life of me figure out how/why this works on arndale, but
> > it does...
> 
> It also works on Midway (without the patch below). I guess it depends on
> the position of the RAM in the p2m.

I was thinking that presenting a three level PT to an MMU which is
expecting two levels was unlikely to work, but the interesting
properties of the leaf-pte's table bit always being set might mean this
would actually work (sort of) if things were suitably aligned as you are
implying.

I am happy with that theory without digging any further.

Ian.