Hi all,

I'm trying to build and run Linux (Debian) on top of Xen on a Hikey960
ARM board but am having trouble.

I've been following Konrad's guide on the Wiki (at
https://wiki.xenproject.org/wiki/HiKey960), namely:
* I have upgraded the board's base firmware to support UEFI
* I am building the hikey960-upstream-rebase branch of
https://github.com/96boards-hikey/linux.git (with a patch from Linus's
tree to fix a Hynix UFS bug)
* I am building the master branch of Xen unstable
* I am building the master branch of GRUB (with a patch to fix a DTB
padding bug)

However, I see two problems when booting.

Firstly, Xen fails to bring up any other CPUs but the one it is booting on:

(XEN) Bringing up CPU1
(XEN) Failed to bring up CPU1
(XEN) Failed to bring up CPU 1 (error -9)
(XEN) Bringing up CPU2
(XEN) Failed to bring up CPU2
(XEN) Failed to bring up CPU 2 (error -9)
(XEN) Bringing up CPU3
(XEN) Failed to bring up CPU3
(XEN) Failed to bring up CPU 3 (error -9)
(XEN) Bringing up CPU4
(XEN) Failed to bring up CPU4
(XEN) Failed to bring up CPU 4 (error -9)
(XEN) Bringing up CPU5
(XEN) Failed to bring up CPU5
(XEN) Failed to bring up CPU 5 (error -9)
(XEN) Bringing up CPU6
(XEN) Failed to bring up CPU6
(XEN) Failed to bring up CPU 6 (error -9)
(XEN) Bringing up CPU7
(XEN) Failed to bring up CPU7
(XEN) Failed to bring up CPU 7 (error -9)
(XEN) Brought up 1 CPUs

I have traced this error code -9 being returned by call_psci_cpu_on.

Secondly, Linux fails when it tries to initialise AMBA devices:

[    0.941352] Synchronous External Abort: synchronous external abort
(0x96000210) at 0xffff0000093fdfe
0
[    0.950601] Internal error: : 96000210 [#1] PREEMPT SMP
[    0.955866] Modules linked in:
[    0.958990] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
4.14.0-rc7-linaro-hikey960+ #8
[    0.966791] Hardware name: HiKey960 (DT)
[    0.970777] task: ffff80001d900000 task.stack: ffff000008058000
[    0.976778] PC is at amba_device_try_add+0x108/0x260
[    0.981791] LR is at amba_device_try_add+0xf0/0x260
[    0.986735] pc : [<ffff0000084eada8>] lr : [<ffff0000084ead90>]
pstate: 60000045
[    0.994192] sp : ffff00000805bbf0
[    0.997572] x29: ffff00000805bbf0 x28: 0000000000000000
[    1.002953] x27: ffff0000090203a8 x26: 0000000000000000
[    1.008326] x25: ffff80001dbf9810 x24: 0000000000000000
[    1.013702] x23: ffff0000093fd000 x22: 0000000000001000
[    1.019079] x21: ffff80001cc1b6f8 x20: 0000000000000000
[    1.024455] x19: ffff80001cc1b400 x18: 0000000000000010
[    1.029832] x17: 0000000000000001 x16: 00000000deadbeef
[    1.035209] x15: 0000000000000006 x14: ffffffffffffffff
[    1.040585] x13: 0000000000000020 x12: 0101010101010101
[    1.045962] x11: 0000000000000020 x10: 0101010101010101
[    1.051338] x9 : 0000000000000000 x8 : ffff80001cc0cf00
[    1.056717] x7 : 0000000000000000 x6 : 000000000000003f
[    1.062092] x5 : 0000000000000000 x4 : 0000000000000000
[    1.067468] x3 : 0000000000000000 x2 : 0000000000000000
[    1.072845] x1 : ffff80001d900000 x0 : ffff0000093fdfe0
[    1.078223] Process swapper/0 (pid: 1, stack limit = 0xffff000008058000)
[    1.084989] Call trace:
[    1.087504] Exception stack(0xffff00000805bab0 to 0xffff00000805bbf0)
[    1.094008] baa0:
ffff0000093fdfe0 ffff80001d900000
[    1.101902] bac0: 0000000000000000 0000000000000000
0000000000000000 0000000000000000
[    1.109790] bae0: 000000000000003f 0000000000000000
ffff80001cc0cf00 0000000000000000
[    1.117684] bb00: 0101010101010101 0000000000000020
0101010101010101 0000000000000020
[    1.125576] bb20: ffffffffffffffff 0000000000000006
00000000deadbeef 0000000000000001
[    1.133468] bb40: 0000000000000010 ffff80001cc1b400
0000000000000000 ffff80001cc1b6f8
[    1.141356] bb60: 0000000000001000 ffff0000093fd000
0000000000000000 ffff80001dbf9810
[    1.149248] bb80: 0000000000000000 ffff0000090203a8
0000000000000000 ffff00000805bbf0
[    1.157139] bba0: ffff0000084ead90 ffff00000805bbf0
ffff0000084eada8 0000000060000045
[    1.165034] bbc0: ffff00000805bbf0 ffff0000084ead90
ffffffffffffffff 00000000fffffffe
[    1.172921] bbe0: ffff00000805bbf0 ffff0000084eada8
[    1.177865] [<ffff0000084eada8>] amba_device_try_add+0x108/0x260
[    1.183935] [<ffff0000084eafec>] amba_device_add+0x1c/0xd8
[    1.189493] [<ffff00000890fbbc>] of_platform_bus_create+0x26c/0x300
[    1.195814] [<ffff00000890fa74>] of_platform_bus_create+0x124/0x300
[    1.202145] [<ffff00000890fd7c>] of_platform_populate+0x4c/0xb0
[    1.208135] [<ffff000008f6e2ac>] of_platform_default_populate_init+0x64/0x78
[    1.215247] [<ffff000008083978>] do_one_initcall+0x38/0x120
[    1.220882] [<ffff000008f20d18>] kernel_init_freeable+0x184/0x224
[    1.227038] [<ffff000008a6a360>] kernel_init+0x10/0x100
[    1.232323] [<ffff000008084b60>] ret_from_fork+0x10/0x18
[    1.237703] Code: d10082c0 52800002 8b0002e0 52800018 (b9400001)
[    1.243880] ---[ end trace dcbf70aa30c979a8 ]---
[    1.248573] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b

This does not occur when booting the same Linux kernel without Xen.

I have traced this to amba_device_try_add trying to access the pid and
cid through ioremap'd addresses of this node in the device tree (in
arch/arm64/boot/dts/hisilicon/hi3660-coresight.dtsi):

/* A73 cluster internal coresight */
etm@4,ed440000 {
        compatible = "arm,coresight-etm4x","arm,primecell";
        reg = <0 0xed440000 0 0x1000>;
        clocks = <&pclk>;
        clock-names = "apb_pclk";
        cpu = <&cpu4>;
        port {
                etm4_out_port: endpoint {
                remote-endpoint = <&funnel1_in_port0>;
                };
        };
};

ARM is still relatively new to me and I'm stuck what I should be
attempting next. I would simply not compile Linux AMBA support (by not
setting CONFIG_AMBA) but it appears that this is selected as a reverse
dependency by CONFIG_ARM on Linux, so am unsure if this is wise or
even possible.

I've attached the complete Xen + Linux serial logs.

Keen to hear any suggestions, hacky or otherwise.

- Matthew Daley