Xen 4.14.1 on RPI4: device tree generation failed

Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

Hi all,
I'm trying to boot Xen 4.14.1 on my RPI4 with the 5.10 kernel, built
using https://github.com/tklengyel/xen-rpi4-builder/tree/update.
Everything builds fine and Xen boots but then I get this error:

(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000000480000
(XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
(XEN) BANK[0] 0x00000008000000-0x00000028000000 (512MB)
(XEN) BANK[1] 0x00000030000000-0x00000038000000 (128MB)
(XEN) BANK[2] 0x00000080000000-0x000000c0000000 (1024MB)
(XEN) BANK[3] 0x000000d8000000-0x000000f0000000 (384MB)
(XEN) Grant table range: 0x00000000200000-0x00000000240000
(XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
(XEN) Device tree generation failed (-22).
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Could not set up DOM0 guest OS
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...


Does anyone have an idea what might be going wrong here? I tried
building the dtb without using the dtb overlay but it didn't seem to
do anything.

Thanks,
Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Nataliya Korovkina]

Hi Tamas,

I had another problem with device tree built with this script (rpixen.sh)...

No promises, but it's worth trying on clean tree:
make O=.build-arm64 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j $(nproc) dtbs
(instead of broadcom/${DTBFILE})

Good luck,
Nataliya

On Sun, Jan 31, 2021 at 2:07 PM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> Hi all,
> I'm trying to boot Xen 4.14.1 on my RPI4 with the 5.10 kernel, built
> using https://github.com/tklengyel/xen-rpi4-builder/tree/update.
> Everything builds fine and Xen boots but then I get this error:
>
> (XEN) *** LOADING DOMAIN 0 ***
> (XEN) Loading d0 kernel from boot module @ 0000000000480000
> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
> (XEN) BANK[0] 0x00000008000000-0x00000028000000 (512MB)
> (XEN) BANK[1] 0x00000030000000-0x00000038000000 (128MB)
> (XEN) BANK[2] 0x00000080000000-0x000000c0000000 (1024MB)
> (XEN) BANK[3] 0x000000d8000000-0x000000f0000000 (384MB)
> (XEN) Grant table range: 0x00000000200000-0x00000000240000
> (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> (XEN) Device tree generation failed (-22).
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Could not set up DOM0 guest OS
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
>
>
> Does anyone have an idea what might be going wrong here? I tried
> building the dtb without using the dtb overlay but it didn't seem to
> do anything.
>
> Thanks,
> Tamas
>

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Elliott Mitchell]

On Sun, Jan 31, 2021 at 02:06:17PM -0500, Tamas K Lengyel wrote:
> (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> (XEN) Device tree generation failed (-22).

> Does anyone have an idea what might be going wrong here? I tried
> building the dtb without using the dtb overlay but it didn't seem to
> do anything.

If you go to line 1412 of the file xen/arch/arm/domain_build.c and
replace the "return res;" with "continue;" that will bypass the issue.
The 3 people I'm copying on this message though may wish to ask questions
about the state of your build tree.

Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
point to that last being touched last year.  Their tree is at
cc39f1c9f82f6fe5a437836811d906c709e0661c.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
>
> On Sun, Jan 31, 2021 at 02:06:17PM -0500, Tamas K Lengyel wrote:
> > (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> > (XEN) Device tree generation failed (-22).
>
> > Does anyone have an idea what might be going wrong here? I tried
> > building the dtb without using the dtb overlay but it didn't seem to
> > do anything.
>
> If you go to line 1412 of the file xen/arch/arm/domain_build.c and
> replace the "return res;" with "continue;" that will bypass the issue.
> The 3 people I'm copying on this message though may wish to ask questions
> about the state of your build tree.

I'll try that but it's a pretty hacky work-around ;)

>
> Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> point to that last being touched last year.  Their tree is at
> cc39f1c9f82f6fe5a437836811d906c709e0661c.

I've moved the Linux branch up to 5.10 because there had been a fair
amount of work that went into fixing Xen on RPI4, which got merged
into 5.9 and I would like to be able to build upstream everything
without the custom patches coming with the rpixen script repo.

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Sun, Jan 31, 2021 at 6:50 PM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> >
> > On Sun, Jan 31, 2021 at 02:06:17PM -0500, Tamas K Lengyel wrote:
> > > (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> > > (XEN) Device tree generation failed (-22).
> >
> > > Does anyone have an idea what might be going wrong here? I tried
> > > building the dtb without using the dtb overlay but it didn't seem to
> > > do anything.
> >
> > If you go to line 1412 of the file xen/arch/arm/domain_build.c and
> > replace the "return res;" with "continue;" that will bypass the issue.
> > The 3 people I'm copying on this message though may wish to ask questions
> > about the state of your build tree.
>
> I'll try that but it's a pretty hacky work-around ;)

That change got Xen to continue but then I don't see any outfrom from
dom0 afterwards and the system just hangs:

(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 0000000000480000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000028000000 (384MB)
(XEN) BANK[1] 0x00000030000000-0x00000038000000 (128MB)
(XEN) Grant table range: 0x00000000200000-0x00000000240000
(XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 0000000000480000 to 0000000010000000-0000000010f80000
(XEN) Loading d0 DTB to 0x0000000018000000-0x000000001800bde9
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) This option is intended to aid debugging of Xen by ensuring
(XEN) that all output is synchronously delivered on the serial line.
(XEN) However it can introduce SIGNIFICANT latencies and affect
(XEN) timekeeping. It is NOT recommended for production use!
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) No support for ARM_SMCCC_ARCH_WORKAROUND_1.
(XEN) Please update your firmware.
(XEN) ***************************************************
(XEN) 3... 2... 1...
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 352kB init memory.

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Elliott Mitchell]

On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> >
> > On Sun, Jan 31, 2021 at 02:06:17PM -0500, Tamas K Lengyel wrote:
> > > (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> > > (XEN) Device tree generation failed (-22).
> >
> > > Does anyone have an idea what might be going wrong here? I tried
> > > building the dtb without using the dtb overlay but it didn't seem to
> > > do anything.
> >
> > If you go to line 1412 of the file xen/arch/arm/domain_build.c and
> > replace the "return res;" with "continue;" that will bypass the issue.
> > The 3 people I'm copying on this message though may wish to ask questions
> > about the state of your build tree.
> 
> I'll try that but it's a pretty hacky work-around ;)

Actually no, it simply causes Xen to ignore these entries.  The patch
I've got ready to submit to this list also adjusts the error message to
avoid misinterpretation, but does pretty well exactly this.

My only concern is whether it should ignore the entries only for Domain 0
or should always ignore them.


> > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > point to that last being touched last year.  Their tree is at
> > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> 
> I've moved the Linux branch up to 5.10 because there had been a fair
> amount of work that went into fixing Xen on RPI4, which got merged
> into 5.9 and I would like to be able to build upstream everything
> without the custom patches coming with the rpixen script repo.

Please keep track of where your kernel source is checked out at since
there was a desire to figure out what was going on with the device-trees.


Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
kernel command-line should ensure you get output from the kernel if it
manages to start (yes, Linux does support having multiple consoles at the
same time).


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
>
> On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > >
> > > On Sun, Jan 31, 2021 at 02:06:17PM -0500, Tamas K Lengyel wrote:
> > > > (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> > > > (XEN) Device tree generation failed (-22).
> > >
> > > > Does anyone have an idea what might be going wrong here? I tried
> > > > building the dtb without using the dtb overlay but it didn't seem to
> > > > do anything.
> > >
> > > If you go to line 1412 of the file xen/arch/arm/domain_build.c and
> > > replace the "return res;" with "continue;" that will bypass the issue.
> > > The 3 people I'm copying on this message though may wish to ask questions
> > > about the state of your build tree.
> >
> > I'll try that but it's a pretty hacky work-around ;)
>
> Actually no, it simply causes Xen to ignore these entries.  The patch
> I've got ready to submit to this list also adjusts the error message to
> avoid misinterpretation, but does pretty well exactly this.
>
> My only concern is whether it should ignore the entries only for Domain 0
> or should always ignore them.
>
>
> > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > point to that last being touched last year.  Their tree is at
> > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> >
> > I've moved the Linux branch up to 5.10 because there had been a fair
> > amount of work that went into fixing Xen on RPI4, which got merged
> > into 5.9 and I would like to be able to build upstream everything
> > without the custom patches coming with the rpixen script repo.
>
> Please keep track of where your kernel source is checked out at since
> there was a desire to figure out what was going on with the device-trees.
>
>
> Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> kernel command-line should ensure you get output from the kernel if it
> manages to start (yes, Linux does support having multiple consoles at the
> same time).

No output from dom0 received even with the added console options
(+earlyprintk=xen). The kernel build was from rpi-5.10.y
c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
with 4.19 next.

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Sun, Jan 31, 2021 at 9:43 PM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> >
> > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > >
> > > > On Sun, Jan 31, 2021 at 02:06:17PM -0500, Tamas K Lengyel wrote:
> > > > > (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> > > > > (XEN) Device tree generation failed (-22).
> > > >
> > > > > Does anyone have an idea what might be going wrong here? I tried
> > > > > building the dtb without using the dtb overlay but it didn't seem to
> > > > > do anything.
> > > >
> > > > If you go to line 1412 of the file xen/arch/arm/domain_build.c and
> > > > replace the "return res;" with "continue;" that will bypass the issue.
> > > > The 3 people I'm copying on this message though may wish to ask questions
> > > > about the state of your build tree.
> > >
> > > I'll try that but it's a pretty hacky work-around ;)
> >
> > Actually no, it simply causes Xen to ignore these entries.  The patch
> > I've got ready to submit to this list also adjusts the error message to
> > avoid misinterpretation, but does pretty well exactly this.
> >
> > My only concern is whether it should ignore the entries only for Domain 0
> > or should always ignore them.
> >
> >
> > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > point to that last being touched last year.  Their tree is at
> > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > >
> > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > amount of work that went into fixing Xen on RPI4, which got merged
> > > into 5.9 and I would like to be able to build upstream everything
> > > without the custom patches coming with the rpixen script repo.
> >
> > Please keep track of where your kernel source is checked out at since
> > there was a desire to figure out what was going on with the device-trees.
> >
> >
> > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > kernel command-line should ensure you get output from the kernel if it
> > manages to start (yes, Linux does support having multiple consoles at the
> > same time).
>
> No output from dom0 received even with the added console options
> (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> with 4.19 next.

The dtb overlay is giving me the following error with both 4.19 and 5.10:

arch/arm64/boot/dts/overlays/pi4-64-xen.dtbo: Warning (pci_bridge):
/fragment@1/__overlay__: node name is not "pci" or "pcie"
arch/arm64/boot/dts/overlays/pi4-64-xen.dtbo: Warning (pci_bridge):
/fragment@1/__overlay__: missing ranges for PCI bridge (or not a
bridge)
arch/arm64/boot/dts/overlays/pi4-64-xen.dtbo: Warning (pci_bridge):
/fragment@1/__overlay__: incorrect #address-cells for PCI bridge
arch/arm64/boot/dts/overlays/pi4-64-xen.dtbo: Warning (pci_bridge):
/fragment@1/__overlay__: incorrect #size-cells for PCI bridge
arch/arm64/boot/dts/overlays/pi4-64-xen.dtbo: Warning
(pci_device_bus_num): Failed prerequisite 'pci_bridge'

The overlays are defined in
https://github.com/dornerworks/xen-rpi4-builder/blob/master/patches/linux/0001-Add-Xen-overlay-for-the-Pi-4.patch
as:

+/dts-v1/;
+/plugin/;
+
+/ {
+ compatible = "brcm,bcm2711";
+
+ fragment@0 {
+ target-path = "/chosen";
+ __overlay__ {
+ #address-cells = <0x1>;
+ #size-cells = <0x1>;
+ xen,xen-bootargs = "console=dtuart dtuart=/soc/serial@7e215040
sync_console dom0_mem=512M dom0_mem=512M bootscrub=0";
+
+ dom0 {
+ compatible = "xen,linux-zimage", "xen,multiboot-module";
+ reg = <0x00400000 0x01000000>;
+ };
+ };
+ };
+
+ fragment@1 {
+ target-path = "/scb/pcie@7d500000";
+ __overlay__ {
+ device_type = "pci";
+ };
+ };
+};
+// Xen configuration for Pi 4

Don't really know what those warnings mean or how to fix them but
perhaps they are relevant to why Xen also complains?

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Sun, Jan 31, 2021 at 9:43 PM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> >
> > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > >
> > > > On Sun, Jan 31, 2021 at 02:06:17PM -0500, Tamas K Lengyel wrote:
> > > > > (XEN) Unable to retrieve address 0 for /scb/pcie@7d500000/pci@1,0/usb@1,0
> > > > > (XEN) Device tree generation failed (-22).
> > > >
> > > > > Does anyone have an idea what might be going wrong here? I tried
> > > > > building the dtb without using the dtb overlay but it didn't seem to
> > > > > do anything.
> > > >
> > > > If you go to line 1412 of the file xen/arch/arm/domain_build.c and
> > > > replace the "return res;" with "continue;" that will bypass the issue.
> > > > The 3 people I'm copying on this message though may wish to ask questions
> > > > about the state of your build tree.
> > >
> > > I'll try that but it's a pretty hacky work-around ;)
> >
> > Actually no, it simply causes Xen to ignore these entries.  The patch
> > I've got ready to submit to this list also adjusts the error message to
> > avoid misinterpretation, but does pretty well exactly this.
> >
> > My only concern is whether it should ignore the entries only for Domain 0
> > or should always ignore them.
> >
> >
> > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > point to that last being touched last year.  Their tree is at
> > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > >
> > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > amount of work that went into fixing Xen on RPI4, which got merged
> > > into 5.9 and I would like to be able to build upstream everything
> > > without the custom patches coming with the rpixen script repo.
> >
> > Please keep track of where your kernel source is checked out at since
> > there was a desire to figure out what was going on with the device-trees.
> >
> >
> > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > kernel command-line should ensure you get output from the kernel if it
> > manages to start (yes, Linux does support having multiple consoles at the
> > same time).
>
> No output from dom0 received even with the added console options
> (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> with 4.19 next.

With rpi-4.19.y kernel and dtbs
(cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
previous error is not present. I get the boot log on the serial with
just console=hvc0 from dom0 but the kernel ends up in a panic down the
line:

(XEN) traps.c:1983:d0v0 HSR=0x93860046 pc=0xffffff80085ac97c
gva=0xffffff800b096000 gpa=0x0000003e330000
[    1.242863] Unhandled fault at 0xffffff800b096000
[    1.242871] Mem abort info:
[    1.242879]   ESR = 0x96000000
[    1.242893]   Exception class = DABT (current EL), IL = 32 bits
[    1.242922]   SET = 0, FnV = 0
[    1.242928]   EA = 0, S1PTW = 0
[    1.242934] Data abort info:
[    1.242941]   ISV = 0, ISS = 0x00000000
[    1.242948]   CM = 0, WnR = 0
[    1.242958] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)
[    1.242965] [ffffff800b096000] pgd=0000000033ffe003,
pud=0000000033ffe003, pmd=000000003230a003, pte=006800003e33070f
[    1.242989] Internal error: ttbr address size fault: 96000000 [#1]
PREEMPT SMP
[    1.242995] Modules linked in:
[    1.243005] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
[    1.243014] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.127-v8+ #1
[    1.243019] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
[    1.243026] pstate: 20000005 (nzCv daif -PAN -UAO)
[    1.243044] pc : cfb_imageblit+0x58c/0x820
[    1.243054] lr : bcm2708_fb_imageblit+0x2c/0x40
[    1.243059] sp : ffffff800802b4e0
[    1.243063] x29: ffffff800802b4e0 x28: 00000000ffffffff
[    1.243073] x27: 0000000000000010 x26: ffffffc03212c000
[    1.243081] x25: 0000000000000020 x24: ffffffc0322c7d80
[    1.243088] x23: 0000000000000008 x22: ffffffc03212a118
[    1.243095] x21: 0000000000000000 x20: ffffff800b096000
[    1.243102] x19: 0000000000000000 x18: 00000000fffffffc
[    1.243109] x17: 0000000000000000 x16: ffffff800b096000
[    1.243116] x15: 0000000000000001 x14: 0000000000001e00
[    1.243124] x13: 0000000000000010 x12: 0000000000000000
[    1.243131] x11: 0000000000000020 x10: 0000000000000001
[    1.243138] x9 : 0000000000000008 x8 : ffffff800b096020
[    1.243145] x7 : ffffffc03212c001 x6 : 0000000000000000
[    1.243152] x5 : ffffff80089e2f78 x4 : 0000000000000000
[    1.243159] x3 : ffffff800b096000 x2 : ffffffc03212c000
[    1.243166] x1 : 0000000000000000 x0 : 0000000000000000
[    1.243173] Call trace:
[    1.243182]  cfb_imageblit+0x58c/0x820
[    1.243190]  bcm2708_fb_imageblit+0x2c/0x40
[    1.243197]  soft_cursor+0x16c/0x200
[    1.243204]  bit_cursor+0x30c/0x53c
[    1.243211]  fbcon_cursor+0x13c/0x1a0
[    1.243220]  hide_cursor+0x44/0xb0
[    1.243228]  redraw_screen+0x218/0x28c
[    1.243234]  fbcon_prepare_logo+0x380/0x3ec
[    1.243241]  fbcon_init+0x364/0x550
[    1.243248]  visual_init+0xbc/0x110
[    1.243256]  do_bind_con_driver.isra.0+0x1c4/0x3a0
[    1.243264]  do_take_over_console+0x148/0x204
[    1.243270]  do_fbcon_takeover+0x7c/0xe4
[    1.243277]  fbcon_event_notify+0x6d4/0x850
[    1.243288]  blocking_notifier_call_chain+0x90/0xc0
[    1.243297]  fb_notifier_call_chain+0x34/0x40
[    1.243303]  register_framebuffer+0x21c/0x300
[    1.243311]  bcm2708_fb_probe+0x340/0x770
[    1.243319]  platform_drv_probe+0x5c/0xb0
[    1.243325]  really_probe+0x290/0x3a4
[    1.243331]  driver_probe_device+0x60/0xf4
[    1.243337]  __driver_attach+0x118/0x13c
[    1.243347]  bus_for_each_dev+0x84/0xe0
[    1.243352]  driver_attach+0x34/0x40
[    1.243358]  bus_add_driver+0x1a8/0x21c
[    1.243364]  driver_register+0x7c/0x124
[    1.243371]  __platform_driver_register+0x58/0x64
[    1.243382]  bcm2708_fb_init+0x24/0x2c
[    1.243390]  do_one_initcall+0x54/0x248
[    1.243399]  kernel_init_freeable+0x2e4/0x384
[    1.243408]  kernel_init+0x1c/0x118
[    1.243415]  ret_from_fork+0x10/0x18
[    1.243425] Code: 8b0608a6 b94050c6 0a060026 4a0400c6 (b9000066)
[    1.243437] ---[ end trace b74230fc2252e944 ]---
[    1.243452] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b
[    1.243452]
[    1.243464] SMP: stopping secondary CPUs
[    1.243512] Kernel Offset: disabled
[    1.243519] CPU features: 0x0,61006000
[    1.243523] Memory Limit: none
[    1.590409] ---[ end Kernel panic - not syncing: Attempted to kill
init! exitcode=0x0000000b
[    1.590409]  ]---

This seems to have been caused by a monitor being attached to the HDMI
port, with HDMI unplugged dom0 boots OK.

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Elliott Mitchell]

On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> With rpi-4.19.y kernel and dtbs
> (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> previous error is not present. I get the boot log on the serial with
> just console=hvc0 from dom0 but the kernel ends up in a panic down the
> line:

> This seems to have been caused by a monitor being attached to the HDMI
> port, with HDMI unplugged dom0 boots OK.

The balance of reports seem to suggest 5.10 is the way to go if you want
graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety on
RP4.


On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> >
> > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > point to that last being touched last year.  Their tree is at
> > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > >
> > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > amount of work that went into fixing Xen on RPI4, which got merged
> > > into 5.9 and I would like to be able to build upstream everything
> > > without the custom patches coming with the rpixen script repo.
> >
> > Please keep track of where your kernel source is checked out at since
> > there was a desire to figure out what was going on with the device-trees.
> >
> >
> > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > kernel command-line should ensure you get output from the kernel if it
> > manages to start (yes, Linux does support having multiple consoles at the
> > same time).
> 
> No output from dom0 received even with the added console options
> (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> with 4.19 next.

So, their current HEAD.  This reads like you've got a problematic kernel
configuration.  What procedure are you following to generate the
configuration you use?

Using their upstream as a base and then adding the configuration options
for Xen has worked fairly well for me (`make bcm2711_defconfig`,
`make menuconfig`, `make zImage`).

Notably the options:
CONFIG_PARAVIRT
CONFIG_XEN_DOM0
CONFIG_XEN
CONFIG_XEN_BLKDEV_BACKEND
CONFIG_XEN_NETDEV_BACKEND
CONFIG_HVC_XEN
CONFIG_HVC_XEN_FRONTEND

Should be set to "y".


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
>
> On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> > With rpi-4.19.y kernel and dtbs
> > (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> > previous error is not present. I get the boot log on the serial with
> > just console=hvc0 from dom0 but the kernel ends up in a panic down the
> > line:
>
> > This seems to have been caused by a monitor being attached to the HDMI
> > port, with HDMI unplugged dom0 boots OK.
>
> The balance of reports seem to suggest 5.10 is the way to go if you want
> graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety on
> RP4.
>
>
> On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > >
> > > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > > point to that last being touched last year.  Their tree is at
> > > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > > >
> > > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > > amount of work that went into fixing Xen on RPI4, which got merged
> > > > into 5.9 and I would like to be able to build upstream everything
> > > > without the custom patches coming with the rpixen script repo.
> > >
> > > Please keep track of where your kernel source is checked out at since
> > > there was a desire to figure out what was going on with the device-trees.
> > >
> > >
> > > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > > kernel command-line should ensure you get output from the kernel if it
> > > manages to start (yes, Linux does support having multiple consoles at the
> > > same time).
> >
> > No output from dom0 received even with the added console options
> > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > with 4.19 next.
>
> So, their current HEAD.  This reads like you've got a problematic kernel
> configuration.  What procedure are you following to generate the
> configuration you use?
>
> Using their upstream as a base and then adding the configuration options
> for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> `make menuconfig`, `make zImage`).
>
> Notably the options:
> CONFIG_PARAVIRT
> CONFIG_XEN_DOM0
> CONFIG_XEN
> CONFIG_XEN_BLKDEV_BACKEND
> CONFIG_XEN_NETDEV_BACKEND
> CONFIG_HVC_XEN
> CONFIG_HVC_XEN_FRONTEND
>
> Should be set to "y".

Yes, these configs are all set the same way for all Linux builds by the script:
        make O=.build-arm64 ARCH=arm64
CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config

I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
dom0. So far only 4.19 boots.

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Stefano Stabellini]

On Sun, 31 Jan 2021, Tamas K Lengyel wrote:
> This seems to have been caused by a monitor being attached to the HDMI
> port, with HDMI unplugged dom0 boots OK.

FYI others have reported issues with swiotlb-xen when using graphics:
https://marc.info/?l=xen-devel&m=161201486021603 Disabling swiotlb-xen
makes it work for them.

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Mon, Feb 1, 2021 at 10:23 AM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> >
> > On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> > > With rpi-4.19.y kernel and dtbs
> > > (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> > > previous error is not present. I get the boot log on the serial with
> > > just console=hvc0 from dom0 but the kernel ends up in a panic down the
> > > line:
> >
> > > This seems to have been caused by a monitor being attached to the HDMI
> > > port, with HDMI unplugged dom0 boots OK.
> >
> > The balance of reports seem to suggest 5.10 is the way to go if you want
> > graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety on
> > RP4.
> >
> >
> > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > >
> > > > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > > > point to that last being touched last year.  Their tree is at
> > > > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > > > >
> > > > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > > > amount of work that went into fixing Xen on RPI4, which got merged
> > > > > into 5.9 and I would like to be able to build upstream everything
> > > > > without the custom patches coming with the rpixen script repo.
> > > >
> > > > Please keep track of where your kernel source is checked out at since
> > > > there was a desire to figure out what was going on with the device-trees.
> > > >
> > > >
> > > > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > > > kernel command-line should ensure you get output from the kernel if it
> > > > manages to start (yes, Linux does support having multiple consoles at the
> > > > same time).
> > >
> > > No output from dom0 received even with the added console options
> > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > > with 4.19 next.
> >
> > So, their current HEAD.  This reads like you've got a problematic kernel
> > configuration.  What procedure are you following to generate the
> > configuration you use?
> >
> > Using their upstream as a base and then adding the configuration options
> > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > `make menuconfig`, `make zImage`).
> >
> > Notably the options:
> > CONFIG_PARAVIRT
> > CONFIG_XEN_DOM0
> > CONFIG_XEN
> > CONFIG_XEN_BLKDEV_BACKEND
> > CONFIG_XEN_NETDEV_BACKEND
> > CONFIG_HVC_XEN
> > CONFIG_HVC_XEN_FRONTEND
> >
> > Should be set to "y".
>
> Yes, these configs are all set the same way for all Linux builds by the script:
>         make O=.build-arm64 ARCH=arm64
> CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
>
> I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> dom0. So far only 4.19 boots.

rpi-5.4.y boots but ends up in yet another different kernel panic:

(XEN) d0v1: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
(XEN) d0v2: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
(XEN) d0v3: vGICD: unhandled word write 0x000000ffffffff to ICACTIVER0
[    0.354800] Detected PIPT I-cache on CPU1
[    0.360473] Xen: initializing cpu1
[    0.360508] CPU1: Booted secondary processor 0x0000000001 [0x410fd083]
[    0.361674] Detected PIPT I-cache on CPU2
[    0.367323] Xen: initializing cpu2
[    0.367357] CPU2: Booted secondary processor 0x0000000002 [0x410fd083]
[    0.368460] Detected PIPT I-cache on CPU3
[    0.374110] Xen: initializing cpu3
[    0.374144] CPU3: Booted secondary processor 0x0000000003 [0x410fd083]
[    0.374357] smp: Brought up 1 node, 4 CPUs
[    0.421250] SMP: Total of 4 processors activated.
[    0.426051] CPU features: detected: 32-bit EL0 Support
[    0.431344] CPU features: detected: CRC32 instructions
[    0.459837] ------------[ cut here ]------------
[    0.463848] CPU: CPUs started in inconsistent modes
[    0.463925] Internal error: aarch64 BRK: f2000800 [#1] PREEMPT SMP
[    0.475143] Modules linked in:
[    0.478308] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.83-v8+ #1
[    0.484712] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
[    0.490686] pstate: 60000005 (nZCv daif -PAN -UAO)
[    0.495615] pc : smp_cpus_done+0x74/0x98
[    0.499642] lr : smp_cpus_done+0x74/0x98
[    0.503687] sp : ffffffc01002bdf0
[    0.507102] x29: ffffffc01002bdf0 x28: 0000000000000000
[    0.512545] x27: 0000000000000000 x26: ffffffc010f08958
[    0.517989] x25: ffffffc010f286c8 x24: 0000000000000040
[    0.523442] x23: ffffffc010f08000 x22: ffffffc010f28000
[    0.528876] x21: ffffffc010f08918 x20: ffffffc010f08aa0
[    0.534331] x19: 0000000000000100 x18: 0000000000000000
[    0.539764] x17: 000000004d4bcabc x16: 00000000deadbeef
[    0.545209] x15: 0000000000000030 x14: ffffffffffffffff
[    0.550652] x13: ffffffc09002ba87 x12: ffffffc01002ba8f
[    0.556096] x11: ffffffc01002bdf0 x10: ffffffc01002bdf0
[    0.561540] x9 : 00000000ffffffc8 x8 : 636e69206e692064
[    0.566984] x7 : 6574726174732073 x6 : ffffffc010f09000
[    0.572427] x5 : ffffffc010f09098 x4 : ffffffc01002c000
[    0.577871] x3 : 0000000000000000 x2 : 0000000000000000
[    0.583315] x1 : 0000000000000000 x0 : ffffff8036948000
[    0.588759] Call trace:
[    0.591310]  smp_cpus_done+0x74/0x98
[    0.594999]  smp_init+0xe4/0xfc
[    0.598245]  kernel_init_freeable+0x134/0x27c
[    0.602726]  kernel_init+0x1c/0x118
[    0.606326]  ret_from_fork+0x10/0x18
[    0.610016] Code: 540000c0 90fff480 9125a000 97cb8545 (d4210000)
[    0.616251] ---[ end trace 828ddf3cc765197a ]---
[    0.620987] note: swapper/0[1] exited with preempt_count 1
[    0.626610] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b
[    0.634425] SMP: stopping secondary CPUs
[    0.638511] ---[ end Kernel panic - not syncing: Attempted to kill
init! exitcode=0x0000000b ]---

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Elliott Mitchell]

On Mon, Feb 01, 2021 at 10:23:34AM -0500, Tamas K Lengyel wrote:
> On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > No output from dom0 received even with the added console options
> > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > > with 4.19 next.
> >
> > So, their current HEAD.  This reads like you've got a problematic kernel
> > configuration.  What procedure are you following to generate the
> > configuration you use?
> >
> > Using their upstream as a base and then adding the configuration options
> > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > `make menuconfig`, `make zImage`).
> >
> > Notably the options:
> > CONFIG_PARAVIRT
> > CONFIG_XEN_DOM0
> > CONFIG_XEN
> > CONFIG_XEN_BLKDEV_BACKEND
> > CONFIG_XEN_NETDEV_BACKEND
> > CONFIG_HVC_XEN
> > CONFIG_HVC_XEN_FRONTEND
> >
> > Should be set to "y".
> 
> Yes, these configs are all set the same way for all Linux builds by the script:
>         make O=.build-arm64 ARCH=arm64
> CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
> 
> I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> dom0. So far only 4.19 boots.

So you're using a scripted procedure to generate the configuration.  The
actual kernel configuration is saved in the file ".config" in the build
directory.  Could you confirm whether those are actually being set?

Try running `grep -eCONFIG_PARAVIRT -eCONFIG_XEN_DOM0 -eCONFIG_XEN
-eCONFIG_HVC_XEN -eCONFIG_HVC_XEN_FRONTEND .config`, those 5 must
be "=y".  Various kernel configuration options depend upon others, so
there could be potential you need to set one before those get enabled.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Mon, Feb 1, 2021 at 4:24 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
>
> On Mon, Feb 01, 2021 at 10:23:34AM -0500, Tamas K Lengyel wrote:
> > On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > > No output from dom0 received even with the added console options
> > > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > > > with 4.19 next.
> > >
> > > So, their current HEAD.  This reads like you've got a problematic kernel
> > > configuration.  What procedure are you following to generate the
> > > configuration you use?
> > >
> > > Using their upstream as a base and then adding the configuration options
> > > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > > `make menuconfig`, `make zImage`).
> > >
> > > Notably the options:
> > > CONFIG_PARAVIRT
> > > CONFIG_XEN_DOM0
> > > CONFIG_XEN
> > > CONFIG_XEN_BLKDEV_BACKEND
> > > CONFIG_XEN_NETDEV_BACKEND
> > > CONFIG_HVC_XEN
> > > CONFIG_HVC_XEN_FRONTEND
> > >
> > > Should be set to "y".
> >
> > Yes, these configs are all set the same way for all Linux builds by the script:
> >         make O=.build-arm64 ARCH=arm64
> > CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
> >
> > I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> > dom0. So far only 4.19 boots.
>
> So you're using a scripted procedure to generate the configuration.  The
> actual kernel configuration is saved in the file ".config" in the build
> directory.  Could you confirm whether those are actually being set?
>
> Try running `grep -eCONFIG_PARAVIRT -eCONFIG_XEN_DOM0 -eCONFIG_XEN
> -eCONFIG_HVC_XEN -eCONFIG_HVC_XEN_FRONTEND .config`, those 5 must
> be "=y".  Various kernel configuration options depend upon others, so
> there could be potential you need to set one before those get enabled.

These options are all set, it was one of the first things I did was to
confirm in the actual config file. There is no output from 5.9 or
5.10. With 4.19 and 5.4 there is, but only 4.19 actually manages to
boot to a workable state.

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Stefano Stabellini]

On Mon, 1 Feb 2021, Tamas K Lengyel wrote:
> On Mon, Feb 1, 2021 at 10:23 AM Tamas K Lengyel
> <tamas.k.lengyel@gmail.com> wrote:
> >
> > On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > >
> > > On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> > > > With rpi-4.19.y kernel and dtbs
> > > > (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> > > > previous error is not present. I get the boot log on the serial with
> > > > just console=hvc0 from dom0 but the kernel ends up in a panic down the
> > > > line:
> > >
> > > > This seems to have been caused by a monitor being attached to the HDMI
> > > > port, with HDMI unplugged dom0 boots OK.
> > >
> > > The balance of reports seem to suggest 5.10 is the way to go if you want
> > > graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety on
> > > RP4.
> > >
> > >
> > > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > > On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > >
> > > > > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > > > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > > > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > > > > point to that last being touched last year.  Their tree is at
> > > > > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > > > > >
> > > > > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > > > > amount of work that went into fixing Xen on RPI4, which got merged
> > > > > > into 5.9 and I would like to be able to build upstream everything
> > > > > > without the custom patches coming with the rpixen script repo.
> > > > >
> > > > > Please keep track of where your kernel source is checked out at since
> > > > > there was a desire to figure out what was going on with the device-trees.
> > > > >
> > > > >
> > > > > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > > > > kernel command-line should ensure you get output from the kernel if it
> > > > > manages to start (yes, Linux does support having multiple consoles at the
> > > > > same time).
> > > >
> > > > No output from dom0 received even with the added console options
> > > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > > > with 4.19 next.
> > >
> > > So, their current HEAD.  This reads like you've got a problematic kernel
> > > configuration.  What procedure are you following to generate the
> > > configuration you use?
> > >
> > > Using their upstream as a base and then adding the configuration options
> > > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > > `make menuconfig`, `make zImage`).
> > >
> > > Notably the options:
> > > CONFIG_PARAVIRT
> > > CONFIG_XEN_DOM0
> > > CONFIG_XEN
> > > CONFIG_XEN_BLKDEV_BACKEND
> > > CONFIG_XEN_NETDEV_BACKEND
> > > CONFIG_HVC_XEN
> > > CONFIG_HVC_XEN_FRONTEND
> > >
> > > Should be set to "y".
> >
> > Yes, these configs are all set the same way for all Linux builds by the script:
> >         make O=.build-arm64 ARCH=arm64
> > CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
> >
> > I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> > dom0. So far only 4.19 boots.
> 
> rpi-5.4.y boots but ends up in yet another different kernel panic:

That's an interesting error. However, I can tell you that I can boot
rpi-5.9.y just fine (without a monitor attached) with:

cd linux
KERNEL=kernel7l
make bcm2711_defconfig

As mentioned here:

https://www.raspberrypi.org/documentation/linux/kernel/building.md

and also taking the device tree from arch/arm64/boot/dts/broadcom/.

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Roman Shaposhnik]

On Mon, Feb 1, 2021 at 5:40 PM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Mon, 1 Feb 2021, Tamas K Lengyel wrote:
> > On Mon, Feb 1, 2021 at 10:23 AM Tamas K Lengyel
> > <tamas.k.lengyel@gmail.com> wrote:
> > >
> > > On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > >
> > > > On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> > > > > With rpi-4.19.y kernel and dtbs
> > > > > (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> > > > > previous error is not present. I get the boot log on the serial with
> > > > > just console=hvc0 from dom0 but the kernel ends up in a panic down the
> > > > > line:
> > > >
> > > > > This seems to have been caused by a monitor being attached to the HDMI
> > > > > port, with HDMI unplugged dom0 boots OK.
> > > >
> > > > The balance of reports seem to suggest 5.10 is the way to go if you want
> > > > graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety on
> > > > RP4.
> > > >
> > > >
> > > > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > > > On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > > >
> > > > > > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > > > > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > > > > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > > > > > point to that last being touched last year.  Their tree is at
> > > > > > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > > > > > >
> > > > > > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > > > > > amount of work that went into fixing Xen on RPI4, which got merged
> > > > > > > into 5.9 and I would like to be able to build upstream everything
> > > > > > > without the custom patches coming with the rpixen script repo.
> > > > > >
> > > > > > Please keep track of where your kernel source is checked out at since
> > > > > > there was a desire to figure out what was going on with the device-trees.
> > > > > >
> > > > > >
> > > > > > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > > > > > kernel command-line should ensure you get output from the kernel if it
> > > > > > manages to start (yes, Linux does support having multiple consoles at the
> > > > > > same time).
> > > > >
> > > > > No output from dom0 received even with the added console options
> > > > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > > > > with 4.19 next.
> > > >
> > > > So, their current HEAD.  This reads like you've got a problematic kernel
> > > > configuration.  What procedure are you following to generate the
> > > > configuration you use?
> > > >
> > > > Using their upstream as a base and then adding the configuration options
> > > > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > > > `make menuconfig`, `make zImage`).
> > > >
> > > > Notably the options:
> > > > CONFIG_PARAVIRT
> > > > CONFIG_XEN_DOM0
> > > > CONFIG_XEN
> > > > CONFIG_XEN_BLKDEV_BACKEND
> > > > CONFIG_XEN_NETDEV_BACKEND
> > > > CONFIG_HVC_XEN
> > > > CONFIG_HVC_XEN_FRONTEND
> > > >
> > > > Should be set to "y".
> > >
> > > Yes, these configs are all set the same way for all Linux builds by the script:
> > >         make O=.build-arm64 ARCH=arm64
> > > CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
> > >
> > > I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> > > dom0. So far only 4.19 boots.
> >
> > rpi-5.4.y boots but ends up in yet another different kernel panic:
>
> That's an interesting error. However, I can tell you that I can boot
> rpi-5.9.y just fine (without a monitor attached) with:
>
> cd linux
> KERNEL=kernel7l
> make bcm2711_defconfig
>
> As mentioned here:
>
> https://www.raspberrypi.org/documentation/linux/kernel/building.md
>
> and also taking the device tree from arch/arm64/boot/dts/broadcom/.

FWIW: I see the same results with stock upstream 5.10.7 effectively
following the steps you're doing.

However, as I keep saying -- the combination of firmware and u-boot
(in my case) is a very sensitive combination -- hence we're relying
on a very particular set of bits for there in EVE and will refuse to work
with anything else.

It may be helpful to take that combination outside of EVE's context and
try it out in your experiments Tamas.

Thanks,
Roman.

P.S. I'm running into a DomU issue in certain places when running with 5.10.7
but that's a subject for a different thread.

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Tamas K Lengyel]

On Mon, Feb 1, 2021 at 9:10 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Mon, Feb 1, 2021 at 5:40 PM Stefano Stabellini
> <sstabellini@kernel.org> wrote:
> >
> > On Mon, 1 Feb 2021, Tamas K Lengyel wrote:
> > > On Mon, Feb 1, 2021 at 10:23 AM Tamas K Lengyel
> > > <tamas.k.lengyel@gmail.com> wrote:
> > > >
> > > > On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > >
> > > > > On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> > > > > > With rpi-4.19.y kernel and dtbs
> > > > > > (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> > > > > > previous error is not present. I get the boot log on the serial with
> > > > > > just console=hvc0 from dom0 but the kernel ends up in a panic down the
> > > > > > line:
> > > > >
> > > > > > This seems to have been caused by a monitor being attached to the HDMI
> > > > > > port, with HDMI unplugged dom0 boots OK.
> > > > >
> > > > > The balance of reports seem to suggest 5.10 is the way to go if you want
> > > > > graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety on
> > > > > RP4.
> > > > >
> > > > >
> > > > > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > > > > On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > > > >
> > > > > > > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > > > > > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > > > > > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > > > > > > point to that last being touched last year.  Their tree is at
> > > > > > > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > > > > > > >
> > > > > > > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > > > > > > amount of work that went into fixing Xen on RPI4, which got merged
> > > > > > > > into 5.9 and I would like to be able to build upstream everything
> > > > > > > > without the custom patches coming with the rpixen script repo.
> > > > > > >
> > > > > > > Please keep track of where your kernel source is checked out at since
> > > > > > > there was a desire to figure out what was going on with the device-trees.
> > > > > > >
> > > > > > >
> > > > > > > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > > > > > > kernel command-line should ensure you get output from the kernel if it
> > > > > > > manages to start (yes, Linux does support having multiple consoles at the
> > > > > > > same time).
> > > > > >
> > > > > > No output from dom0 received even with the added console options
> > > > > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > > > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > > > > > with 4.19 next.
> > > > >
> > > > > So, their current HEAD.  This reads like you've got a problematic kernel
> > > > > configuration.  What procedure are you following to generate the
> > > > > configuration you use?
> > > > >
> > > > > Using their upstream as a base and then adding the configuration options
> > > > > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > > > > `make menuconfig`, `make zImage`).
> > > > >
> > > > > Notably the options:
> > > > > CONFIG_PARAVIRT
> > > > > CONFIG_XEN_DOM0
> > > > > CONFIG_XEN
> > > > > CONFIG_XEN_BLKDEV_BACKEND
> > > > > CONFIG_XEN_NETDEV_BACKEND
> > > > > CONFIG_HVC_XEN
> > > > > CONFIG_HVC_XEN_FRONTEND
> > > > >
> > > > > Should be set to "y".
> > > >
> > > > Yes, these configs are all set the same way for all Linux builds by the script:
> > > >         make O=.build-arm64 ARCH=arm64
> > > > CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
> > > >
> > > > I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> > > > dom0. So far only 4.19 boots.
> > >
> > > rpi-5.4.y boots but ends up in yet another different kernel panic:
> >
> > That's an interesting error. However, I can tell you that I can boot
> > rpi-5.9.y just fine (without a monitor attached) with:
> >
> > cd linux
> > KERNEL=kernel7l
> > make bcm2711_defconfig
> >
> > As mentioned here:
> >
> > https://www.raspberrypi.org/documentation/linux/kernel/building.md
> >
> > and also taking the device tree from arch/arm64/boot/dts/broadcom/.
>
> FWIW: I see the same results with stock upstream 5.10.7 effectively
> following the steps you're doing.
>
> However, as I keep saying -- the combination of firmware and u-boot
> (in my case) is a very sensitive combination -- hence we're relying
> on a very particular set of bits for there in EVE and will refuse to work
> with anything else.
>
> It may be helpful to take that combination outside of EVE's context and
> try it out in your experiments Tamas.

Well, I'm giving up on this for now. I ran out of ideas to try and I
don't see any useful suggestions on how to debug this further. Looks
like it's super fragile and works only under specific conditions
that's not well documented - if documented at all.

Tamas

Re: Xen 4.14.1 on RPI4: device tree generation failed

[Roman Shaposhnik]

On Mon, Feb 1, 2021 at 6:53 PM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> On Mon, Feb 1, 2021 at 9:10 PM Roman Shaposhnik <roman@zededa.com> wrote:
> >
> > On Mon, Feb 1, 2021 at 5:40 PM Stefano Stabellini
> > <sstabellini@kernel.org> wrote:
> > >
> > > On Mon, 1 Feb 2021, Tamas K Lengyel wrote:
> > > > On Mon, Feb 1, 2021 at 10:23 AM Tamas K Lengyel
> > > > <tamas.k.lengyel@gmail.com> wrote:
> > > > >
> > > > > On Mon, Feb 1, 2021 at 12:54 AM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > > >
> > > > > > On Sun, Jan 31, 2021 at 10:06:21PM -0500, Tamas K Lengyel wrote:
> > > > > > > With rpi-4.19.y kernel and dtbs
> > > > > > > (cc39f1c9f82f6fe5a437836811d906c709e0661c) Xen boots fine and the
> > > > > > > previous error is not present. I get the boot log on the serial with
> > > > > > > just console=hvc0 from dom0 but the kernel ends up in a panic down the
> > > > > > > line:
> > > > > >
> > > > > > > This seems to have been caused by a monitor being attached to the HDMI
> > > > > > > port, with HDMI unplugged dom0 boots OK.
> > > > > >
> > > > > > The balance of reports seem to suggest 5.10 is the way to go if you want
> > > > > > graphics on a RP4 with Xen.  Even without Xen 4.19 is looking rickety on
> > > > > > RP4.
> > > > > >
> > > > > >
> > > > > > On Sun, Jan 31, 2021 at 09:43:13PM -0500, Tamas K Lengyel wrote:
> > > > > > > On Sun, Jan 31, 2021 at 8:59 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > > > > >
> > > > > > > > On Sun, Jan 31, 2021 at 06:50:36PM -0500, Tamas K Lengyel wrote:
> > > > > > > > > On Sun, Jan 31, 2021 at 6:33 PM Elliott Mitchell <ehem+undef@m5p.com> wrote:
> > > > > > > > > > Presently the rpixen script is grabbing the RPF's 4.19 branch, dates
> > > > > > > > > > point to that last being touched last year.  Their tree is at
> > > > > > > > > > cc39f1c9f82f6fe5a437836811d906c709e0661c.
> > > > > > > > >
> > > > > > > > > I've moved the Linux branch up to 5.10 because there had been a fair
> > > > > > > > > amount of work that went into fixing Xen on RPI4, which got merged
> > > > > > > > > into 5.9 and I would like to be able to build upstream everything
> > > > > > > > > without the custom patches coming with the rpixen script repo.
> > > > > > > >
> > > > > > > > Please keep track of where your kernel source is checked out at since
> > > > > > > > there was a desire to figure out what was going on with the device-trees.
> > > > > > > >
> > > > > > > >
> > > > > > > > Including "console=hvc0 console=AMA0 console=ttyS0 console=tty0" in the
> > > > > > > > kernel command-line should ensure you get output from the kernel if it
> > > > > > > > manages to start (yes, Linux does support having multiple consoles at the
> > > > > > > > same time).
> > > > > > >
> > > > > > > No output from dom0 received even with the added console options
> > > > > > > (+earlyprintk=xen). The kernel build was from rpi-5.10.y
> > > > > > > c9226080e513181ffb3909a905e9c23b8a6e8f62. I'll check if it still boots
> > > > > > > with 4.19 next.
> > > > > >
> > > > > > So, their current HEAD.  This reads like you've got a problematic kernel
> > > > > > configuration.  What procedure are you following to generate the
> > > > > > configuration you use?
> > > > > >
> > > > > > Using their upstream as a base and then adding the configuration options
> > > > > > for Xen has worked fairly well for me (`make bcm2711_defconfig`,
> > > > > > `make menuconfig`, `make zImage`).
> > > > > >
> > > > > > Notably the options:
> > > > > > CONFIG_PARAVIRT
> > > > > > CONFIG_XEN_DOM0
> > > > > > CONFIG_XEN
> > > > > > CONFIG_XEN_BLKDEV_BACKEND
> > > > > > CONFIG_XEN_NETDEV_BACKEND
> > > > > > CONFIG_HVC_XEN
> > > > > > CONFIG_HVC_XEN_FRONTEND
> > > > > >
> > > > > > Should be set to "y".
> > > > >
> > > > > Yes, these configs are all set the same way for all Linux builds by the script:
> > > > >         make O=.build-arm64 ARCH=arm64
> > > > > CROSS_COMPILE=aarch64-none-linux-gnu- bcm2711_defconfig xen.config
> > > > >
> > > > > I tried with both the rpi-5.10.y and rpi-5.9.y, neither boot up as
> > > > > dom0. So far only 4.19 boots.
> > > >
> > > > rpi-5.4.y boots but ends up in yet another different kernel panic:
> > >
> > > That's an interesting error. However, I can tell you that I can boot
> > > rpi-5.9.y just fine (without a monitor attached) with:
> > >
> > > cd linux
> > > KERNEL=kernel7l
> > > make bcm2711_defconfig
> > >
> > > As mentioned here:
> > >
> > > https://www.raspberrypi.org/documentation/linux/kernel/building.md
> > >
> > > and also taking the device tree from arch/arm64/boot/dts/broadcom/.
> >
> > FWIW: I see the same results with stock upstream 5.10.7 effectively
> > following the steps you're doing.
> >
> > However, as I keep saying -- the combination of firmware and u-boot
> > (in my case) is a very sensitive combination -- hence we're relying
> > on a very particular set of bits for there in EVE and will refuse to work
> > with anything else.
> >
> > It may be helpful to take that combination outside of EVE's context and
> > try it out in your experiments Tamas.
>
> Well, I'm giving up on this for now. I ran out of ideas to try and I
> don't see any useful suggestions on how to debug this further. Looks
> like it's super fragile and works only under specific conditions
> that's not well documented - if documented at all.

That's fair -- at the same time I honestly don't see how any other approach
but documenting that a BOM of versions is known to work together can be
really practical.

I mean -- at the end of the day that's why no user (well no sane user ;-)) takes
kernel from kernel.org directly -- most of them come through Linux distro BOM.

Thanks,
Roman.