Building xen-image-minimal for RPi4 Compute Module

Building xen-image-minimal for RPi4 Compute Module

[pbg.dev]

I'm in the process of building xen-image-minimal for my Raspberry Pi 4 Compute Module and am hitting some issues when running the image on physical hardware. Upon inspecting the boot partition of the sd-card, I noticed that the image only contained the dtb for the Raspberry Pi 4B; this makes sense when I look at xen-raspberrypi4-64.inc:

    # Override the meta-raspberrypi default kernel preference
    PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
    KERNEL_DEVICETREE ?= "broadcom/bcm2711-rpi-4-b.dtb"

Compared with the device tree blobs declared in meta-raspberrypi's raspberrypi4-64.conf:

    RPI_KERNEL_DEVICETREE = " \
        broadcom/bcm2711-rpi-4-b.dtb \
        broadcom/bcm2711-rpi-400.dtb \
        broadcom/bcm2711-rpi-cm4.dtb \
    "

Looking into it further, it seems that the device tree listing in the kernel used in the xen-image-minimal build doesn't contain the newer firmware blobs contained in meta-raspberrypi, which causes devices like ethernet to fail to start up when the image finally boots.

I've tried adding the bcm2711-rpi-cm4 firmware blob to the SD card manually, but that causes u-boot to fail with the error "Bad Linux ARM64 Image Magic!" after the Boot Xen step in output. I also tried manually replacing the bootfiles written by the xen sd card image with the latest blobs from meta-raspberrypi's packages, but once the OS boots I find that xen is no longer running, making me suspect I screwed up the configuration somewhere.

Is this something I can trivially fix? Is there a way for me to override the device tree selection used by the minimal xen image in my local.conf? Or do I need to investigate patching one of the meta-virtualization recipes?

(Apologies if I've used any of the wrong lingo/terms here; still new to RasPi/Yocto/Xen :D)

Re: [meta-virtualization] Building xen-image-minimal for RPi4 Compute Module

[Bruce Ashfield]

Sorry for the slow reply,

On Fri, Nov 5, 2021 at 11:45 PM Patrick Godwin <pbg.dev@outlook.com> wrote:
>
> I'm in the process of building xen-image-minimal for my Raspberry Pi 4 Compute Module and am hitting some issues when running the image on physical hardware. Upon inspecting the boot partition of the sd-card, I noticed that the image only contained the dtb for the Raspberry Pi 4B; this makes sense when I look at xen-raspberrypi4-64.inc:
>

Not sure if you've sorted this out yet.

We do have reference images that booted on the RPI, but I can't say that I've
been testing them myself. I'm hoping that my reply will catch the attention of
those that do have the h/w and they can comment in more detail.

>     # Override the meta-raspberrypi default kernel preference
>     PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
>     KERNEL_DEVICETREE ?= "broadcom/bcm2711-rpi-4-b.dtb"
>
> Compared with the device tree blobs declared in meta-raspberrypi's raspberrypi4-64.conf:
>
>     RPI_KERNEL_DEVICETREE = " \
>         broadcom/bcm2711-rpi-4-b.dtb \
>         broadcom/bcm2711-rpi-400.dtb \
>         broadcom/bcm2711-rpi-cm4.dtb \
>     "
>
> Looking into it further, it seems that the device tree listing in the kernel used in the xen-image-minimal build doesn't contain the newer firmware blobs contained in meta-raspberrypi, which causes devices like ethernet to fail to start up when the image finally boots.
>

There's a different level of support on the h/w between linux-yocto and the
'vendor' rpi tree. We use linux-yocto for the core enablement because the
branches are stable/not rebased and have a cadence we can predict (all
mentioned things are breakages we've hit before!).

That being said, you can change the kernel provider to the linux-rpi, and
the rest of the meta-virt settings, etc, are still applicable and should work.

> I've tried adding the bcm2711-rpi-cm4 firmware blob to the SD card manually, but that causes u-boot to fail with the error "Bad Linux ARM64 Image Magic!" after the Boot Xen step in output. I also tried manually replacing the bootfiles written by the xen sd card image with the latest blobs from meta-raspberrypi's packages, but once the OS boots I find that xen is no longer running, making me suspect I screwed up the configuration somewhere.
>

It could be a kernel configuration issue between the two images, if
the Xen packages are on the image, but nothing is running. Can you
interact with the Xen support via the xen cli at all ? Just to get a
better error message ?

> Is this something I can trivially fix? Is there a way for me to override the device tree selection used by the minimal xen image in my local.conf? Or do I need to investigate patching one of the meta-virtualization recipes?

Anything that needs changing, should be overridable via variable or
through bbappends, so hopefully no patching is required.

Bruce

>
> (Apologies if I've used any of the wrong lingo/terms here; still new to RasPi/Yocto/Xen :D)
>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] Building xen-image-minimal for RPi4 Compute Module

[Patrick Godwin]

No worries, now it's my turn to apologize :) Got pulled away from this over the last week and haven't had a chance to fully dive back on in, but I really appreciate you taking the time to reply here! Now that I've had more time with Yocto and the meta-virt layer I think I'm closing the knowledge gaps that are blocking me, I just have a few more edges to sort out. I think that you're right that a patch shouldn't be needed, I think it's just been a lot of user error on my part.

I'll be sure to update this thread once I have the device up and running; I think I'm close. Thanks again!

-----Original Message-----
From: Bruce Ashfield <bruce.ashfield@gmail.com> 
Sent: Thursday, November 11, 2021 6:53 AM
To: Patrick Godwin <pbg.dev@outlook.com>
Cc: meta-virtualization@lists.yoctoproject.org
Subject: Re: [meta-virtualization] Building xen-image-minimal for RPi4 Compute Module

Sorry for the slow reply,

On Fri, Nov 5, 2021 at 11:45 PM Patrick Godwin <pbg.dev@outlook.com> wrote:
>
> I'm in the process of building xen-image-minimal for my Raspberry Pi 4 Compute Module and am hitting some issues when running the image on physical hardware. Upon inspecting the boot partition of the sd-card, I noticed that the image only contained the dtb for the Raspberry Pi 4B; this makes sense when I look at xen-raspberrypi4-64.inc:
>

Not sure if you've sorted this out yet.

We do have reference images that booted on the RPI, but I can't say that I've been testing them myself. I'm hoping that my reply will catch the attention of those that do have the h/w and they can comment in more detail.

>     # Override the meta-raspberrypi default kernel preference
>     PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
>     KERNEL_DEVICETREE ?= "broadcom/bcm2711-rpi-4-b.dtb"
>
> Compared with the device tree blobs declared in meta-raspberrypi's raspberrypi4-64.conf:
>
>     RPI_KERNEL_DEVICETREE = " \
>         broadcom/bcm2711-rpi-4-b.dtb \
>         broadcom/bcm2711-rpi-400.dtb \
>         broadcom/bcm2711-rpi-cm4.dtb \
>     "
>
> Looking into it further, it seems that the device tree listing in the kernel used in the xen-image-minimal build doesn't contain the newer firmware blobs contained in meta-raspberrypi, which causes devices like ethernet to fail to start up when the image finally boots.
>

There's a different level of support on the h/w between linux-yocto and the 'vendor' rpi tree. We use linux-yocto for the core enablement because the branches are stable/not rebased and have a cadence we can predict (all mentioned things are breakages we've hit before!).

That being said, you can change the kernel provider to the linux-rpi, and the rest of the meta-virt settings, etc, are still applicable and should work.

> I've tried adding the bcm2711-rpi-cm4 firmware blob to the SD card manually, but that causes u-boot to fail with the error "Bad Linux ARM64 Image Magic!" after the Boot Xen step in output. I also tried manually replacing the bootfiles written by the xen sd card image with the latest blobs from meta-raspberrypi's packages, but once the OS boots I find that xen is no longer running, making me suspect I screwed up the configuration somewhere.
>

It could be a kernel configuration issue between the two images, if the Xen packages are on the image, but nothing is running. Can you interact with the Xen support via the xen cli at all ? Just to get a better error message ?

> Is this something I can trivially fix? Is there a way for me to override the device tree selection used by the minimal xen image in my local.conf? Or do I need to investigate patching one of the meta-virtualization recipes?

Anything that needs changing, should be overridable via variable or through bbappends, so hopefully no patching is required.

Bruce

>
> (Apologies if I've used any of the wrong lingo/terms here; still new 
> to RasPi/Yocto/Xen :D)
>
>
> 
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization] Building xen-image-minimal for RPi4 Compute Module

[Patrick Godwin]

Just a small update to this thread: I finally got a Xen image booting to dom0 with ethernet on the Raspberry Pi Compute Module 4. Unfortunately, my config is very ugly and still has some issues (such as USB not working...), but I can at least outline what I did to get booted into Xen:
- Added the meta-virt-rpi layer (https://github.com/aananthcn/meta-virt-rpi) and switched to their dom0-image. This probably wasn't necessary, but this image had some nice quality of life changes I was going to need anyways,
- Use a kernel that contains the device tree blob for the CM4; I ended up using Raspberry Pi's 5.15 kernel.
        - I couldn't get my local.conf to override the kernel selected in meta-virt's inc file without commenting it out, but I'm assuming this is still user error on my part.
        - It looks like support for the CM4 is hitting upstream in 5.16 so yocto-dev will probably work fine starting around then
- Use a more recent u-boot with fixes for CM4. I ended up using 2021.10, though I think the necessary fixes were made in April of this year.
        - I still ended up using the boot script from meta-virtualization instead of the one in meta-virt-rpi; this was easier to modify for my needs

Hopefully these notes help anyone else who decides to try this out. My next goals are to get USB working and then get the configuration cleaned up good enough to share. 

-----Original Message-----
From: meta-virtualization@lists.yoctoproject.org <meta-virtualization@lists.yoctoproject.org> On Behalf Of Patrick Godwin
Sent: Sunday, November 14, 2021 11:30 PM
To: Bruce Ashfield <bruce.ashfield@gmail.com>
Cc: meta-virtualization@lists.yoctoproject.org
Subject: Re: [meta-virtualization] Building xen-image-minimal for RPi4 Compute Module

No worries, now it's my turn to apologize :) Got pulled away from this over the last week and haven't had a chance to fully dive back on in, but I really appreciate you taking the time to reply here! Now that I've had more time with Yocto and the meta-virt layer I think I'm closing the knowledge gaps that are blocking me, I just have a few more edges to sort out. I think that you're right that a patch shouldn't be needed, I think it's just been a lot of user error on my part.

I'll be sure to update this thread once I have the device up and running; I think I'm close. Thanks again!

-----Original Message-----
From: Bruce Ashfield <bruce.ashfield@gmail.com>
Sent: Thursday, November 11, 2021 6:53 AM
To: Patrick Godwin <pbg.dev@outlook.com>
Cc: meta-virtualization@lists.yoctoproject.org
Subject: Re: [meta-virtualization] Building xen-image-minimal for RPi4 Compute Module

Sorry for the slow reply,

On Fri, Nov 5, 2021 at 11:45 PM Patrick Godwin <pbg.dev@outlook.com> wrote:
>
> I'm in the process of building xen-image-minimal for my Raspberry Pi 4 Compute Module and am hitting some issues when running the image on physical hardware. Upon inspecting the boot partition of the sd-card, I noticed that the image only contained the dtb for the Raspberry Pi 4B; this makes sense when I look at xen-raspberrypi4-64.inc:
>

Not sure if you've sorted this out yet.

We do have reference images that booted on the RPI, but I can't say that I've been testing them myself. I'm hoping that my reply will catch the attention of those that do have the h/w and they can comment in more detail.

>     # Override the meta-raspberrypi default kernel preference
>     PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
>     KERNEL_DEVICETREE ?= "broadcom/bcm2711-rpi-4-b.dtb"
>
> Compared with the device tree blobs declared in meta-raspberrypi's raspberrypi4-64.conf:
>
>     RPI_KERNEL_DEVICETREE = " \
>         broadcom/bcm2711-rpi-4-b.dtb \
>         broadcom/bcm2711-rpi-400.dtb \
>         broadcom/bcm2711-rpi-cm4.dtb \
>     "
>
> Looking into it further, it seems that the device tree listing in the kernel used in the xen-image-minimal build doesn't contain the newer firmware blobs contained in meta-raspberrypi, which causes devices like ethernet to fail to start up when the image finally boots.
>

There's a different level of support on the h/w between linux-yocto and the 'vendor' rpi tree. We use linux-yocto for the core enablement because the branches are stable/not rebased and have a cadence we can predict (all mentioned things are breakages we've hit before!).

That being said, you can change the kernel provider to the linux-rpi, and the rest of the meta-virt settings, etc, are still applicable and should work.

> I've tried adding the bcm2711-rpi-cm4 firmware blob to the SD card manually, but that causes u-boot to fail with the error "Bad Linux ARM64 Image Magic!" after the Boot Xen step in output. I also tried manually replacing the bootfiles written by the xen sd card image with the latest blobs from meta-raspberrypi's packages, but once the OS boots I find that xen is no longer running, making me suspect I screwed up the configuration somewhere.
>

It could be a kernel configuration issue between the two images, if the Xen packages are on the image, but nothing is running. Can you interact with the Xen support via the xen cli at all ? Just to get a better error message ?

> Is this something I can trivially fix? Is there a way for me to override the device tree selection used by the minimal xen image in my local.conf? Or do I need to investigate patching one of the meta-virtualization recipes?

Anything that needs changing, should be overridable via variable or through bbappends, so hopefully no patching is required.

Bruce

>
> (Apologies if I've used any of the wrong lingo/terms here; still new 
> to RasPi/Yocto/Xen :D)
>
>
> 
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
- "Use the force Harry" - Gandalf, Star Trek II