Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Corey Minyard]

I had been working on Xen on the Pi4 by throwing kernels I compiled onto
existing sd cards, and this was working fine.  I finally got to a full
yocto build of the system, and it didn't boot.

In fact, Xen didn't print anything at all, and nothing happens that
might suggest it's booting without any console output.

I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
else the same, the 4.19 version of that overlay works, and the 5.4
version does not work.  It also didn't work if I completely removed the
overlay.  The base device trees are the same between the two kernels.

Looking at the overlay changes between the versions and Xen source, I
can't trace down anything that would cause an issue.  Has anyone seen
this issue of have any ideas?

-corey

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Roman Shaposhnik]

On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
>
> I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> existing sd cards, and this was working fine.  I finally got to a full
> yocto build of the system, and it didn't boot.
>
> In fact, Xen didn't print anything at all, and nothing happens that
> might suggest it's booting without any console output.
>
> I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> else the same, the 4.19 version of that overlay works, and the 5.4
> version does not work.  It also didn't work if I completely removed the
> overlay.  The base device trees are the same between the two kernels.
>
> Looking at the overlay changes between the versions and Xen source, I
> can't trace down anything that would cause an issue.  Has anyone seen
> this issue of have any ideas?

FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
    https://github.com/raspberrypi/linux/tree/rpi-5.6.y

The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).

Thanks,
Roman.

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Christopher Clark]

On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> >
> > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > existing sd cards, and this was working fine.  I finally got to a full
> > yocto build of the system, and it didn't boot.
> >
> > In fact, Xen didn't print anything at all, and nothing happens that
> > might suggest it's booting without any console output.

I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
works fine, whereas I see no console output from the kernel once Xen
tries to hand off to dom0 with either a 5.4 or 5.6 kernel.

> > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > else the same, the 4.19 version of that overlay works, and the 5.4
> > version does not work.  It also didn't work if I completely removed the
> > overlay.  The base device trees are the same between the two kernels.
> >
> > Looking at the overlay changes between the versions and Xen source, I
> > can't trace down anything that would cause an issue.  Has anyone seen
> > this issue of have any ideas?

Seen it: yes, but no progress on resolving it to report at this point.

> FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
>     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
>
> The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).

Hi Roman - is there a specific commit in that rpi-5.6.y branch that
you guys have working ok?
It looks like the bcm2711_defconfig file wasn't included in the kernel
source tree of that branch at the point the kernel version was bumped
up to 5.6.14, so is there somewhere else to look for a matching kernel
config?

Christopher

>
> Thanks,
> Roman.
>

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Stefano Stabellini]

On Mon, 15 Jun 2020, Christopher Clark wrote:
> On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
> >
> > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> > >
> > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > existing sd cards, and this was working fine.  I finally got to a full
> > > yocto build of the system, and it didn't boot.
> > >
> > > In fact, Xen didn't print anything at all, and nothing happens that
> > > might suggest it's booting without any console output.
> 
> I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> works fine, whereas I see no console output from the kernel once Xen
> tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> 
> > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > version does not work.  It also didn't work if I completely removed the
> > > overlay.  The base device trees are the same between the two kernels.
> > >
> > > Looking at the overlay changes between the versions and Xen source, I
> > > can't trace down anything that would cause an issue.  Has anyone seen
> > > this issue of have any ideas?
> 
> Seen it: yes, but no progress on resolving it to report at this point.
> 
> > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> >
> > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
> 
> Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> you guys have working ok?
> It looks like the bcm2711_defconfig file wasn't included in the kernel
> source tree of that branch at the point the kernel version was bumped
> up to 5.6.14, so is there somewhere else to look for a matching kernel
> config?

I don't know if that is the issue but beware that some device trees
invert serial0 with serial1. Make sure to use /soc/serial@7e215040. You
can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
line.

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Corey Minyard]

On Mon, Jun 15, 2020 at 05:14:21PM -0700, Stefano Stabellini wrote:
> On Mon, 15 Jun 2020, Christopher Clark wrote:
> > On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
> > >
> > > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> > > >
> > > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > > existing sd cards, and this was working fine.  I finally got to a full
> > > > yocto build of the system, and it didn't boot.
> > > >
> > > > In fact, Xen didn't print anything at all, and nothing happens that
> > > > might suggest it's booting without any console output.
> > 
> > I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> > works fine, whereas I see no console output from the kernel once Xen
> > tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> > 
> > > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > > version does not work.  It also didn't work if I completely removed the
> > > > overlay.  The base device trees are the same between the two kernels.
> > > >
> > > > Looking at the overlay changes between the versions and Xen source, I
> > > > can't trace down anything that would cause an issue.  Has anyone seen
> > > > this issue of have any ideas?
> > 
> > Seen it: yes, but no progress on resolving it to report at this point.
> > 
> > > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> > >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> > >
> > > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
> > 
> > Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> > you guys have working ok?
> > It looks like the bcm2711_defconfig file wasn't included in the kernel
> > source tree of that branch at the point the kernel version was bumped
> > up to 5.6.14, so is there somewhere else to look for a matching kernel
> > config?
> 
> I don't know if that is the issue but beware that some device trees
> invert serial0 with serial1. Make sure to use /soc/serial@7e215040. You
> can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
> line.

I already have that set as part of the boot process, but it still
doesn't print anything out once Xen is started.

I tried the 5.6 device tree, and no help there, eithers.  I'm wondering
if everyone is still running with the 4.19 device trees.

The device tree differences between 4.19 and 5.4 are rather large,
unfortunately.

-corey

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Roman Shaposhnik]

On Mon, Jun 15, 2020 at 6:02 PM Corey Minyard <cminyard@mvista.com> wrote:
>
> On Mon, Jun 15, 2020 at 05:14:21PM -0700, Stefano Stabellini wrote:
> > On Mon, 15 Jun 2020, Christopher Clark wrote:
> > > On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
> > > >
> > > > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> > > > >
> > > > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > > > existing sd cards, and this was working fine.  I finally got to a full
> > > > > yocto build of the system, and it didn't boot.
> > > > >
> > > > > In fact, Xen didn't print anything at all, and nothing happens that
> > > > > might suggest it's booting without any console output.
> > >
> > > I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> > > works fine, whereas I see no console output from the kernel once Xen
> > > tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> > >
> > > > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > > > version does not work.  It also didn't work if I completely removed the
> > > > > overlay.  The base device trees are the same between the two kernels.
> > > > >
> > > > > Looking at the overlay changes between the versions and Xen source, I
> > > > > can't trace down anything that would cause an issue.  Has anyone seen
> > > > > this issue of have any ideas?
> > >
> > > Seen it: yes, but no progress on resolving it to report at this point.
> > >
> > > > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> > > >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> > > >
> > > > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
> > >
> > > Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> > > you guys have working ok?
> > > It looks like the bcm2711_defconfig file wasn't included in the kernel
> > > source tree of that branch at the point the kernel version was bumped
> > > up to 5.6.14, so is there somewhere else to look for a matching kernel
> > > config?
> >
> > I don't know if that is the issue but beware that some device trees
> > invert serial0 with serial1. Make sure to use /soc/serial@7e215040. You
> > can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
> > line.
>
> I already have that set as part of the boot process, but it still
> doesn't print anything out once Xen is started.
>
> I tried the 5.6 device tree, and no help there, eithers.  I'm wondering
> if everyone is still running with the 4.19 device trees.

As I pointed out in the EVE link above -- we're very happily running
with 5.6 device trees. They are, of course, taken from RPI kernel
tree.

Thanks,
Roman.

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Roman Shaposhnik]

On Mon, Jun 15, 2020 at 4:35 PM Christopher Clark
<christopher.w.clark@gmail.com> wrote:
>
> On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
> >
> > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> > >
> > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > existing sd cards, and this was working fine.  I finally got to a full
> > > yocto build of the system, and it didn't boot.
> > >
> > > In fact, Xen didn't print anything at all, and nothing happens that
> > > might suggest it's booting without any console output.
>
> I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> works fine, whereas I see no console output from the kernel once Xen
> tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
>
> > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > version does not work.  It also didn't work if I completely removed the
> > > overlay.  The base device trees are the same between the two kernels.
> > >
> > > Looking at the overlay changes between the versions and Xen source, I
> > > can't trace down anything that would cause an issue.  Has anyone seen
> > > this issue of have any ideas?
>
> Seen it: yes, but no progress on resolving it to report at this point.
>
> > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> >
> > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
>
> Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> you guys have working ok?

Pretty much the latest really. The problem is that it seems RPi ppl.
keep forcepushing to that branch. Hence what we had to do on
the EVE side is to basically "snapshot" it with this gigantic patch:
     https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/patches-5.6.x/0000-rpi-kernel-changes.patch

This needs to be applied on top of straight up upstream 5.6.14 (since
that's what the closest common ancestor was on that branch at the time)

In fact, the entire build process is captured here (if you're curious):
      https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/Dockerfile
(you can literally just docker build . build the whole thing)

> It looks like the bcm2711_defconfig file wasn't included in the kernel
> source tree of that branch at the point the kernel version was bumped
> up to 5.6.14, so is there somewhere else to look for a matching kernel
> config?

Yes ;-) I can share the kind of config that has been pretty extensively
tested by us here at Project EVE:
     https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/kernel_config-5.6.x-aarch64

None of it is terribly EVE specific -- so you should have a reasonable
base-line.

Thanks,
Roman.

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Corey Minyard]

On Mon, Jun 15, 2020 at 06:05:57PM -0700, Roman Shaposhnik wrote:
> On Mon, Jun 15, 2020 at 6:02 PM Corey Minyard <cminyard@mvista.com> wrote:
> >
> > On Mon, Jun 15, 2020 at 05:14:21PM -0700, Stefano Stabellini wrote:
> > > On Mon, 15 Jun 2020, Christopher Clark wrote:
> > > > On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
> > > > >
> > > > > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> > > > > >
> > > > > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > > > > existing sd cards, and this was working fine.  I finally got to a full
> > > > > > yocto build of the system, and it didn't boot.
> > > > > >
> > > > > > In fact, Xen didn't print anything at all, and nothing happens that
> > > > > > might suggest it's booting without any console output.
> > > >
> > > > I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> > > > works fine, whereas I see no console output from the kernel once Xen
> > > > tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> > > >
> > > > > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > > > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > > > > version does not work.  It also didn't work if I completely removed the
> > > > > > overlay.  The base device trees are the same between the two kernels.
> > > > > >
> > > > > > Looking at the overlay changes between the versions and Xen source, I
> > > > > > can't trace down anything that would cause an issue.  Has anyone seen
> > > > > > this issue of have any ideas?
> > > >
> > > > Seen it: yes, but no progress on resolving it to report at this point.
> > > >
> > > > > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> > > > >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> > > > >
> > > > > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
> > > >
> > > > Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> > > > you guys have working ok?
> > > > It looks like the bcm2711_defconfig file wasn't included in the kernel
> > > > source tree of that branch at the point the kernel version was bumped
> > > > up to 5.6.14, so is there somewhere else to look for a matching kernel
> > > > config?
> > >
> > > I don't know if that is the issue but beware that some device trees
> > > invert serial0 with serial1. Make sure to use /soc/serial@7e215040. You
> > > can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
> > > line.
> >
> > I already have that set as part of the boot process, but it still
> > doesn't print anything out once Xen is started.
> >
> > I tried the 5.6 device tree, and no help there, eithers.  I'm wondering
> > if everyone is still running with the 4.19 device trees.
> 
> As I pointed out in the EVE link above -- we're very happily running
> with 5.6 device trees. They are, of course, taken from RPI kernel
> tree.

Ok, what version of Xen are you running?  I'm using 4.13 with the Pi
patches, but I have not tried the master branch.

-corey

> 
> Thanks,
> Roman.

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Roman Shaposhnik]

On Mon, Jun 15, 2020 at 6:45 PM Corey Minyard <cminyard@mvista.com> wrote:
>
> On Mon, Jun 15, 2020 at 06:05:57PM -0700, Roman Shaposhnik wrote:
> > On Mon, Jun 15, 2020 at 6:02 PM Corey Minyard <cminyard@mvista.com> wrote:
> > >
> > > On Mon, Jun 15, 2020 at 05:14:21PM -0700, Stefano Stabellini wrote:
> > > > On Mon, 15 Jun 2020, Christopher Clark wrote:
> > > > > On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
> > > > > >
> > > > > > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> > > > > > >
> > > > > > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > > > > > existing sd cards, and this was working fine.  I finally got to a full
> > > > > > > yocto build of the system, and it didn't boot.
> > > > > > >
> > > > > > > In fact, Xen didn't print anything at all, and nothing happens that
> > > > > > > might suggest it's booting without any console output.
> > > > >
> > > > > I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> > > > > works fine, whereas I see no console output from the kernel once Xen
> > > > > tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> > > > >
> > > > > > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > > > > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > > > > > version does not work.  It also didn't work if I completely removed the
> > > > > > > overlay.  The base device trees are the same between the two kernels.
> > > > > > >
> > > > > > > Looking at the overlay changes between the versions and Xen source, I
> > > > > > > can't trace down anything that would cause an issue.  Has anyone seen
> > > > > > > this issue of have any ideas?
> > > > >
> > > > > Seen it: yes, but no progress on resolving it to report at this point.
> > > > >
> > > > > > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> > > > > >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> > > > > >
> > > > > > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
> > > > >
> > > > > Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> > > > > you guys have working ok?
> > > > > It looks like the bcm2711_defconfig file wasn't included in the kernel
> > > > > source tree of that branch at the point the kernel version was bumped
> > > > > up to 5.6.14, so is there somewhere else to look for a matching kernel
> > > > > config?
> > > >
> > > > I don't know if that is the issue but beware that some device trees
> > > > invert serial0 with serial1. Make sure to use /soc/serial@7e215040. You
> > > > can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
> > > > line.
> > >
> > > I already have that set as part of the boot process, but it still
> > > doesn't print anything out once Xen is started.
> > >
> > > I tried the 5.6 device tree, and no help there, eithers.  I'm wondering
> > > if everyone is still running with the 4.19 device trees.
> >
> > As I pointed out in the EVE link above -- we're very happily running
> > with 5.6 device trees. They are, of course, taken from RPI kernel
> > tree.
>
> Ok, what version of Xen are you running?  I'm using 4.13 with the Pi
> patches, but I have not tried the master branch.

We're running 4.14 + additional patches (not sure which ones will make
it into 4.14 proper yet):
    https://github.com/lf-edge/eve/tree/master/pkg/xen/arch/aarch64

FWIW: the first patch is basically the delta between 4.13 and 4.14

Thanks,
Roman.

Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Christopher Clark]

On Mon, Jun 15, 2020 at 6:49 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Mon, Jun 15, 2020 at 6:45 PM Corey Minyard <cminyard@mvista.com> wrote:
> >
> > On Mon, Jun 15, 2020 at 06:05:57PM -0700, Roman Shaposhnik wrote:
> > > On Mon, Jun 15, 2020 at 6:02 PM Corey Minyard <cminyard@mvista.com> wrote:
> > > >
> > > > On Mon, Jun 15, 2020 at 05:14:21PM -0700, Stefano Stabellini wrote:
> > > > > On Mon, 15 Jun 2020, Christopher Clark wrote:
> > > > > > On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@zededa.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@mvista.com> wrote:
> > > > > > > >
> > > > > > > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > > > > > > existing sd cards, and this was working fine.  I finally got to a full
> > > > > > > > yocto build of the system, and it didn't boot.
> > > > > > > >
> > > > > > > > In fact, Xen didn't print anything at all, and nothing happens that
> > > > > > > > might suggest it's booting without any console output.
> > > > > >
> > > > > > I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> > > > > > works fine, whereas I see no console output from the kernel once Xen
> > > > > > tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> > > > > >
> > > > > > > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > > > > > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > > > > > > version does not work.  It also didn't work if I completely removed the
> > > > > > > > overlay.  The base device trees are the same between the two kernels.
> > > > > > > >
> > > > > > > > Looking at the overlay changes between the versions and Xen source, I
> > > > > > > > can't trace down anything that would cause an issue.  Has anyone seen
> > > > > > > > this issue of have any ideas?
> > > > > >
> > > > > > Seen it: yes, but no progress on resolving it to report at this point.
> > > > > >
> > > > > > > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> > > > > > >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> > > > > > >
> > > > > > > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
> > > > > >
> > > > > > Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> > > > > > you guys have working ok?
> > > > > > It looks like the bcm2711_defconfig file wasn't included in the kernel
> > > > > > source tree of that branch at the point the kernel version was bumped
> > > > > > up to 5.6.14, so is there somewhere else to look for a matching kernel
> > > > > > config?
> > > > >
> > > > > I don't know if that is the issue but beware that some device trees
> > > > > invert serial0 with serial1. Make sure to use /soc/serial@7e215040. You
> > > > > can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
> > > > > line.
> > > >
> > > > I already have that set as part of the boot process, but it still
> > > > doesn't print anything out once Xen is started.
> > > >
> > > > I tried the 5.6 device tree, and no help there, eithers.  I'm wondering
> > > > if everyone is still running with the 4.19 device trees.
> > >
> > > As I pointed out in the EVE link above -- we're very happily running
> > > with 5.6 device trees. They are, of course, taken from RPI kernel
> > > tree.
> >
> > Ok, what version of Xen are you running?  I'm using 4.13 with the Pi
> > patches, but I have not tried the master branch.
>
> We're running 4.14 + additional patches (not sure which ones will make
> it into 4.14 proper yet):
>     https://github.com/lf-edge/eve/tree/master/pkg/xen/arch/aarch64
>
> FWIW: the first patch is basically the delta between 4.13 and 4.14

Corey: try adding these to your u-boot script:

# Define the size and address cells
fdt set /chosen '#size-cells' <1>
fdt set /chosen '#address-cells' <1>

It looks like the 4.19 kernel device tree has those nodes set by the
time the u-boot script picks up after the GPU's work on the device
tree, but the more recent 5.4 and 5.6 kernels do not. Adding those in
within the u-boot script, I've been able to successfully boot the
Linux 5.4 and 5.6 kernels from meta-raspberrypi as well as a local
build of the Eve 5.6 kernel.

Stefano has a patch to do something similar in the Eve series here:
https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/patches-5.6.x/0013-adding-stdout-path-overlay.patch

I've just posted an updated series to meta-virt to enable Xen on Rpi4
with Linux 5.4 (the current Yocto kernel version) here:
https://lists.yoctoproject.org/g/meta-virtualization/message/5449

Christopher