Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Julien Grall <julien@xen.org>]

Hi,

On 06/05/2020 14:48, Corey Minyard wrote:
> On Sat, May 02, 2020 at 12:46:14PM +0100, Julien Grall wrote:
>> Hi,
>>
>> On 02/05/2020 03:16, Corey Minyard wrote:
>>> On Fri, May 01, 2020 at 06:06:11PM -0700, Roman Shaposhnik wrote:
>>>> On Fri, May 1, 2020 at 4:42 AM Corey Minyard <minyard@acm.org> wrote:
>>>>>
>>>>> On Thu, Apr 30, 2020 at 07:20:05PM -0700, Roman Shaposhnik wrote:
>>>>>> Hi!
>>>>>>
>>>>>> I'm trying to run Xen on Raspberry Pi 4 with 5.6.1 stock,
>>>>>> upstream kernel. The kernel itself works perfectly well
>>>>>> on the board. When I try booting it as Dom0 under Xen,
>>>>>> it goes into a stacktrace (attached).
>>>>>
>>>>> Getting Xen working on the Pi4 requires a lot of moving parts, and they
>>>>> all have to be right.
>>>>
>>>> Tell me about it! It is a pretty frustrating journey to align
>>>> everything just right.
>>>> On the other hand -- it seems worth to enable RPi as an ARM development
>>>> platform for Xen given how ubiquitous it is.
>>>>
>>>> Hence me trying to combine pristine upstream kernel (5.6.1) with
>>>> pristine upstream
>>>> Xen to enable 100% upstream developer workflow on RPi.
>>>>
>>>>>> Looking at what nice folks over at Dornerworks have previously
>>>>>> done to make RPi kernels boot as Dom0 I've come across these
>>>>>> 3 patches:
>>>>>>       https://github.com/dornerworks/xen-rpi4-builder/tree/master/patches/linux
>>>>>>
>>>>>> The first patch seems irrelevant (unless I'm missing something
>>>>>> really basic here).
>>>>>
>>>>> It might be irrelevant for your configuration, assuming that Xen gets
>>>>> the right information from EFI.  I haven't tried EFI booting.
>>>>
>>>> I'd doing a bit of belt-and-suspenders strategy really -- I'm actually
>>>> using UEFI u-boot implementation that pre-populates device trees
>>>> and then I'm also forcing an extra copy of it to be load explicitly
>>>> via the GRUB devicetree command (GRUB runs as a UEFI payload).
>>>>
>>>> I also have access to the semi-official TianoCore RPi4 port that seems
>>>> to be working pretty well: https://github.com/pftf/RPi4/releases/tag/v1.5
>>>> for booting all sort of UEFI payloads on RPi4.
>>>>
>>>>>> The 2nd patch applied with no issue (but
>>>>>> I don't think it is related) but the 3d patch failed to apply on
>>>>>> account of 5.6.1 kernel no longer having:
>>>>>>       dev->archdata.dev_dma_ops
>>>>>> E.g.
>>>>>>       https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/mm/dma-mapping.c?h=v5.6.1#n55
>>>>>>
>>>>>> I've tried to emulate the effect of the patch by simply introducing
>>>>>> a static variable that would signal that we already initialized
>>>>>> dev->dma_ops -- but that didn't help at all.
>>>>>>
>>>>>> I'm CCing Jeff Kubascik to see if the original authors of that Corey Minyard
>>>>>> to see if may be they have any suggestions on how this may be dealt
>>>>>> with.
>>>>>>
>>>>>> Any advice would be greatly appreciated!
>>>>>
>>>>> What's your Pi4 config.txt file look like? The GIC is not being handled
>>>>> correctly, and I'm guessing that configuration is wrong.  You can't use
>>>>> the stock config.txt file with Xen, you have to modify the configuration a
>>>>> bit.
>>>>
>>>> Understood. I'm actually using a custom one:
>>>>       https://github.com/lf-edge/eve/blob/master/pkg/u-boot/rpi/config.txt
>>>>
>>>> I could swear that I had a GIC setting in it -- but apparently no -- so
>>>> I added the following at the top of what you could see at the URL above:
>>>>
>>>> total_mem=4096
>>>> enable_gic=1
>>>>
>>>>> I think just adding:
>>>>>
>>>>> enable_gic=1
>>>>> total_mem=1024
>>>>
>>>> Right -- but my board has 4G memory -- so I think what I did above should work.
>>>
>>> Nope.  If you say 4096M of RAM, your issue is almost certainly DMA, but
>>> it's not (just) the Linux code.  On the Raspberry Pi 4, several devices
>>> cannot DMA to above 1024M of RAM, but Xen does not handle this.  The
>>> 1024M of RAM is a limitation you will have to live with until Xen has a
>>> fix.
>>
>> IIUC, dom0 would need to have some memory below 1GB for this to work, am I
>> correct?
> 
> FYI, this also seems to fix the issue with HDMI not working.

Thank you for the testing! I will have a look how I can properly 
upstream this fix (I think we want to keep 4GB limit for other platforms).

Cheers,

-- 
Julien Grall