Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

From: Tamas K Lengyel <tamas.k.lengyel@gmail.com>
To: Roman Shaposhnik <roman@zededa.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>,
	Julien Grall <julien@xen.org>,
	minyard@acm.org, Paul Durrant <paul@xen.org>,
	Andrew Cooper <andrew.cooper3@citrix.com>,
	Julien Grall <jgrall@amazon.com>,
	George Dunlap <george.dunlap@citrix.com>,
	Jeff Kubascik <jeff.kubascik@dornerworks.com>,
	Jan Beulich <jbeulich@suse.com>,
	Xen-devel <xen-devel@lists.xenproject.org>,
	Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>
Subject: Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4
Date: Tue, 19 May 2020 21:06:01 -0600	[thread overview]
Message-ID: <CABfawhkoSz-zSdyuFvu=p7pPE=uG1QN1E0XawjRbLa47Cx3Nww@mail.gmail.com> (raw)
In-Reply-To: <CAMmSBy-372BGtHGDsx6SHTwj7sZV4Qvq1XF+kbenkEcwboZF5w@mail.gmail.com>

On Tue, May 19, 2020 at 8:28 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Tue, May 19, 2020, 7:15 PM Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:
>>
>> On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik <roman@zededa.com> wrote:
>> >
>> > On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
>> > <tamas.k.lengyel@gmail.com> wrote:
>> > >
>> > > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
>> > > >
>> > > >
>> > > >
>> > > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
>> > > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
>> > > > >>
>> > > > >> From: Julien Grall <jgrall@amazon.com>
>> > > > >>
>> > > > >> Hi all,
>> > > > >>
>> > > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
>> > > > >> only use the first GB of memory.
>> > > > >>
>> > > > >> This is because several devices cannot DMA above 1GB but Xen doesn't
>> > > > >> necessarily allocate memory for Dom0 below 1GB.
>> > > > >>
>> > > > >> This small series is trying to address the problem by allowing a
>> > > > >> platform to restrict where Dom0 banks are allocated.
>> > > > >>
>> > > > >> This is also a candidate for Xen 4.14. Without it, a user will not be
>> > > > >> able to use all the RAM on the Raspberry Pi 4.
>> > > > >>
>> > > > >> This series has only be slighlty tested. I would appreciate more test on
>> > > > >> the Rasbperry Pi 4 to confirm this removing the restriction.
>> > > > >
>> > > > > Hi Julien,
>> > > >
>> > > > Hi,
>> > > >
>> > > > > could you post a git branch somewhere? I can try this on my rpi4 that
>> > > > > already runs 4.13.
>> > > >
>> > > > I have pushed a branch based on unstable and the v2 of the series:
>> > > >
>> > > > git://xenbits.xen.org/people/julieng/xen-unstable.git
>> > > >
>> > > > branch arm-dma/v2
>> > > >
>> > >
>> > > I've updated my image I built with
>> > > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
>> > > defined 2048m as total_mem and Xen seems to be booting fine and passes
>> > > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
>> > > it was working. When I increased the mem for dom0 the boot is now
>> > > stuck at:
>> > >
>> > > [    1.427788] of_cfs_init
>> > > [    1.429667] of_cfs_init: OK
>> > > [    1.432561] clk: Not disabling unused clocks
>> > > [    1.437239] Waiting for root device /dev/mmcblk0p2...
>> > > [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
>> > > [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
>> > > [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.546902] random: fast init done
>> > > [    1.564590] mmc1: new high speed SDIO card at address 0001
>> > >
>> > > Could this be because the DTB I compiled from a fresh checkout of
>> > > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
>> > > kernel itself is from a checkout ~5 months ago? I guess that must be
>> > > the cause because even if I decrease the dom0_mem to 512m it still
>> > > gets stuck at the same spot whereas it was booting fine before.
>> >
>> > Stefano and I are testing the fix right now -- for now just set your
>> > Dom0 mem to less than 512m.
>>
>> Actually seems to work after I recompiled the kernel and reinstalled
>> all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:
>>
>> xl info:
>> ...
>> total_memory           : 3956
>> free_memory            : 1842
>>
>> cat /proc/meminfo
>> MemTotal:        1963844 kB
>>
>> I get an emergency shell during boot on the console complaining about
>> xenbr0 not coming up but if I just hit continue it boots fine and the
>> network is up. So AFAICT things are good.
>
>
> What exact version of the kernel are you using and what did you build it from?
>
> FWIW: 5.6.x clearly has an issue with DMA.

As I said above: https://github.com/raspberrypi/linux.git branch
rpi-4.19.y, I applied the Linux patches from the xen-rpi4-builder
repo, just changing the dom0_mem option in patch 1. I reverted the
xen-rpi4-builder a couple revisions as to not build using the DTB
overlay.

Tamas