From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0519CC433E0 for ; Wed, 20 May 2020 02:29:18 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BF3902075F for ; Wed, 20 May 2020 02:29:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=zededa.com header.i=@zededa.com header.b="LQGR0USg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BF3902075F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=zededa.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jbETa-0007oh-Bi; Wed, 20 May 2020 02:28:58 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1jbETY-0007oc-K5 for xen-devel@lists.xenproject.org; Wed, 20 May 2020 02:28:56 +0000 X-Inumbo-ID: a7210b3c-9a41-11ea-b07b-bc764e2007e4 Received: from mail-qk1-x743.google.com (unknown [2607:f8b0:4864:20::743]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a7210b3c-9a41-11ea-b07b-bc764e2007e4; Wed, 20 May 2020 02:28:55 +0000 (UTC) Received: by mail-qk1-x743.google.com with SMTP id i14so2109660qka.10 for ; Tue, 19 May 2020 19:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zededa.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WGOOkDJsScw7s5V+9RdisLqP/Tk+LGbdqMPrItt2fl4=; b=LQGR0USgPmirv0ZmSovlpHX7FcGiV4vlQdS9Yw/5drsGXKk7IZYsgcrxwjqHosQpEC 2mtBjK7z2MVmYZFMOVkdLCXJFe+ST6/pMNhpIcrBAqo4k6fO/HzeCVFMvMioMWUnPLpR p4UXA5y8zDtGm1DbIM+CfTjCbR9zyWAsHBgXf7OKKQZ1Ol1KFGcriNQ2CHFF/OvfmiIl nuPaOtEQvngUIYnGc8rOE8RfhJ/EggSU5u/CV/rO5qRJM4ryAxjrMZ3ybsXte2YI8mSg B83T75SzaLgL7usdy52fQfMQWl809SXBbw00hElvWbDt5ApnvzcPFNWHOG6oNk13j5qa +GnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WGOOkDJsScw7s5V+9RdisLqP/Tk+LGbdqMPrItt2fl4=; b=BjOgUQ66Tp8hjJ7Xt2J8BWR2JPVjbgGYz5UKAvVlblaggtLjaa85DKn7CYabRUz0WO VQAeXhuFr17Yp75JHrqJ7RmZMwFiUiR31/LgvR/A8eF0UQ9OGFoDuaAxliGiLXbGZ50e RZgCq5aCgGO+GPULcniSNaOQahJIh6JPcFDIv98I4IlKh20PqZXgKCr19DEK3MDfrc/Y h1k+//mECr24LWZncfqlikE7mB5nqnA8zW69lZywJPlcBQcCIuMHxrtwVwltEvumawUH Xv4vxJwP6xv9V+fZPBaWsXc4bNsUu1PPh/OBClgiqKzMEYdNHoHmr36UupspiyyPgIm6 TJAA== X-Gm-Message-State: AOAM5308bU4I1/djMcrWYXRrcneH1rr8Dn7QRw0ogS1vKPeqdFlprQug fOEfxSGh6FyBjPBtDrhsYSdYRgQcpoEZhBGncteZvQ== X-Google-Smtp-Source: ABdhPJxCXhDxAh7AefgL873OGWdFwOMWlURpaRs5ogyGVpXBqsVkzyPHmtuW/Y9Ns+X/7PkatZcAsuXb9JjRanZVNy4= X-Received: by 2002:a37:4017:: with SMTP id n23mr2388775qka.291.1589941735312; Tue, 19 May 2020 19:28:55 -0700 (PDT) MIME-Version: 1.0 References: <20200518113008.15422-1-julien@xen.org> <297448b7-7837-cbe5-dee4-da80ca03cd29@xen.org> In-Reply-To: From: Roman Shaposhnik Date: Tue, 19 May 2020 19:28:43 -0700 Message-ID: Subject: Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4 To: Tamas K Lengyel Content-Type: multipart/alternative; boundary="00000000000058d77105a60b28fc" X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Julien Grall , minyard@acm.org, Paul Durrant , Andrew Cooper , Julien Grall , George Dunlap , Jeff Kubascik , Jan Beulich , xen-devel@lists.xenproject.org, Volodymyr Babchuk Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" --00000000000058d77105a60b28fc Content-Type: text/plain; charset="UTF-8" On Tue, May 19, 2020, 7:15 PM Tamas K Lengyel wrote: > On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik wrote: > > > > On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel > > wrote: > > > > > > On Tue, May 19, 2020 at 11:23 AM Julien Grall wrote: > > > > > > > > > > > > > > > > On 19/05/2020 04:08, Tamas K Lengyel wrote: > > > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall > wrote: > > > > >> > > > > >> From: Julien Grall > > > > >> > > > > >> Hi all, > > > > >> > > > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 > can > > > > >> only use the first GB of memory. > > > > >> > > > > >> This is because several devices cannot DMA above 1GB but Xen > doesn't > > > > >> necessarily allocate memory for Dom0 below 1GB. > > > > >> > > > > >> This small series is trying to address the problem by allowing a > > > > >> platform to restrict where Dom0 banks are allocated. > > > > >> > > > > >> This is also a candidate for Xen 4.14. Without it, a user will > not be > > > > >> able to use all the RAM on the Raspberry Pi 4. > > > > >> > > > > >> This series has only be slighlty tested. I would appreciate more > test on > > > > >> the Rasbperry Pi 4 to confirm this removing the restriction. > > > > > > > > > > Hi Julien, > > > > > > > > Hi, > > > > > > > > > could you post a git branch somewhere? I can try this on my rpi4 > that > > > > > already runs 4.13. > > > > > > > > I have pushed a branch based on unstable and the v2 of the series: > > > > > > > > git://xenbits.xen.org/people/julieng/xen-unstable.git > > > > > > > > branch arm-dma/v2 > > > > > > > > > > I've updated my image I built with > > > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've > > > defined 2048m as total_mem and Xen seems to be booting fine and passes > > > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem > > > it was working. When I increased the mem for dom0 the boot is now > > > stuck at: > > > > > > [ 1.427788] of_cfs_init > > > [ 1.429667] of_cfs_init: OK > > > [ 1.432561] clk: Not disabling unused clocks > > > [ 1.437239] Waiting for root device /dev/mmcblk0p2... > > > [ 1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes) > > > [ 1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes) > > > [ 1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes) > > > [ 1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes) > > > [ 1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes) > > > [ 1.546902] random: fast init done > > > [ 1.564590] mmc1: new high speed SDIO card at address 0001 > > > > > > Could this be because the DTB I compiled from a fresh checkout of > > > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the > > > kernel itself is from a checkout ~5 months ago? I guess that must be > > > the cause because even if I decrease the dom0_mem to 512m it still > > > gets stuck at the same spot whereas it was booting fine before. > > > > Stefano and I are testing the fix right now -- for now just set your > > Dom0 mem to less than 512m. > > Actually seems to work after I recompiled the kernel and reinstalled > all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g: > > xl info: > ... > total_memory : 3956 > free_memory : 1842 > > cat /proc/meminfo > MemTotal: 1963844 kB > > I get an emergency shell during boot on the console complaining about > xenbr0 not coming up but if I just hit continue it boots fine and the > network is up. So AFAICT things are good. > What exact version of the kernel are you using and what did you build it from? FWIW: 5.6.x clearly has an issue with DMA. Thanks, Roman. > --00000000000058d77105a60b28fc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, May 19, 2020, 7:15 PM Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:
On Tue, May 19, 2020 at 5:50 PM Roma= n Shaposhnik <roman@zededa.com> wrote:
>
> On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
> <tamas.k.lengyel@gmail.com> wrote:
> >
> > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org&= gt; wrote:
> > >
> > >
> > >
> > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen= .org> wrote:
> > > >>
> > > >> From: Julien Grall <jgrall@amazon.com>
> > > >>
> > > >> Hi all,
> > > >>
> > > >> At the moment, a user who wants to boot Xen on the = Raspberry Pi 4 can
> > > >> only use the first GB of memory.
> > > >>
> > > >> This is because several devices cannot DMA above 1G= B but Xen doesn't
> > > >> necessarily allocate memory for Dom0 below 1GB.
> > > >>
> > > >> This small series is trying to address the problem = by allowing a
> > > >> platform to restrict where Dom0 banks are allocated= .
> > > >>
> > > >> This is also a candidate for Xen 4.14. Without it, = a user will not be
> > > >> able to use all the RAM on the Raspberry Pi 4.
> > > >>
> > > >> This series has only be slighlty tested. I would ap= preciate more test on
> > > >> the Rasbperry Pi 4 to confirm this removing the res= triction.
> > > >
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > > could you post a git branch somewhere? I can try this o= n my rpi4 that
> > > > already runs 4.13.
> > >
> > > I have pushed a branch based on unstable and the v2 of the s= eries:
> > >
> > > git://xenbits.xen.or= g/people/julieng/xen-unstable.git
> > >
> > > branch arm-dma/v2
> > >
> >
> > I've updated my image I built with
> > https://github.com/tklengyel/xen-r= pi4-builder a while ago and I've
> > defined 2048m as total_mem and Xen seems to be booting fine and p= asses
> > execution to dom0. With 512m being set as the Xen cmdline for dom= 0_mem
> > it was working. When I increased the mem for dom0 the boot is now=
> > stuck at:
> >
> > [=C2=A0 =C2=A0 1.427788] of_cfs_init
> > [=C2=A0 =C2=A0 1.429667] of_cfs_init: OK
> > [=C2=A0 =C2=A0 1.432561] clk: Not disabling unused clocks
> > [=C2=A0 =C2=A0 1.437239] Waiting for root device /dev/mmcblk0p2..= .
> > [=C2=A0 =C2=A0 1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 = bytes)
> > [=C2=A0 =C2=A0 1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 = bytes)
> > [=C2=A0 =C2=A0 1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 = bytes)
> > [=C2=A0 =C2=A0 1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 = bytes)
> > [=C2=A0 =C2=A0 1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 = bytes)
> > [=C2=A0 =C2=A0 1.546902] random: fast init done
> > [=C2=A0 =C2=A0 1.564590] mmc1: new high speed SDIO card at addres= s 0001
> >
> > Could this be because the DTB I compiled from a fresh checkout of=
> > https://github.com/raspberrypi/linux.gi= t branch rpi-4.19.y whereas the
> > kernel itself is from a checkout ~5 months ago? I guess that must= be
> > the cause because even if I decrease the dom0_mem to 512m it stil= l
> > gets stuck at the same spot whereas it was booting fine before. >
> Stefano and I are testing the fix right now -- for now just set your > Dom0 mem to less than 512m.

Actually seems to work after I recompiled the kernel and reinstalled
all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:

xl info:
...
total_memory=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 3956
free_memory=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 1842

cat /proc/meminfo
MemTotal:=C2=A0 =C2=A0 =C2=A0 =C2=A0 1963844 kB

I get an emergency shell during boot on the console complaining about
xenbr0 not coming up but if I just hit continue it boots fine and the
network is up. So AFAICT things are good.

What exact version of the kernel a= re you using and what did you build it from?

FWIW: 5.6.x clearly has an issue with DMA.

Thanks,
Roma= n.
--00000000000058d77105a60b28fc--