Re: [PATCH] RISC-V: Place DTB at 3GB boundary instead of 4GB

[Bin Meng <bmeng.cn@gmail.com>]

Hi Atish,

On Wed, Dec 23, 2020 at 9:20 AM Bin Meng <bmeng.cn@gmail.com> wrote:
>
> Hi Atish,
>
> On Wed, Dec 23, 2020 at 3:59 AM Atish Patra <Atish.Patra@wdc.com> wrote:
> >
> > On Tue, 2020-12-22 at 13:35 +0800, Bin Meng wrote:
> > > Hi Atish,
> > >
> > > On Sat, Dec 19, 2020 at 3:46 AM Atish Patra <Atish.Patra@wdc.com>
> > > wrote:
> > > >
> > > > On Fri, 2020-12-18 at 16:42 +0800, Bin Meng wrote:
> > > > > Hi Atish,
> > > > >
> > > > > On Fri, Dec 18, 2020 at 4:00 PM Atish Patra <Atish.Patra@wdc.com>
> > > > > wrote:
> > > > > >
> > > > > > On Fri, 2020-12-18 at 15:33 +0800, Bin Meng wrote:
> > > > > > > Hi Atish,
> > > > > > >
> > > > > > > On Fri, Dec 18, 2020 at 3:27 PM Atish Patra <
> > > > > > > Atish.Patra@wdc.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > On Fri, 2020-12-18 at 15:21 +0800, Bin Meng wrote:
> > > > > > > > > Hi Atish,
> > > > > > > > >
> > > > > > > > > On Fri, Dec 18, 2020 at 5:48 AM Atish Patra
> > > > > > > > > <atish.patra@wdc.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Currently, we place the DTB at 2MB from 4GB or end of
> > > > > > > > > > DRAM
> > > > > > > > > > which
> > > > > > > > > > ever is
> > > > > > > > > > lesser. However, Linux kernel can address only 1GB of
> > > > > > > > > > memory
> > > > > > > > > > for
> > > > > > > > > > RV32.
> > > > > > > > > > Thus, it can not map anything beyond 3GB (assuming 2GB
> > > > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > starting address).
> > > > > > > > > > As a result, it can not process DT and panic if opensbi
> > > > > > > > > > dynamic
> > > > > > > > > > firmware
> > > > > > > > > > is used.
> > > > > > > > > >
> > > > > > > > > > Fix this by placing the DTB at 2MB from 3GB or end of
> > > > > > > > > > DRAM
> > > > > > > > > > whichever is lower.
> > > > > > > > > >
> > > > > > > > > > Signed-off-by: Atish Patra <atish.patra@wdc.com>
> > > > > > > > > > ---
> > > > > > > > > >  hw/riscv/boot.c | 4 ++--
> > > > > > > > > >  1 file changed, 2 insertions(+), 2 deletions(-)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > With this patch, 32-bit sifive_u still does not boot
> > > > > > > > > kernel
> > > > > > > > > with
> > > > > > > > > the
> > > > > > > > > following patch applied on 5.10:
> > > > > > > > >
> > > > > > > > > https://patchwork.kernel.org/project/linux-riscv/patch/20201217074855.1948743-1-atish.patra@wdc.com/
> > > > > > > > >
> > > > > > > > > Command I used:
> > > > > > > > > $ qemu-system-riscv32 -nographic -M sifive_u -m 1G -smp 5
> > > > > > > > > -
> > > > > > > > > kernel
> > > > > > > > > arch/riscv/boot/Image
> > > > > > > > >
> > > > > > > > > 32-bit virt cannot boot the same kernel image with memory
> > > > > > > > > set
> > > > > > > > > to
> > > > > > > > > 2G
> > > > > > > > > either:
> > > > > > > > > $ qemu-system-riscv32 -nographic -M virt -m 2G -smp 4 -
> > > > > > > > > kernel
> > > > > > > > > arch/riscv/boot/Image
> > > > > > > > >
> > > > > > > >
> > > > > > > > Hi Bin,
> > > > > > > > As mentioned in the email on the linux mailing list, this
> > > > > > > > patch
> > > > > > > > only
> > > > > > > > solves 2GB problem. sifive_u problem is solved by
> > > > > > > > Alistair's
> > > > > > > > patch[1].
> > > > > > > >
> > > > > > > > He is planning to send the PR soon. The issue with sifive_u
> > > > > > > > boot
> > > > > > > > was it
> > > > > > > > was failing the 32 bit test earlier resulting a 2MB aligned
> > > > > > > > address
> > > > > > > > instead of 4MB.
> > > > > > >
> > > > > > > Ah, I see. However my testing shows that virt with 2G still
> > > > > > > does
> > > > > > > not
> > > > > > > boot with this patch.
> > > > > > >
> > > > > >
> > > > > > Strange. I verified again with following combination with -bios
> > > > > > and
> > > > > > without bios parameter.
> > > > > >
> > > > > > 1. virt 32/64 with 1GB/2GB memory
> > > > > > 2. sifive_u 32/64 bit with 1GB/2GB memory (Alistair's patch
> > > > > > included)
> > > > > >
> > > > > > Can you share the boot log along with the head commit of Qemu
> > > > > > and
> > > > > > commandline ? I am using 5.10 kernel with my kernel fix.
> > > > > >
> > > > >
> > > > > I was using Alistair's QEMU repo for testing and 5.10 kernel with
> > > > > your
> > > > > kernel fix:
> > > > >
> > > > > $ git checkout -b testing pull-riscv-to-apply-20201217-1
> > > > > $ apply this patch
> > > > > $ mkdir build;cd build;../configure
> > > > > --target-list=riscv64-softmmu,riscv32-softmmu;make -j
> > > > >
> > > > > $ ./qemu-system-riscv32 -nographic -M virt -m 2G -smp 4 -kernel
> > > > > ~/work/git/linux/arch/riscv/boot/Image
> > > > >
> > > > > OpenSBI v0.8
> > > > >    ____                    _____ ____ _____
> > > > >   / __ \                  / ____|  _ \_   _|
> > > > >  | |  | |_ __   ___ _ __ | (___ | |_) || |
> > > > >  | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
> > > > >  | |__| | |_) |  __/ | | |____) | |_) || |_
> > > > >   \____/| .__/ \___|_| |_|_____/|____/_____|
> > > > >         | |
> > > > >         |_|
> > > > >
> > > > > Platform Name       : riscv-virtio,qemu
> > > > > Platform Features   : timer,mfdeleg
> > > > > Platform HART Count : 4
> > > > > Boot HART ID        : 3
> > > > > Boot HART ISA       : rv32imafdcsu
> > > > > BOOT HART Features  : pmp,scounteren,mcounteren,time
> > > > > BOOT HART PMP Count : 16
> > > > > Firmware Base       : 0x80000000
> > > > > Firmware Size       : 104 KB
> > > > > Runtime SBI Version : 0.2
> > > > >
> > > > > MIDELEG : 0x00000222
> > > > > MEDELEG : 0x0000b109
> > > > > PMP0    : 0x80000000-0x8001ffff (A)
> > > > > PMP1    : 0x00000000-0xffffffff (A,R,W,X)
> > > > > <hangs here>
> > > > >
> > > > > $ ./qemu-system-riscv32 -nographic -M sifive_u -m 2G -smp 5 -
> > > > > kernel
> > > > > ~/work/git/linux/arch/riscv/boot/Image
> > > > >
> > > > > OpenSBI v0.8
> > > > >    ____                    _____ ____ _____
> > > > >   / __ \                  / ____|  _ \_   _|
> > > > >  | |  | |_ __   ___ _ __ | (___ | |_) || |
> > > > >  | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
> > > > >  | |__| | |_) |  __/ | | |____) | |_) || |_
> > > > >   \____/| .__/ \___|_| |_|_____/|____/_____|
> > > > >         | |
> > > > >         |_|
> > > > >
> > > > > Platform Name       : SiFive HiFive Unleashed A00
> > > > > Platform Features   : timer,mfdeleg
> > > > > Platform HART Count : 5
> > > > > Boot HART ID        : 4
> > > > > Boot HART ISA       : rv32imafdcsu
> > > > > BOOT HART Features  : pmp,scounteren,mcounteren
> > > > > BOOT HART PMP Count : 16
> > > > > Firmware Base       : 0x80000000
> > > > > Firmware Size       : 112 KB
> > > > > Runtime SBI Version : 0.2
> > > > >
> > > > > MIDELEG : 0x00000222
> > > > > MEDELEG : 0x0000b109
> > > > > PMP0    : 0x80000000-0x8001ffff (A)
> > > > > PMP1    : 0x00000000-0xffffffff (A,R,W,X)
> > > > > <hangs here>
> > > > >
> > > > > The following is sifive_u with 1G:
> > > > >
> > > > > $ ./qemu-system-riscv32 -nographic -M sifive_u -m 1G -smp 5 -
> > > > > kernel
> > > > > ~/work/git/linux/arch/riscv/boot/Image
> > > > >
> > > > > OpenSBI v0.8
> > > > >    ____                    _____ ____ _____
> > > > >   / __ \                  / ____|  _ \_   _|
> > > > >  | |  | |_ __   ___ _ __ | (___ | |_) || |
> > > > >  | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
> > > > >  | |__| | |_) |  __/ | | |____) | |_) || |_
> > > > >   \____/| .__/ \___|_| |_|_____/|____/_____|
> > > > >         | |
> > > > >         |_|
> > > > >
> > > > > Platform Name       : SiFive HiFive Unleashed A00
> > > > > Platform Features   : timer,mfdeleg
> > > > > Platform HART Count : 5
> > > > > Boot HART ID        : 3
> > > > > Boot HART ISA       : rv32imafdcsu
> > > > > BOOT HART Features  : pmp,scounteren,mcounteren
> > > > > BOOT HART PMP Count : 16
> > > > > Firmware Base       : 0x80000000
> > > > > Firmware Size       : 112 KB
> > > > > Runtime SBI Version : 0.2
> > > > >
> > > > > MIDELEG : 0x00000222
> > > > > MEDELEG : 0x0000b109
> > > > > PMP0    : 0x80000000-0x8001ffff (A)
> > > > > PMP1    : 0x00000000-0xffffffff (A,R,W,X)
> > > > > [    0.000000] Linux version 5.10.0-00001-gbf0dad61896d
> > > > > (bmeng@pek-vx-bsp2) (riscv64-linux-gcc (GCC) 8.1.0, GNU ld (GNU
> > > > > Binutils) 2.30) #1 SMP Thu Dec 17 16:48:13 CST 2020
> > > > > [    0.000000] OF: fdt: Ignoring memory range 0x80000000 -
> > > > > 0x80400000
> > > > > [    0.000000] efi: UEFI not found.
> > > > > [    0.000000] Zone ranges:
> > > > > [    0.000000]   Normal   [mem 0x0000000080400000-
> > > > > 0x00000000bfffffff]
> > > > > [    0.000000] Movable zone start for each node
> > > > > [    0.000000] Early memory node ranges
> > > > > [    0.000000]   node   0: [mem 0x0000000080400000-
> > > > > 0x00000000bfffffff]
> > > > > [    0.000000] Initmem setup node 0 [mem 0x0000000080400000-
> > > > > 0x00000000bfffffff]
> > > > > [    0.000000] SBI specification v0.2 detected
> > > > > [    0.000000] SBI implementation ID=0x1 Version=0x8
> > > > > [    0.000000] SBI v0.2 TIME extension detected
> > > > > [    0.000000] SBI v0.2 IPI extension detected
> > > > > [    0.000000] SBI v0.2 RFENCE extension detected
> > > > > [    0.000000] SBI v0.2 HSM extension detected
> > > > > [    0.000000] CPU with hartid=0 is not available
> > > > > [    0.000000] CPU with hartid=0 is not available
> > > > >
> > > > > Regards,
> > > > > Bin
> > > >
> > > > May be you forgot to apply this patch for 2GB case on top of
> > > > Alistair's
> > > > tree? I don't see any issues with exact same setup.
> > >
> > > That's really weird. I have:
> > >
> > > $ git log --oneline
> > > 3ced2fb RISC-V: Place DTB at 3GB boundary instead of 4GB
> > > d31e970 riscv/opentitan: Update the OpenTitan memory layout
> > > 3ed2b8a hw/riscv: Use the CPU to determine if 32-bit
> > > 094b072 target/riscv: cpu: Set XLEN independently from target
> > > 8987cdc4 target/riscv: csr: Remove compile time XLEN checks
> > >
> > > I just rebuilt the QEMU binaries but still have the same result. 2G
> > > does not boot on either 'virt' or 'sifive_u'.
> > >
> > > $ ./qemu-system-riscv32 -version
> > > QEMU emulator version 5.2.50 (v5.2.0-551-g3ced2fb)
> > > Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project
> > > developers
> > >
> > > Maybe it's toolchain related? I am using kernel.org toolchain
> > > riscv64-linux-gcc (GCC) 8.1.0. I see you were using
> > > riscv64-unknown-linux-gnu-gcc (GCC) 10.2.0.
> > >
> >
> > Strange. If it is a toolchain related issue, the cause might be
> > completely different. Can you give it a try to with updated toolchain ?
> > The pre-built toolchain available in bootlin[1] is 10.2.0.
> >
> > [1] https://toolchains.bootlin.com/
>
> I will give it a try.

I used the bootlin pre-built 10.2.0 toolchain to build the Linux
kernel. Still, 2GB does not boot on QEMU.

I looked at your kernel log again, and found:

[    0.000000] Linux version 5.10.0-00022-ge20097fb37e2-dirty

It seems you have 22 commits on top of 5.10.

I only have one commit "RISC-V: Fix usage of
memblock_enforce_memory_limit" on top of 5.10.

Could you please check?

>
> >
> > Can I download pre-built gcc 8.1.0 toolchain from somewhere ?
>
> Yes, you can download it from
> http://cdn.kernel.org/pub/tools/crosstool/files/bin/x86_64/8.1.0/x86_64-gcc-8.1.0-nolibc-riscv64-linux.tar.xz

Regards,
Bin