Re: Usage of device-tree for blobs

[Mark Kettenis <mark.kettenis@xs4all.nl>]

> From: Simon Glass <sjg@chromium.org>
> Date: Thu, 26 Aug 2021 21:57:19 -0600
> Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>,
>         Ilias Apalodimas <ilias.apalodimas@linaro.org>,
>         U-Boot Mailing List <u-boot@lists.denx.de>
> Content-Type: text/plain; charset="UTF-8"
> 
> Hi Mark,
> 
> On Thu, 26 Aug 2021 at 14:18, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> >
> > > From: Simon Glass <sjg@chromium.org>
> > > Date: Thu, 26 Aug 2021 13:54:49 -0600
> > >
> > > Hi Heinrich,
> > >
> > >
> > > On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > >
> > > > On 8/26/21 5:15 AM, Simon Glass wrote:
> > > > > Hi Heinrich,
> > > > >
> > > > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > > >>
> > > > >> Hello Simon,
> > > > >>
> > > > >> some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > > >> build time. A device-tree is only supplied at runtime by the prior boot
> > > > >> stage (CONFIG_OF_PRIOR_STAGE=y).
> > > > >>
> > > > >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > > >> the device-tree.
> > > > >>
> > > > >> Could you, please, update the documentation to explain how adding blobs
> > > > >> to the device-tree works in the different scenarios depending on the
> > > > >> values of:
> > > > >>
> > > > >> CONFIG_OF_EMBED
> > > > >> CONFIG_OF_SEPARATE
> > > > >> CONFIG_OF_BOARD
> > > > >> CONFIG_OF_HOSTFILE
> > > > >> CONFIG_OF_PRIOR_STAGE
> > > > >>
> > > > >> It would be especially important to understand how one can develop a
> > > > >> board independent feature which works for all of these settings.
> > > > >
> > > > > OK I will take a crack at this tomorrow.
> > > > >
> > > > > But I think there is a disconnect here, since the only options that
> > > > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > > > u-boot.dtb file. There is nothing tricky here.
> > > >
> > > > The following boards use OF_PRIO_STAGE:
> > > >
> > > > * QEMU
> > > > * bcm7260_defconfig
> > > > * bcm7445_defconfig
> > > > * ae350_rv32_defconfig
> > > > * ae350_rv32_spl_defconfig
> > > > * ae350_rv64_defconfig
> > > > * ae350_rv64_spl_defconfig
> > >
> > > Most of these seem OK as they have an in-tree DT. But the arm and
> > > riscv qemus and the bcm builds do not:
> > >
> > > bcm7260_defconfig
> > > bcm7445_defconfig
> > > configs/qemu_arm64_defconfig
> > > configs/qemu_arm_defconfig
> > > configs/qemu-ppce500_defconfig
> > > configs/qemu-riscv32_defconfig
> > > configs/qemu-riscv32_smode_defconfig
> > > configs/qemu-riscv64_defconfig
> > > configs/qemu-riscv64_smode_defconfig
> > >
> > > I think we are going to have to ban this. We are not really testing
> > > the build at all, and it presumably depends on the version of qemu
> > > that is used. It's OK to provide the DT to U-Boot as one flow, but not
> > > to completely drop it from the tree.
> >
> > So you want to have a DT in the U-Boot tree that is completely unused?
> 
> Why would I want it to be unused?
> 
> It is used by the U-Boot build, at least.

To do what?

> Does this all work because the qemu support has been in there for ages
> and never changes? Otherwise I wonder about version compatibility.

Well, DT bindings are supposed to be stable.  See the discussion we
had earlier this week about the layerscape bindings...  So yes, I
think the folks involved on both the QEMU and Linux side are committed
to backwards compatibility here.

> > > Where is the qemu source code that creates these DTs?
> >
> > For ARM the code is in:
> >
> > https://github.com/qemu/qemu/blob/master/hw/arm/virt.c
> 
> Gosh that's 3k LOC that I didn't even imagine existed! Thank you for
> the pointer.
> 
> I don't see any mention in there about U-Boot. Are you sure that it is
> passed to U-Boot, or is it just passed to Linux? If U-Boot uses it,
> someone should make a note of that in the file.

I don't know why you want to single out U-Boot and Linux.  The DT
bindings are OS agnostic.  They work equally well for OpenBSD and
FreeBSD.  So why wouldn't they work for U-Boot?

> > > > > The OF_BOARD business is for when the board does special things.
> > > > > Presumably signing will do special things too. We cannot really know
> > > > > what those things are because the board as 'opted out' of the standard
> > > > > options.
> > > > >
> > > > >>
> > > > >> Please, describe CONFIG_OF_PRIOR_STAGE in
> > > > >> doc/develop/devicetree/control.rst.
> > > > >
> > > > > So I'm not allowed to delete that option? :-)
> > > >
> > > > No.
> > > >
> > > > > It seems to me to be
> > > > > extremely sparse on documentation. We need an explanation of what it
> > > > > means and what effect it has on the build system, U-Boot and some
> > > > > discussion of how qemu works. It seems to have been added as part of
> > > > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > > > anyone noticed it. Since then it has apparently proved useful
> > > > > elsewhere, but no one has added more docs.
> > > > >
> > > > > So perhaps you can help me with my doc by explaining how
> > > > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > > > which project actually hosts the DT that qemu provides? Armed with
> > > > > that information, I might be able to make sense of it all.
> > > >
> > > > The DT provided by QEMU is not hosted anywhere. It is generated on the
> > > > fly by QEMU in dependence of the command line arguments that are used
> > > > for calling QEMU. The project is hosted at
> > > >
> > > >      https://github.com/qemu/qemu.
> > >
> > > 404 on that. Do you have a link to the code that actually generates the DT?
> > >
> > > >
> > > > On RISC-V the address of the device-tree of the prior bootstage is
> > > > provided in register t0.
> > > >
> > > > On ARM platforms QEMU places the device-tree at 0x40000000.
> > > >
> > > > QEMU is not the only platform where the prior boot stage supplies the
> > > > device-tree which is to be used for booting the operating system. Any
> > > > platform using TF-A or OpenSBI can be setup in this way.
> > > >
> > > > Generally it would be preferable that the prior boot stage provides the
> > > > device-tree. But unfortunately Linux is not always backwards compatible.
> > > >
> > > > Don't expect the device-tree of the prior stage to contain any U-Boot
> > > > specific quirks.
> > >
> > > This is my concern. I think every board in U-Boot needs at least a
> > > basic DT, even if only a skeleton, so we have places to put things.
> > > Any packaging at all needs a binman node, for example. U-Boot also has
> > > its own options for certain things.
> >
> > Most of the options are for SPL though, and these platforms don't have
> > that as there is other firmware that does the low-level bringup.  And
> > these platforms typically don't need packaging.  I believe QEMU just
> > needs the ELF binary, and that's also the case for how we boot the
> > Apple M1 using m1n1.
> 
> You mean that you load u-boot (the ELF file) with no DT?

What actually happens is that m1n1, u-boot and the DT get concatenated
into a single file.  m1n1 recognizes the DT, modifies it as needed and
passes it along to the Linux kernel or U-Boot.  I suppose we could use
binman to do this concatenation if we really wanted to and have the
"base" device trees in the U-Boot repository.  But I expect that for
Asahi Linux we really want to have these in the m1n1 repository.  That
way we can add device trees for new Mac models quickly without being
tied to the U-Boot release cycle.

> > I suppose we could have a DT for these platforms that doesn't describe
> > the hardware and just does the config stuff.  But then you'll be
> 
> No, I'd like a basic DT that describes the hardware and has a node for
> each of the enabled drivres.

I suppose we can have one for documentation purposes.
 
> > juggling two DTs and have to make sure that the real one is passed on
> > to the kernel booted by U-Boot.  But I think your idea of having
> > everything configured in a single DT is flawed.
> 
> Perhaps the real issue here is that OF_PRIOR_STAGE is a build-time
> option. Perhaps it should be a run-time option, selected by the prior
> stage. If not provided, U-Bot uses its own DT?

In the case of the Apple M1 this makes no sense.  U-Boot has to use
the device tree that gets passed by m1n1.  Without the modifications
made by m1n1 some stuff simply won't work and the user experience will
suck.

> What is the flaw? Do you want to use overlays? I don't think it is
> unreasonable for U-Boot to have a DT in its own project. It has a
> defconfig, an environment and all the drivers.
> 
> >
> > > Also, where does the environment come from on these boards? Having the
> > > env in one tree and the DT in another must make things very
> > > interesting.
> >
> > What environment do you mean?  The U-Boot variables are defined in the
> > config files, not in the DT.
> 
> The U-Boot environment. Eg. here:
> 
> https://www.denx.de/wiki/DULG/UBootEnvVariables

Right.  I don't see how that is tied to the device tree.