Re: Usage of device-tree for blobs

[Simon Glass <sjg@chromium.org>]

Hi Mark,

On Fri, 27 Aug 2021 at 13:52, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
>
> > From: Simon Glass <sjg@chromium.org>
> > Date: Thu, 26 Aug 2021 21:57:19 -0600
> > Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>,
> >         Ilias Apalodimas <ilias.apalodimas@linaro.org>,
> >         U-Boot Mailing List <u-boot@lists.denx.de>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Mark,
> >
> > On Thu, 26 Aug 2021 at 14:18, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> > >
> > > > From: Simon Glass <sjg@chromium.org>
> > > > Date: Thu, 26 Aug 2021 13:54:49 -0600
> > > >
> > > > Hi Heinrich,
> > > >
> > > >
> > > > On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > > >
> > > > > On 8/26/21 5:15 AM, Simon Glass wrote:
> > > > > > Hi Heinrich,
> > > > > >
> > > > > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > > > >>
> > > > > >> Hello Simon,
> > > > > >>
> > > > > >> some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > > > >> build time. A device-tree is only supplied at runtime by the prior boot
> > > > > >> stage (CONFIG_OF_PRIOR_STAGE=y).
> > > > > >>
> > > > > >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > > > >> the device-tree.
> > > > > >>
> > > > > >> Could you, please, update the documentation to explain how adding blobs
> > > > > >> to the device-tree works in the different scenarios depending on the
> > > > > >> values of:
> > > > > >>
> > > > > >> CONFIG_OF_EMBED
> > > > > >> CONFIG_OF_SEPARATE
> > > > > >> CONFIG_OF_BOARD
> > > > > >> CONFIG_OF_HOSTFILE
> > > > > >> CONFIG_OF_PRIOR_STAGE
> > > > > >>
> > > > > >> It would be especially important to understand how one can develop a
> > > > > >> board independent feature which works for all of these settings.
> > > > > >
> > > > > > OK I will take a crack at this tomorrow.
> > > > > >
> > > > > > But I think there is a disconnect here, since the only options that
> > > > > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > > > > u-boot.dtb file. There is nothing tricky here.
> > > > >
> > > > > The following boards use OF_PRIO_STAGE:
> > > > >
> > > > > * QEMU
> > > > > * bcm7260_defconfig
> > > > > * bcm7445_defconfig
> > > > > * ae350_rv32_defconfig
> > > > > * ae350_rv32_spl_defconfig
> > > > > * ae350_rv64_defconfig
> > > > > * ae350_rv64_spl_defconfig
> > > >
> > > > Most of these seem OK as they have an in-tree DT. But the arm and
> > > > riscv qemus and the bcm builds do not:
> > > >
> > > > bcm7260_defconfig
> > > > bcm7445_defconfig
> > > > configs/qemu_arm64_defconfig
> > > > configs/qemu_arm_defconfig
> > > > configs/qemu-ppce500_defconfig
> > > > configs/qemu-riscv32_defconfig
> > > > configs/qemu-riscv32_smode_defconfig
> > > > configs/qemu-riscv64_defconfig
> > > > configs/qemu-riscv64_smode_defconfig
> > > >
> > > > I think we are going to have to ban this. We are not really testing
> > > > the build at all, and it presumably depends on the version of qemu
> > > > that is used. It's OK to provide the DT to U-Boot as one flow, but not
> > > > to completely drop it from the tree.
> > >
> > > So you want to have a DT in the U-Boot tree that is completely unused?
> >
> > Why would I want it to be unused?
> >
> > It is used by the U-Boot build, at least.
>
> To do what?

Discoverability, documentation, for CI to check it. I cover that in
the doc I sent.

>
> > Does this all work because the qemu support has been in there for ages
> > and never changes? Otherwise I wonder about version compatibility.
>
> Well, DT bindings are supposed to be stable.  See the discussion we
> had earlier this week about the layerscape bindings...  So yes, I
> think the folks involved on both the QEMU and Linux side are committed
> to backwards compatibility here.

OK good. Then so long as U-Boot does the same, QEMU should be fine.

>
> > > > Where is the qemu source code that creates these DTs?
> > >
> > > For ARM the code is in:
> > >
> > > https://github.com/qemu/qemu/blob/master/hw/arm/virt.c
> >
> > Gosh that's 3k LOC that I didn't even imagine existed! Thank you for
> > the pointer.
> >
> > I don't see any mention in there about U-Boot. Are you sure that it is
> > passed to U-Boot, or is it just passed to Linux? If U-Boot uses it,
> > someone should make a note of that in the file.
>
> I don't know why you want to single out U-Boot and Linux.  The DT
> bindings are OS agnostic.  They work equally well for OpenBSD and
> FreeBSD.  So why wouldn't they work for U-Boot?

You are completely missing my point, sorry.

My problem with all of this is that I had no idea that it was going
on....and I'm fairly active in U-Boot on and off. I am even the FDT
maintainer in U-Boot. If I am not aware of it, how is anyone else
supposed to find out? For example I don't see mention of this in
doc/board/emulation/qemu-arm.rst

After close inspection I see a passing reference to it in the RISC-V
doc: doc/board/emulation/qemu-riscv.rst

"...and the memory node in the embedded DTB created by QEMU reflects..."

(and I note in that case that the ELF file is passed to QEMU which
doesn't seem right to me)

So however we got into this state (and I cover that somewhat in my
series). I would like to see things documented much more clearly.

>
> > > > > > The OF_BOARD business is for when the board does special things.
> > > > > > Presumably signing will do special things too. We cannot really know
> > > > > > what those things are because the board as 'opted out' of the standard
> > > > > > options.
> > > > > >
> > > > > >>
> > > > > >> Please, describe CONFIG_OF_PRIOR_STAGE in
> > > > > >> doc/develop/devicetree/control.rst.
> > > > > >
> > > > > > So I'm not allowed to delete that option? :-)
> > > > >
> > > > > No.
> > > > >
> > > > > > It seems to me to be
> > > > > > extremely sparse on documentation. We need an explanation of what it
> > > > > > means and what effect it has on the build system, U-Boot and some
> > > > > > discussion of how qemu works. It seems to have been added as part of
> > > > > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > > > > anyone noticed it. Since then it has apparently proved useful
> > > > > > elsewhere, but no one has added more docs.
> > > > > >
> > > > > > So perhaps you can help me with my doc by explaining how
> > > > > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > > > > which project actually hosts the DT that qemu provides? Armed with
> > > > > > that information, I might be able to make sense of it all.
> > > > >
> > > > > The DT provided by QEMU is not hosted anywhere. It is generated on the
> > > > > fly by QEMU in dependence of the command line arguments that are used
> > > > > for calling QEMU. The project is hosted at
> > > > >
> > > > >      https://github.com/qemu/qemu.
> > > >
> > > > 404 on that. Do you have a link to the code that actually generates the DT?
> > > >
> > > > >
> > > > > On RISC-V the address of the device-tree of the prior bootstage is
> > > > > provided in register t0.
> > > > >
> > > > > On ARM platforms QEMU places the device-tree at 0x40000000.
> > > > >
> > > > > QEMU is not the only platform where the prior boot stage supplies the
> > > > > device-tree which is to be used for booting the operating system. Any
> > > > > platform using TF-A or OpenSBI can be setup in this way.
> > > > >
> > > > > Generally it would be preferable that the prior boot stage provides the
> > > > > device-tree. But unfortunately Linux is not always backwards compatible.
> > > > >
> > > > > Don't expect the device-tree of the prior stage to contain any U-Boot
> > > > > specific quirks.
> > > >
> > > > This is my concern. I think every board in U-Boot needs at least a
> > > > basic DT, even if only a skeleton, so we have places to put things.
> > > > Any packaging at all needs a binman node, for example. U-Boot also has
> > > > its own options for certain things.
> > >
> > > Most of the options are for SPL though, and these platforms don't have
> > > that as there is other firmware that does the low-level bringup.  And
> > > these platforms typically don't need packaging.  I believe QEMU just
> > > needs the ELF binary, and that's also the case for how we boot the
> > > Apple M1 using m1n1.
> >
> > You mean that you load u-boot (the ELF file) with no DT?
>
> What actually happens is that m1n1, u-boot and the DT get concatenated
> into a single file.  m1n1 recognizes the DT, modifies it as needed and
> passes it along to the Linux kernel or U-Boot.  I suppose we could use
> binman to do this concatenation if we really wanted to and have the
> "base" device trees in the U-Boot repository.  But I expect that for
> Asahi Linux we really want to have these in the m1n1 repository.  That
> way we can add device trees for new Mac models quickly without being
> tied to the U-Boot release cycle.

Yes, please don't use ELF as it bypasses part of the U-Boot build system.

I am not worried about passing a DT to U-Boot, but there should be
something representative in the tree.

>
> > > I suppose we could have a DT for these platforms that doesn't describe
> > > the hardware and just does the config stuff.  But then you'll be
> >
> > No, I'd like a basic DT that describes the hardware and has a node for
> > each of the enabled drivres.
>
> I suppose we can have one for documentation purposes.

Yes please and thank you.

>
> > > juggling two DTs and have to make sure that the real one is passed on
> > > to the kernel booted by U-Boot.  But I think your idea of having
> > > everything configured in a single DT is flawed.
> >
> > Perhaps the real issue here is that OF_PRIOR_STAGE is a build-time
> > option. Perhaps it should be a run-time option, selected by the prior
> > stage. If not provided, U-Bot uses its own DT?
>
> In the case of the Apple M1 this makes no sense.  U-Boot has to use
> the device tree that gets passed by m1n1.  Without the modifications
> made by m1n1 some stuff simply won't work and the user experience will
> suck.

Again I think you are missing my point.

OF_PRIOR_STAGE should not mean there is no DT in U-Boot and it is
unable to produce a complete image. I am fine with passing a DT
through to U-Boot, But, as above, please put something in the U-Boot
tree and build it.

>
> > What is the flaw? Do you want to use overlays? I don't think it is
> > unreasonable for U-Boot to have a DT in its own project. It has a
> > defconfig, an environment and all the drivers.
> >
> > >
> > > > Also, where does the environment come from on these boards? Having the
> > > > env in one tree and the DT in another must make things very
> > > > interesting.
> > >
> > > What environment do you mean?  The U-Boot variables are defined in the
> > > config files, not in the DT.
> >
> > The U-Boot environment. Eg. here:
> >
> > https://www.denx.de/wiki/DULG/UBootEnvVariables
>
> Right.  I don't see how that is tied to the device tree.

My point is that the scheme here is able to update the devicetree
built into U-Boot but not the environment built into U-Boot (actually
I believe you can modify the env in DT these days). But both affect
how U-Boot operates. I suppose my larger point is that IMO none of
this appears to have been thought out very clearly, nor documented
anywhere in U-Boot. That's actually the theme of the last several
weeks for me and I really want to straighten this all out.

Regards,
Simon