Usage of device-tree for blobs

Usage of device-tree for blobs

[Heinrich Schuchardt]

Hello Simon,

some boards like qemu-riscv64_defconfig do not use any device-tree at
build time. A device-tree is only supplied at runtime by the prior boot
stage (CONFIG_OF_PRIOR_STAGE=y).

In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
the device-tree.

Could you, please, update the documentation to explain how adding blobs
to the device-tree works in the different scenarios depending on the
values of:

CONFIG_OF_EMBED
CONFIG_OF_SEPARATE
CONFIG_OF_BOARD
CONFIG_OF_HOSTFILE
CONFIG_OF_PRIOR_STAGE

It would be especially important to understand how one can develop a
board independent feature which works for all of these settings.

Please, describe CONFIG_OF_PRIOR_STAGE in
doc/develop/devicetree/control.rst.

Best regards

Heinrich

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Heinrich,

On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Hello Simon,
>
> some boards like qemu-riscv64_defconfig do not use any device-tree at
> build time. A device-tree is only supplied at runtime by the prior boot
> stage (CONFIG_OF_PRIOR_STAGE=y).
>
> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> the device-tree.
>
> Could you, please, update the documentation to explain how adding blobs
> to the device-tree works in the different scenarios depending on the
> values of:
>
> CONFIG_OF_EMBED
> CONFIG_OF_SEPARATE
> CONFIG_OF_BOARD
> CONFIG_OF_HOSTFILE
> CONFIG_OF_PRIOR_STAGE
>
> It would be especially important to understand how one can develop a
> board independent feature which works for all of these settings.

OK I will take a crack at this tomorrow.

But I think there is a disconnect here, since the only options that
matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
u-boot.dtb file. There is nothing tricky here.

The OF_BOARD business is for when the board does special things.
Presumably signing will do special things too. We cannot really know
what those things are because the board as 'opted out' of the standard
options.

>
> Please, describe CONFIG_OF_PRIOR_STAGE in
> doc/develop/devicetree/control.rst.

So I'm not allowed to delete that option? :-) It seems to me to be
extremely sparse on documentation. We need an explanation of what it
means and what effect it has on the build system, U-Boot and some
discussion of how qemu works. It seems to have been added as part of
an unrelated broadcom commit. The tags were incorrect so I doubt
anyone noticed it. Since then it has apparently proved useful
elsewhere, but no one has added more docs.

So perhaps you can help me with my doc by explaining how
OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
which project actually hosts the DT that qemu provides? Armed with
that information, I might be able to make sense of it all.

Regards,
Simon

Re: Usage of device-tree for blobs

[Heinrich Schuchardt]

On 8/26/21 5:15 AM, Simon Glass wrote:
> Hi Heinrich,
>
> On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>
>> Hello Simon,
>>
>> some boards like qemu-riscv64_defconfig do not use any device-tree at
>> build time. A device-tree is only supplied at runtime by the prior boot
>> stage (CONFIG_OF_PRIOR_STAGE=y).
>>
>> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
>> the device-tree.
>>
>> Could you, please, update the documentation to explain how adding blobs
>> to the device-tree works in the different scenarios depending on the
>> values of:
>>
>> CONFIG_OF_EMBED
>> CONFIG_OF_SEPARATE
>> CONFIG_OF_BOARD
>> CONFIG_OF_HOSTFILE
>> CONFIG_OF_PRIOR_STAGE
>>
>> It would be especially important to understand how one can develop a
>> board independent feature which works for all of these settings.
>
> OK I will take a crack at this tomorrow.
>
> But I think there is a disconnect here, since the only options that
> matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> u-boot.dtb file. There is nothing tricky here.

The following boards use OF_PRIO_STAGE:

* QEMU
* bcm7260_defconfig
* bcm7445_defconfig
* ae350_rv32_defconfig
* ae350_rv32_spl_defconfig
* ae350_rv64_defconfig
* ae350_rv64_spl_defconfig

>
> The OF_BOARD business is for when the board does special things.
> Presumably signing will do special things too. We cannot really know
> what those things are because the board as 'opted out' of the standard
> options.
>
>>
>> Please, describe CONFIG_OF_PRIOR_STAGE in
>> doc/develop/devicetree/control.rst.
>
> So I'm not allowed to delete that option? :-)

No.

> It seems to me to be
> extremely sparse on documentation. We need an explanation of what it
> means and what effect it has on the build system, U-Boot and some
> discussion of how qemu works. It seems to have been added as part of
> an unrelated broadcom commit. The tags were incorrect so I doubt
> anyone noticed it. Since then it has apparently proved useful
> elsewhere, but no one has added more docs.
>
> So perhaps you can help me with my doc by explaining how
> OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> which project actually hosts the DT that qemu provides? Armed with
> that information, I might be able to make sense of it all.

The DT provided by QEMU is not hosted anywhere. It is generated on the
fly by QEMU in dependence of the command line arguments that are used
for calling QEMU. The project is hosted at

     https://github.com/qemu/qemu.

On RISC-V the address of the device-tree of the prior bootstage is
provided in register t0.

On ARM platforms QEMU places the device-tree at 0x40000000.

QEMU is not the only platform where the prior boot stage supplies the
device-tree which is to be used for booting the operating system. Any
platform using TF-A or OpenSBI can be setup in this way.

Generally it would be preferable that the prior boot stage provides the
device-tree. But unfortunately Linux is not always backwards compatible.

Don't expect the device-tree of the prior stage to contain any U-Boot
specific quirks.

Best regards

Heinrich

Re: Usage of device-tree for blobs

[Mark Kettenis]

> From: Simon Glass <sjg@chromium.org>
> Date: Wed, 25 Aug 2021 21:15:00 -0600
> 
> Hi Heinrich,
> 
> On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >
> > Hello Simon,
> >
> > some boards like qemu-riscv64_defconfig do not use any device-tree at
> > build time. A device-tree is only supplied at runtime by the prior boot
> > stage (CONFIG_OF_PRIOR_STAGE=y).
> >
> > In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > the device-tree.
> >
> > Could you, please, update the documentation to explain how adding blobs
> > to the device-tree works in the different scenarios depending on the
> > values of:
> >
> > CONFIG_OF_EMBED
> > CONFIG_OF_SEPARATE
> > CONFIG_OF_BOARD
> > CONFIG_OF_HOSTFILE
> > CONFIG_OF_PRIOR_STAGE
> >
> > It would be especially important to understand how one can develop a
> > board independent feature which works for all of these settings.
> 
> OK I will take a crack at this tomorrow.
> 
> But I think there is a disconnect here, since the only options that
> matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> u-boot.dtb file. There is nothing tricky here.
> 
> The OF_BOARD business is for when the board does special things.
> Presumably signing will do special things too. We cannot really know
> what those things are because the board as 'opted out' of the standard
> options.
> 
> >
> > Please, describe CONFIG_OF_PRIOR_STAGE in
> > doc/develop/devicetree/control.rst.
> 
> So I'm not allowed to delete that option? :-) It seems to me to be
> extremely sparse on documentation. We need an explanation of what it
> means and what effect it has on the build system, U-Boot and some
> discussion of how qemu works. It seems to have been added as part of
> an unrelated broadcom commit. The tags were incorrect so I doubt
> anyone noticed it. Since then it has apparently proved useful
> elsewhere, but no one has added more docs.
> 
> So perhaps you can help me with my doc by explaining how
> OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> which project actually hosts the DT that qemu provides? Armed with
> that information, I might be able to make sense of it all.

Well, QEMU allows configuration of (emulated/paravirtualised) devices
from the command line.  So it is pretty much impossible for U-Boot to
provide a DT that matches that configuration without adding a lot of
code the dynamically build it from some sort of configuration
specification provided by QEMU to U-Boot.  But at that point QEMU
might as well provide the DT itself, don't you agree?

Anyway, replying to this thread since for the Apple M1 support that
I'm working on we're in a somewhat similar situation.  The Asahi Linux
project has implemented m1n1 as a bootloader and plans to use U-Boot
as a "payload" to implement booting a standard Linux distribution (and
other OSes).  In this scenario m1n1 actually provides the DT since it
parses Apple's version of the DT (which isn't in the standard DT
format) and adds/updates certain properties that depend on the actual
hardware it is running on.

For this code I use CONFIG_OF_BOARD and implement
board_fdt_blob_setup() to simple return a pointer to the DT that m1n1
set up.  But that seems to be exactly what CONFIG_OF_PRIOR_STAGE
does...

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Heinrich,


On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 8/26/21 5:15 AM, Simon Glass wrote:
> > Hi Heinrich,
> >
> > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >>
> >> Hello Simon,
> >>
> >> some boards like qemu-riscv64_defconfig do not use any device-tree at
> >> build time. A device-tree is only supplied at runtime by the prior boot
> >> stage (CONFIG_OF_PRIOR_STAGE=y).
> >>
> >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> >> the device-tree.
> >>
> >> Could you, please, update the documentation to explain how adding blobs
> >> to the device-tree works in the different scenarios depending on the
> >> values of:
> >>
> >> CONFIG_OF_EMBED
> >> CONFIG_OF_SEPARATE
> >> CONFIG_OF_BOARD
> >> CONFIG_OF_HOSTFILE
> >> CONFIG_OF_PRIOR_STAGE
> >>
> >> It would be especially important to understand how one can develop a
> >> board independent feature which works for all of these settings.
> >
> > OK I will take a crack at this tomorrow.
> >
> > But I think there is a disconnect here, since the only options that
> > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > u-boot.dtb file. There is nothing tricky here.
>
> The following boards use OF_PRIO_STAGE:
>
> * QEMU
> * bcm7260_defconfig
> * bcm7445_defconfig
> * ae350_rv32_defconfig
> * ae350_rv32_spl_defconfig
> * ae350_rv64_defconfig
> * ae350_rv64_spl_defconfig

Most of these seem OK as they have an in-tree DT. But the arm and
riscv qemus and the bcm builds do not:

bcm7260_defconfig
bcm7445_defconfig
configs/qemu_arm64_defconfig
configs/qemu_arm_defconfig
configs/qemu-ppce500_defconfig
configs/qemu-riscv32_defconfig
configs/qemu-riscv32_smode_defconfig
configs/qemu-riscv64_defconfig
configs/qemu-riscv64_smode_defconfig

I think we are going to have to ban this. We are not really testing
the build at all, and it presumably depends on the version of qemu
that is used. It's OK to provide the DT to U-Boot as one flow, but not
to completely drop it from the tree.

Where is the qemu source code that creates these DTs?

>
> >
> > The OF_BOARD business is for when the board does special things.
> > Presumably signing will do special things too. We cannot really know
> > what those things are because the board as 'opted out' of the standard
> > options.
> >
> >>
> >> Please, describe CONFIG_OF_PRIOR_STAGE in
> >> doc/develop/devicetree/control.rst.
> >
> > So I'm not allowed to delete that option? :-)
>
> No.
>
> > It seems to me to be
> > extremely sparse on documentation. We need an explanation of what it
> > means and what effect it has on the build system, U-Boot and some
> > discussion of how qemu works. It seems to have been added as part of
> > an unrelated broadcom commit. The tags were incorrect so I doubt
> > anyone noticed it. Since then it has apparently proved useful
> > elsewhere, but no one has added more docs.
> >
> > So perhaps you can help me with my doc by explaining how
> > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > which project actually hosts the DT that qemu provides? Armed with
> > that information, I might be able to make sense of it all.
>
> The DT provided by QEMU is not hosted anywhere. It is generated on the
> fly by QEMU in dependence of the command line arguments that are used
> for calling QEMU. The project is hosted at
>
>      https://github.com/qemu/qemu.

404 on that. Do you have a link to the code that actually generates the DT?

>
> On RISC-V the address of the device-tree of the prior bootstage is
> provided in register t0.
>
> On ARM platforms QEMU places the device-tree at 0x40000000.
>
> QEMU is not the only platform where the prior boot stage supplies the
> device-tree which is to be used for booting the operating system. Any
> platform using TF-A or OpenSBI can be setup in this way.
>
> Generally it would be preferable that the prior boot stage provides the
> device-tree. But unfortunately Linux is not always backwards compatible.
>
> Don't expect the device-tree of the prior stage to contain any U-Boot
> specific quirks.

This is my concern. I think every board in U-Boot needs at least a
basic DT, even if only a skeleton, so we have places to put things.
Any packaging at all needs a binman node, for example. U-Boot also has
its own options for certain things.

Also, where does the environment come from on these boards? Having the
env in one tree and the DT in another must make things very
interesting.

Regards,
Simon

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Mark,

On Thu, 26 Aug 2021 at 06:55, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
>
> > From: Simon Glass <sjg@chromium.org>
> > Date: Wed, 25 Aug 2021 21:15:00 -0600
> >
> > Hi Heinrich,
> >
> > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > >
> > > Hello Simon,
> > >
> > > some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > build time. A device-tree is only supplied at runtime by the prior boot
> > > stage (CONFIG_OF_PRIOR_STAGE=y).
> > >
> > > In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > the device-tree.
> > >
> > > Could you, please, update the documentation to explain how adding blobs
> > > to the device-tree works in the different scenarios depending on the
> > > values of:
> > >
> > > CONFIG_OF_EMBED
> > > CONFIG_OF_SEPARATE
> > > CONFIG_OF_BOARD
> > > CONFIG_OF_HOSTFILE
> > > CONFIG_OF_PRIOR_STAGE
> > >
> > > It would be especially important to understand how one can develop a
> > > board independent feature which works for all of these settings.
> >
> > OK I will take a crack at this tomorrow.
> >
> > But I think there is a disconnect here, since the only options that
> > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > u-boot.dtb file. There is nothing tricky here.
> >
> > The OF_BOARD business is for when the board does special things.
> > Presumably signing will do special things too. We cannot really know
> > what those things are because the board as 'opted out' of the standard
> > options.
> >
> > >
> > > Please, describe CONFIG_OF_PRIOR_STAGE in
> > > doc/develop/devicetree/control.rst.
> >
> > So I'm not allowed to delete that option? :-) It seems to me to be
> > extremely sparse on documentation. We need an explanation of what it
> > means and what effect it has on the build system, U-Boot and some
> > discussion of how qemu works. It seems to have been added as part of
> > an unrelated broadcom commit. The tags were incorrect so I doubt
> > anyone noticed it. Since then it has apparently proved useful
> > elsewhere, but no one has added more docs.
> >
> > So perhaps you can help me with my doc by explaining how
> > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > which project actually hosts the DT that qemu provides? Armed with
> > that information, I might be able to make sense of it all.
>
> Well, QEMU allows configuration of (emulated/paravirtualised) devices
> from the command line.  So it is pretty much impossible for U-Boot to
> provide a DT that matches that configuration without adding a lot of
> code the dynamically build it from some sort of configuration
> specification provided by QEMU to U-Boot.  But at that point QEMU
> might as well provide the DT itself, don't you agree?

Don't these devices have a DT in U-Boot? Is qemu enabling them (status
= "okay") or actually inventing them?!

>
> Anyway, replying to this thread since for the Apple M1 support that
> I'm working on we're in a somewhat similar situation.  The Asahi Linux
> project has implemented m1n1 as a bootloader and plans to use U-Boot
> as a "payload" to implement booting a standard Linux distribution (and
> other OSes).  In this scenario m1n1 actually provides the DT since it
> parses Apple's version of the DT (which isn't in the standard DT
> format) and adds/updates certain properties that depend on the actual
> hardware it is running on.

Yes I see. Still I'd like a basic DT in U-Boot, even if just for
discoverability.

>
> For this code I use CONFIG_OF_BOARD and implement
> board_fdt_blob_setup() to simple return a pointer to the DT that m1n1
> set up.  But that seems to be exactly what CONFIG_OF_PRIOR_STAGE
> does...

Right, and neither is really documented to the level that people can
understand the purpose. They just come across as hacks to me.

I'd like to see an API for passing a info (including DT) to U-Boot. I
think riscv has it? I suggest we use a register that points to a
bloblist.

Regards,
Simon

Re: Usage of device-tree for blobs

[Mark Kettenis]

> From: Simon Glass <sjg@chromium.org>
> Date: Thu, 26 Aug 2021 13:54:49 -0600
> 
> Hi Heinrich,
> 
> 
> On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >
> > On 8/26/21 5:15 AM, Simon Glass wrote:
> > > Hi Heinrich,
> > >
> > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > >>
> > >> Hello Simon,
> > >>
> > >> some boards like qemu-riscv64_defconfig do not use any device-tree at
> > >> build time. A device-tree is only supplied at runtime by the prior boot
> > >> stage (CONFIG_OF_PRIOR_STAGE=y).
> > >>
> > >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > >> the device-tree.
> > >>
> > >> Could you, please, update the documentation to explain how adding blobs
> > >> to the device-tree works in the different scenarios depending on the
> > >> values of:
> > >>
> > >> CONFIG_OF_EMBED
> > >> CONFIG_OF_SEPARATE
> > >> CONFIG_OF_BOARD
> > >> CONFIG_OF_HOSTFILE
> > >> CONFIG_OF_PRIOR_STAGE
> > >>
> > >> It would be especially important to understand how one can develop a
> > >> board independent feature which works for all of these settings.
> > >
> > > OK I will take a crack at this tomorrow.
> > >
> > > But I think there is a disconnect here, since the only options that
> > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > u-boot.dtb file. There is nothing tricky here.
> >
> > The following boards use OF_PRIO_STAGE:
> >
> > * QEMU
> > * bcm7260_defconfig
> > * bcm7445_defconfig
> > * ae350_rv32_defconfig
> > * ae350_rv32_spl_defconfig
> > * ae350_rv64_defconfig
> > * ae350_rv64_spl_defconfig
> 
> Most of these seem OK as they have an in-tree DT. But the arm and
> riscv qemus and the bcm builds do not:
> 
> bcm7260_defconfig
> bcm7445_defconfig
> configs/qemu_arm64_defconfig
> configs/qemu_arm_defconfig
> configs/qemu-ppce500_defconfig
> configs/qemu-riscv32_defconfig
> configs/qemu-riscv32_smode_defconfig
> configs/qemu-riscv64_defconfig
> configs/qemu-riscv64_smode_defconfig
> 
> I think we are going to have to ban this. We are not really testing
> the build at all, and it presumably depends on the version of qemu
> that is used. It's OK to provide the DT to U-Boot as one flow, but not
> to completely drop it from the tree.

So you want to have a DT in the U-Boot tree that is completely unused?

> Where is the qemu source code that creates these DTs?

For ARM the code is in:

https://github.com/qemu/qemu/blob/master/hw/arm/virt.c

> > > The OF_BOARD business is for when the board does special things.
> > > Presumably signing will do special things too. We cannot really know
> > > what those things are because the board as 'opted out' of the standard
> > > options.
> > >
> > >>
> > >> Please, describe CONFIG_OF_PRIOR_STAGE in
> > >> doc/develop/devicetree/control.rst.
> > >
> > > So I'm not allowed to delete that option? :-)
> >
> > No.
> >
> > > It seems to me to be
> > > extremely sparse on documentation. We need an explanation of what it
> > > means and what effect it has on the build system, U-Boot and some
> > > discussion of how qemu works. It seems to have been added as part of
> > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > anyone noticed it. Since then it has apparently proved useful
> > > elsewhere, but no one has added more docs.
> > >
> > > So perhaps you can help me with my doc by explaining how
> > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > which project actually hosts the DT that qemu provides? Armed with
> > > that information, I might be able to make sense of it all.
> >
> > The DT provided by QEMU is not hosted anywhere. It is generated on the
> > fly by QEMU in dependence of the command line arguments that are used
> > for calling QEMU. The project is hosted at
> >
> >      https://github.com/qemu/qemu.
> 
> 404 on that. Do you have a link to the code that actually generates the DT?
> 
> >
> > On RISC-V the address of the device-tree of the prior bootstage is
> > provided in register t0.
> >
> > On ARM platforms QEMU places the device-tree at 0x40000000.
> >
> > QEMU is not the only platform where the prior boot stage supplies the
> > device-tree which is to be used for booting the operating system. Any
> > platform using TF-A or OpenSBI can be setup in this way.
> >
> > Generally it would be preferable that the prior boot stage provides the
> > device-tree. But unfortunately Linux is not always backwards compatible.
> >
> > Don't expect the device-tree of the prior stage to contain any U-Boot
> > specific quirks.
> 
> This is my concern. I think every board in U-Boot needs at least a
> basic DT, even if only a skeleton, so we have places to put things.
> Any packaging at all needs a binman node, for example. U-Boot also has
> its own options for certain things.

Most of the options are for SPL though, and these platforms don't have
that as there is other firmware that does the low-level bringup.  And
these platforms typically don't need packaging.  I believe QEMU just
needs the ELF binary, and that's also the case for how we boot the
Apple M1 using m1n1.

I suppose we could have a DT for these platforms that doesn't describe
the hardware and just does the config stuff.  But then you'll be
juggling two DTs and have to make sure that the real one is passed on
to the kernel booted by U-Boot.  But I think your idea of having
everything configured in a single DT is flawed.

> Also, where does the environment come from on these boards? Having the
> env in one tree and the DT in another must make things very
> interesting.

What environment do you mean?  The U-Boot variables are defined in the
config files, not in the DT.

Re: Usage of device-tree for blobs

[Mark Kettenis]

> From: Simon Glass <sjg@chromium.org>
> Date: Thu, 26 Aug 2021 14:00:12 -0600
> 
> Hi Mark,
> 
> On Thu, 26 Aug 2021 at 06:55, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> >
> > > From: Simon Glass <sjg@chromium.org>
> > > Date: Wed, 25 Aug 2021 21:15:00 -0600
> > >
> > > Hi Heinrich,
> > >
> > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > >
> > > > Hello Simon,
> > > >
> > > > some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > > build time. A device-tree is only supplied at runtime by the prior boot
> > > > stage (CONFIG_OF_PRIOR_STAGE=y).
> > > >
> > > > In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > > the device-tree.
> > > >
> > > > Could you, please, update the documentation to explain how adding blobs
> > > > to the device-tree works in the different scenarios depending on the
> > > > values of:
> > > >
> > > > CONFIG_OF_EMBED
> > > > CONFIG_OF_SEPARATE
> > > > CONFIG_OF_BOARD
> > > > CONFIG_OF_HOSTFILE
> > > > CONFIG_OF_PRIOR_STAGE
> > > >
> > > > It would be especially important to understand how one can develop a
> > > > board independent feature which works for all of these settings.
> > >
> > > OK I will take a crack at this tomorrow.
> > >
> > > But I think there is a disconnect here, since the only options that
> > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > u-boot.dtb file. There is nothing tricky here.
> > >
> > > The OF_BOARD business is for when the board does special things.
> > > Presumably signing will do special things too. We cannot really know
> > > what those things are because the board as 'opted out' of the standard
> > > options.
> > >
> > > >
> > > > Please, describe CONFIG_OF_PRIOR_STAGE in
> > > > doc/develop/devicetree/control.rst.
> > >
> > > So I'm not allowed to delete that option? :-) It seems to me to be
> > > extremely sparse on documentation. We need an explanation of what it
> > > means and what effect it has on the build system, U-Boot and some
> > > discussion of how qemu works. It seems to have been added as part of
> > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > anyone noticed it. Since then it has apparently proved useful
> > > elsewhere, but no one has added more docs.
> > >
> > > So perhaps you can help me with my doc by explaining how
> > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > which project actually hosts the DT that qemu provides? Armed with
> > > that information, I might be able to make sense of it all.
> >
> > Well, QEMU allows configuration of (emulated/paravirtualised) devices
> > from the command line.  So it is pretty much impossible for U-Boot to
> > provide a DT that matches that configuration without adding a lot of
> > code the dynamically build it from some sort of configuration
> > specification provided by QEMU to U-Boot.  But at that point QEMU
> > might as well provide the DT itself, don't you agree?
> 
> Don't these devices have a DT in U-Boot? Is qemu enabling them (status
> = "okay") or actually inventing them?!

I actually invents them.  See the mail I just sent for a reference to
the code.

> > Anyway, replying to this thread since for the Apple M1 support that
> > I'm working on we're in a somewhat similar situation.  The Asahi Linux
> > project has implemented m1n1 as a bootloader and plans to use U-Boot
> > as a "payload" to implement booting a standard Linux distribution (and
> > other OSes).  In this scenario m1n1 actually provides the DT since it
> > parses Apple's version of the DT (which isn't in the standard DT
> > format) and adds/updates certain properties that depend on the actual
> > hardware it is running on.
> 
> Yes I see. Still I'd like a basic DT in U-Boot, even if just for
> discoverability.

Discoverability of what?

> > For this code I use CONFIG_OF_BOARD and implement
> > board_fdt_blob_setup() to simple return a pointer to the DT that m1n1
> > set up.  But that seems to be exactly what CONFIG_OF_PRIOR_STAGE
> > does...
> 
> Right, and neither is really documented to the level that people can
> understand the purpose. They just come across as hacks to me.

Well, I think U-Boot by its very nature is a giant hack ;).

> I'd like to see an API for passing a info (including DT) to U-Boot. I
> think riscv has it? I suggest we use a register that points to a
> bloblist.

Well, having a generic API will be hard.  It will depend on the
low-level firmware.  On RISC-V the SBI specification defines such an
API.  For the Apple M1 I just use the existing code that makes U-Boot
look like a Linux kernel and use the same API that is used to pass the
DTB to the kernel.  The Raspberry Pi code does something similar.  In
the end it just boils down to passing a pointer to the DT in a
specific register.

Re: Usage of device-tree for blobs

[Sean Anderson]

On 8/26/21 3:54 PM, Simon Glass wrote:
> Hi Heinrich,
> 
> 
> On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>
>> On 8/26/21 5:15 AM, Simon Glass wrote:
>> > Hi Heinrich,
>> >
>> > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>> >>
>> >> Hello Simon,
>> >>
>> >> some boards like qemu-riscv64_defconfig do not use any device-tree at
>> >> build time. A device-tree is only supplied at runtime by the prior boot
>> >> stage (CONFIG_OF_PRIOR_STAGE=y).
>> >>
>> >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
>> >> the device-tree.
>> >>
>> >> Could you, please, update the documentation to explain how adding blobs
>> >> to the device-tree works in the different scenarios depending on the
>> >> values of:
>> >>
>> >> CONFIG_OF_EMBED
>> >> CONFIG_OF_SEPARATE
>> >> CONFIG_OF_BOARD
>> >> CONFIG_OF_HOSTFILE
>> >> CONFIG_OF_PRIOR_STAGE
>> >>
>> >> It would be especially important to understand how one can develop a
>> >> board independent feature which works for all of these settings.
>> >
>> > OK I will take a crack at this tomorrow.
>> >
>> > But I think there is a disconnect here, since the only options that
>> > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
>> > u-boot.dtb file. There is nothing tricky here.
>>
>> The following boards use OF_PRIO_STAGE:
>>
>> * QEMU
>> * bcm7260_defconfig
>> * bcm7445_defconfig
>> * ae350_rv32_defconfig
>> * ae350_rv32_spl_defconfig
>> * ae350_rv64_defconfig
>> * ae350_rv64_spl_defconfig
> 
> Most of these seem OK as they have an in-tree DT. But the arm and
> riscv qemus and the bcm builds do not:
> 
> bcm7260_defconfig
> bcm7445_defconfig
> configs/qemu_arm64_defconfig
> configs/qemu_arm_defconfig
> configs/qemu-ppce500_defconfig
> configs/qemu-riscv32_defconfig
> configs/qemu-riscv32_smode_defconfig
> configs/qemu-riscv64_defconfig
> configs/qemu-riscv64_smode_defconfig
> 
> I think we are going to have to ban this. We are not really testing
> the build at all, and it presumably depends on the version of qemu
> that is used. It's OK to provide the DT to U-Boot as one flow, but not
> to completely drop it from the tree.
> 
> Where is the qemu source code that creates these DTs?
> 
>>
>> >
>> > The OF_BOARD business is for when the board does special things.
>> > Presumably signing will do special things too. We cannot really know
>> > what those things are because the board as 'opted out' of the standard
>> > options.
>> >
>> >>
>> >> Please, describe CONFIG_OF_PRIOR_STAGE in
>> >> doc/develop/devicetree/control.rst.
>> >
>> > So I'm not allowed to delete that option? :-)
>>
>> No.
>>
>> > It seems to me to be
>> > extremely sparse on documentation. We need an explanation of what it
>> > means and what effect it has on the build system, U-Boot and some
>> > discussion of how qemu works. It seems to have been added as part of
>> > an unrelated broadcom commit. The tags were incorrect so I doubt
>> > anyone noticed it. Since then it has apparently proved useful
>> > elsewhere, but no one has added more docs.
>> >
>> > So perhaps you can help me with my doc by explaining how
>> > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
>> > which project actually hosts the DT that qemu provides? Armed with
>> > that information, I might be able to make sense of it all.
>>
>> The DT provided by QEMU is not hosted anywhere. It is generated on the
>> fly by QEMU in dependence of the command line arguments that are used
>> for calling QEMU. The project is hosted at
>>
>>      https://github.com/qemu/qemu.
> 
> 404 on that. Do you have a link to the code that actually generates the DT?

https://github.com/qemu/qemu/blob/master/hw/riscv/virt.c#L179

> 
>>
>> On RISC-V the address of the device-tree of the prior bootstage is
>> provided in register t0.
>>
>> On ARM platforms QEMU places the device-tree at 0x40000000.
>>
>> QEMU is not the only platform where the prior boot stage supplies the
>> device-tree which is to be used for booting the operating system. Any
>> platform using TF-A or OpenSBI can be setup in this way.
>>
>> Generally it would be preferable that the prior boot stage provides the
>> device-tree. But unfortunately Linux is not always backwards compatible.
>>
>> Don't expect the device-tree of the prior stage to contain any U-Boot
>> specific quirks.
> 
> This is my concern. I think every board in U-Boot needs at least a
> basic DT, even if only a skeleton, so we have places to put things.
> Any packaging at all needs a binman node, for example. U-Boot also has
> its own options for certain things.
> 
> Also, where does the environment come from on these boards? Having the
> env in one tree and the DT in another must make things very
> interesting.
> 
> Regards,
> Simon
> 

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Mark,

On Thu, 26 Aug 2021 at 14:18, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
>
> > From: Simon Glass <sjg@chromium.org>
> > Date: Thu, 26 Aug 2021 13:54:49 -0600
> >
> > Hi Heinrich,
> >
> >
> > On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > >
> > > On 8/26/21 5:15 AM, Simon Glass wrote:
> > > > Hi Heinrich,
> > > >
> > > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > >>
> > > >> Hello Simon,
> > > >>
> > > >> some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > >> build time. A device-tree is only supplied at runtime by the prior boot
> > > >> stage (CONFIG_OF_PRIOR_STAGE=y).
> > > >>
> > > >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > >> the device-tree.
> > > >>
> > > >> Could you, please, update the documentation to explain how adding blobs
> > > >> to the device-tree works in the different scenarios depending on the
> > > >> values of:
> > > >>
> > > >> CONFIG_OF_EMBED
> > > >> CONFIG_OF_SEPARATE
> > > >> CONFIG_OF_BOARD
> > > >> CONFIG_OF_HOSTFILE
> > > >> CONFIG_OF_PRIOR_STAGE
> > > >>
> > > >> It would be especially important to understand how one can develop a
> > > >> board independent feature which works for all of these settings.
> > > >
> > > > OK I will take a crack at this tomorrow.
> > > >
> > > > But I think there is a disconnect here, since the only options that
> > > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > > u-boot.dtb file. There is nothing tricky here.
> > >
> > > The following boards use OF_PRIO_STAGE:
> > >
> > > * QEMU
> > > * bcm7260_defconfig
> > > * bcm7445_defconfig
> > > * ae350_rv32_defconfig
> > > * ae350_rv32_spl_defconfig
> > > * ae350_rv64_defconfig
> > > * ae350_rv64_spl_defconfig
> >
> > Most of these seem OK as they have an in-tree DT. But the arm and
> > riscv qemus and the bcm builds do not:
> >
> > bcm7260_defconfig
> > bcm7445_defconfig
> > configs/qemu_arm64_defconfig
> > configs/qemu_arm_defconfig
> > configs/qemu-ppce500_defconfig
> > configs/qemu-riscv32_defconfig
> > configs/qemu-riscv32_smode_defconfig
> > configs/qemu-riscv64_defconfig
> > configs/qemu-riscv64_smode_defconfig
> >
> > I think we are going to have to ban this. We are not really testing
> > the build at all, and it presumably depends on the version of qemu
> > that is used. It's OK to provide the DT to U-Boot as one flow, but not
> > to completely drop it from the tree.
>
> So you want to have a DT in the U-Boot tree that is completely unused?

Why would I want it to be unused?

It is used by the U-Boot build, at least.

Does this all work because the qemu support has been in there for ages
and never changes? Otherwise I wonder about version compatibility.

>
> > Where is the qemu source code that creates these DTs?
>
> For ARM the code is in:
>
> https://github.com/qemu/qemu/blob/master/hw/arm/virt.c

Gosh that's 3k LOC that I didn't even imagine existed! Thank you for
the pointer.

I don't see any mention in there about U-Boot. Are you sure that it is
passed to U-Boot, or is it just passed to Linux? If U-Boot uses it,
someone should make a note of that in the file.

>
> > > > The OF_BOARD business is for when the board does special things.
> > > > Presumably signing will do special things too. We cannot really know
> > > > what those things are because the board as 'opted out' of the standard
> > > > options.
> > > >
> > > >>
> > > >> Please, describe CONFIG_OF_PRIOR_STAGE in
> > > >> doc/develop/devicetree/control.rst.
> > > >
> > > > So I'm not allowed to delete that option? :-)
> > >
> > > No.
> > >
> > > > It seems to me to be
> > > > extremely sparse on documentation. We need an explanation of what it
> > > > means and what effect it has on the build system, U-Boot and some
> > > > discussion of how qemu works. It seems to have been added as part of
> > > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > > anyone noticed it. Since then it has apparently proved useful
> > > > elsewhere, but no one has added more docs.
> > > >
> > > > So perhaps you can help me with my doc by explaining how
> > > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > > which project actually hosts the DT that qemu provides? Armed with
> > > > that information, I might be able to make sense of it all.
> > >
> > > The DT provided by QEMU is not hosted anywhere. It is generated on the
> > > fly by QEMU in dependence of the command line arguments that are used
> > > for calling QEMU. The project is hosted at
> > >
> > >      https://github.com/qemu/qemu.
> >
> > 404 on that. Do you have a link to the code that actually generates the DT?
> >
> > >
> > > On RISC-V the address of the device-tree of the prior bootstage is
> > > provided in register t0.
> > >
> > > On ARM platforms QEMU places the device-tree at 0x40000000.
> > >
> > > QEMU is not the only platform where the prior boot stage supplies the
> > > device-tree which is to be used for booting the operating system. Any
> > > platform using TF-A or OpenSBI can be setup in this way.
> > >
> > > Generally it would be preferable that the prior boot stage provides the
> > > device-tree. But unfortunately Linux is not always backwards compatible.
> > >
> > > Don't expect the device-tree of the prior stage to contain any U-Boot
> > > specific quirks.
> >
> > This is my concern. I think every board in U-Boot needs at least a
> > basic DT, even if only a skeleton, so we have places to put things.
> > Any packaging at all needs a binman node, for example. U-Boot also has
> > its own options for certain things.
>
> Most of the options are for SPL though, and these platforms don't have
> that as there is other firmware that does the low-level bringup.  And
> these platforms typically don't need packaging.  I believe QEMU just
> needs the ELF binary, and that's also the case for how we boot the
> Apple M1 using m1n1.

You mean that you load u-boot (the ELF file) with no DT?

>
> I suppose we could have a DT for these platforms that doesn't describe
> the hardware and just does the config stuff.  But then you'll be

No, I'd like a basic DT that describes the hardware and has a node for
each of the enabled drivres.

> juggling two DTs and have to make sure that the real one is passed on
> to the kernel booted by U-Boot.  But I think your idea of having
> everything configured in a single DT is flawed.

Perhaps the real issue here is that OF_PRIOR_STAGE is a build-time
option. Perhaps it should be a run-time option, selected by the prior
stage. If not provided, U-Bot uses its own DT?

What is the flaw? Do you want to use overlays? I don't think it is
unreasonable for U-Boot to have a DT in its own project. It has a
defconfig, an environment and all the drivers.

>
> > Also, where does the environment come from on these boards? Having the
> > env in one tree and the DT in another must make things very
> > interesting.
>
> What environment do you mean?  The U-Boot variables are defined in the
> config files, not in the DT.

The U-Boot environment. Eg. here:

https://www.denx.de/wiki/DULG/UBootEnvVariables

Regards,
Simon

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Mark,

On Thu, 26 Aug 2021 at 14:27, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
>
> > From: Simon Glass <sjg@chromium.org>
> > Date: Thu, 26 Aug 2021 14:00:12 -0600
> >
> > Hi Mark,
> >
> > On Thu, 26 Aug 2021 at 06:55, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> > >
> > > > From: Simon Glass <sjg@chromium.org>
> > > > Date: Wed, 25 Aug 2021 21:15:00 -0600
> > > >
> > > > Hi Heinrich,
> > > >
> > > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > > >
> > > > > Hello Simon,
> > > > >
> > > > > some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > > > build time. A device-tree is only supplied at runtime by the prior boot
> > > > > stage (CONFIG_OF_PRIOR_STAGE=y).
> > > > >
> > > > > In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > > > the device-tree.
> > > > >
> > > > > Could you, please, update the documentation to explain how adding blobs
> > > > > to the device-tree works in the different scenarios depending on the
> > > > > values of:
> > > > >
> > > > > CONFIG_OF_EMBED
> > > > > CONFIG_OF_SEPARATE
> > > > > CONFIG_OF_BOARD
> > > > > CONFIG_OF_HOSTFILE
> > > > > CONFIG_OF_PRIOR_STAGE
> > > > >
> > > > > It would be especially important to understand how one can develop a
> > > > > board independent feature which works for all of these settings.
> > > >
> > > > OK I will take a crack at this tomorrow.
> > > >
> > > > But I think there is a disconnect here, since the only options that
> > > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > > u-boot.dtb file. There is nothing tricky here.
> > > >
> > > > The OF_BOARD business is for when the board does special things.
> > > > Presumably signing will do special things too. We cannot really know
> > > > what those things are because the board as 'opted out' of the standard
> > > > options.
> > > >
> > > > >
> > > > > Please, describe CONFIG_OF_PRIOR_STAGE in
> > > > > doc/develop/devicetree/control.rst.
> > > >
> > > > So I'm not allowed to delete that option? :-) It seems to me to be
> > > > extremely sparse on documentation. We need an explanation of what it
> > > > means and what effect it has on the build system, U-Boot and some
> > > > discussion of how qemu works. It seems to have been added as part of
> > > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > > anyone noticed it. Since then it has apparently proved useful
> > > > elsewhere, but no one has added more docs.
> > > >
> > > > So perhaps you can help me with my doc by explaining how
> > > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > > which project actually hosts the DT that qemu provides? Armed with
> > > > that information, I might be able to make sense of it all.
> > >
> > > Well, QEMU allows configuration of (emulated/paravirtualised) devices
> > > from the command line.  So it is pretty much impossible for U-Boot to
> > > provide a DT that matches that configuration without adding a lot of
> > > code the dynamically build it from some sort of configuration
> > > specification provided by QEMU to U-Boot.  But at that point QEMU
> > > might as well provide the DT itself, don't you agree?
> >
> > Don't these devices have a DT in U-Boot? Is qemu enabling them (status
> > = "okay") or actually inventing them?!
>
> I actually invents them.  See the mail I just sent for a reference to
> the code.

OK I see.

>
> > > Anyway, replying to this thread since for the Apple M1 support that
> > > I'm working on we're in a somewhat similar situation.  The Asahi Linux
> > > project has implemented m1n1 as a bootloader and plans to use U-Boot
> > > as a "payload" to implement booting a standard Linux distribution (and
> > > other OSes).  In this scenario m1n1 actually provides the DT since it
> > > parses Apple's version of the DT (which isn't in the standard DT
> > > format) and adds/updates certain properties that depend on the actual
> > > hardware it is running on.
> >
> > Yes I see. Still I'd like a basic DT in U-Boot, even if just for
> > discoverability.
>
> Discoverability of what?

Devices that are available / used on the board.

>
> > > For this code I use CONFIG_OF_BOARD and implement
> > > board_fdt_blob_setup() to simple return a pointer to the DT that m1n1
> > > set up.  But that seems to be exactly what CONFIG_OF_PRIOR_STAGE
> > > does...
> >
> > Right, and neither is really documented to the level that people can
> > understand the purpose. They just come across as hacks to me.
>
> Well, I think U-Boot by its very nature is a giant hack ;).

I'm not even sure where to go with that one...

>
> > I'd like to see an API for passing a info (including DT) to U-Boot. I
> > think riscv has it? I suggest we use a register that points to a
> > bloblist.
>
> Well, having a generic API will be hard.  It will depend on the
> low-level firmware.  On RISC-V the SBI specification defines such an
> API.  For the Apple M1 I just use the existing code that makes U-Boot
> look like a Linux kernel and use the same API that is used to pass the
> DTB to the kernel.  The Raspberry Pi code does something similar.  In
> the end it just boils down to passing a pointer to the DT in a
> specific register.

Yes that's what I'm talking about. if RISC-V does it, great. Perhaps
ARM can also?

Regards,
Simon

Re: Usage of device-tree for blobs

[Mark Kettenis]

> From: Simon Glass <sjg@chromium.org>
> Date: Thu, 26 Aug 2021 21:57:19 -0600
> Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>,
>         Ilias Apalodimas <ilias.apalodimas@linaro.org>,
>         U-Boot Mailing List <u-boot@lists.denx.de>
> Content-Type: text/plain; charset="UTF-8"
> 
> Hi Mark,
> 
> On Thu, 26 Aug 2021 at 14:18, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> >
> > > From: Simon Glass <sjg@chromium.org>
> > > Date: Thu, 26 Aug 2021 13:54:49 -0600
> > >
> > > Hi Heinrich,
> > >
> > >
> > > On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > >
> > > > On 8/26/21 5:15 AM, Simon Glass wrote:
> > > > > Hi Heinrich,
> > > > >
> > > > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > > >>
> > > > >> Hello Simon,
> > > > >>
> > > > >> some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > > >> build time. A device-tree is only supplied at runtime by the prior boot
> > > > >> stage (CONFIG_OF_PRIOR_STAGE=y).
> > > > >>
> > > > >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > > >> the device-tree.
> > > > >>
> > > > >> Could you, please, update the documentation to explain how adding blobs
> > > > >> to the device-tree works in the different scenarios depending on the
> > > > >> values of:
> > > > >>
> > > > >> CONFIG_OF_EMBED
> > > > >> CONFIG_OF_SEPARATE
> > > > >> CONFIG_OF_BOARD
> > > > >> CONFIG_OF_HOSTFILE
> > > > >> CONFIG_OF_PRIOR_STAGE
> > > > >>
> > > > >> It would be especially important to understand how one can develop a
> > > > >> board independent feature which works for all of these settings.
> > > > >
> > > > > OK I will take a crack at this tomorrow.
> > > > >
> > > > > But I think there is a disconnect here, since the only options that
> > > > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > > > u-boot.dtb file. There is nothing tricky here.
> > > >
> > > > The following boards use OF_PRIO_STAGE:
> > > >
> > > > * QEMU
> > > > * bcm7260_defconfig
> > > > * bcm7445_defconfig
> > > > * ae350_rv32_defconfig
> > > > * ae350_rv32_spl_defconfig
> > > > * ae350_rv64_defconfig
> > > > * ae350_rv64_spl_defconfig
> > >
> > > Most of these seem OK as they have an in-tree DT. But the arm and
> > > riscv qemus and the bcm builds do not:
> > >
> > > bcm7260_defconfig
> > > bcm7445_defconfig
> > > configs/qemu_arm64_defconfig
> > > configs/qemu_arm_defconfig
> > > configs/qemu-ppce500_defconfig
> > > configs/qemu-riscv32_defconfig
> > > configs/qemu-riscv32_smode_defconfig
> > > configs/qemu-riscv64_defconfig
> > > configs/qemu-riscv64_smode_defconfig
> > >
> > > I think we are going to have to ban this. We are not really testing
> > > the build at all, and it presumably depends on the version of qemu
> > > that is used. It's OK to provide the DT to U-Boot as one flow, but not
> > > to completely drop it from the tree.
> >
> > So you want to have a DT in the U-Boot tree that is completely unused?
> 
> Why would I want it to be unused?
> 
> It is used by the U-Boot build, at least.

To do what?

> Does this all work because the qemu support has been in there for ages
> and never changes? Otherwise I wonder about version compatibility.

Well, DT bindings are supposed to be stable.  See the discussion we
had earlier this week about the layerscape bindings...  So yes, I
think the folks involved on both the QEMU and Linux side are committed
to backwards compatibility here.

> > > Where is the qemu source code that creates these DTs?
> >
> > For ARM the code is in:
> >
> > https://github.com/qemu/qemu/blob/master/hw/arm/virt.c
> 
> Gosh that's 3k LOC that I didn't even imagine existed! Thank you for
> the pointer.
> 
> I don't see any mention in there about U-Boot. Are you sure that it is
> passed to U-Boot, or is it just passed to Linux? If U-Boot uses it,
> someone should make a note of that in the file.

I don't know why you want to single out U-Boot and Linux.  The DT
bindings are OS agnostic.  They work equally well for OpenBSD and
FreeBSD.  So why wouldn't they work for U-Boot?

> > > > > The OF_BOARD business is for when the board does special things.
> > > > > Presumably signing will do special things too. We cannot really know
> > > > > what those things are because the board as 'opted out' of the standard
> > > > > options.
> > > > >
> > > > >>
> > > > >> Please, describe CONFIG_OF_PRIOR_STAGE in
> > > > >> doc/develop/devicetree/control.rst.
> > > > >
> > > > > So I'm not allowed to delete that option? :-)
> > > >
> > > > No.
> > > >
> > > > > It seems to me to be
> > > > > extremely sparse on documentation. We need an explanation of what it
> > > > > means and what effect it has on the build system, U-Boot and some
> > > > > discussion of how qemu works. It seems to have been added as part of
> > > > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > > > anyone noticed it. Since then it has apparently proved useful
> > > > > elsewhere, but no one has added more docs.
> > > > >
> > > > > So perhaps you can help me with my doc by explaining how
> > > > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > > > which project actually hosts the DT that qemu provides? Armed with
> > > > > that information, I might be able to make sense of it all.
> > > >
> > > > The DT provided by QEMU is not hosted anywhere. It is generated on the
> > > > fly by QEMU in dependence of the command line arguments that are used
> > > > for calling QEMU. The project is hosted at
> > > >
> > > >      https://github.com/qemu/qemu.
> > >
> > > 404 on that. Do you have a link to the code that actually generates the DT?
> > >
> > > >
> > > > On RISC-V the address of the device-tree of the prior bootstage is
> > > > provided in register t0.
> > > >
> > > > On ARM platforms QEMU places the device-tree at 0x40000000.
> > > >
> > > > QEMU is not the only platform where the prior boot stage supplies the
> > > > device-tree which is to be used for booting the operating system. Any
> > > > platform using TF-A or OpenSBI can be setup in this way.
> > > >
> > > > Generally it would be preferable that the prior boot stage provides the
> > > > device-tree. But unfortunately Linux is not always backwards compatible.
> > > >
> > > > Don't expect the device-tree of the prior stage to contain any U-Boot
> > > > specific quirks.
> > >
> > > This is my concern. I think every board in U-Boot needs at least a
> > > basic DT, even if only a skeleton, so we have places to put things.
> > > Any packaging at all needs a binman node, for example. U-Boot also has
> > > its own options for certain things.
> >
> > Most of the options are for SPL though, and these platforms don't have
> > that as there is other firmware that does the low-level bringup.  And
> > these platforms typically don't need packaging.  I believe QEMU just
> > needs the ELF binary, and that's also the case for how we boot the
> > Apple M1 using m1n1.
> 
> You mean that you load u-boot (the ELF file) with no DT?

What actually happens is that m1n1, u-boot and the DT get concatenated
into a single file.  m1n1 recognizes the DT, modifies it as needed and
passes it along to the Linux kernel or U-Boot.  I suppose we could use
binman to do this concatenation if we really wanted to and have the
"base" device trees in the U-Boot repository.  But I expect that for
Asahi Linux we really want to have these in the m1n1 repository.  That
way we can add device trees for new Mac models quickly without being
tied to the U-Boot release cycle.

> > I suppose we could have a DT for these platforms that doesn't describe
> > the hardware and just does the config stuff.  But then you'll be
> 
> No, I'd like a basic DT that describes the hardware and has a node for
> each of the enabled drivres.

I suppose we can have one for documentation purposes.
 
> > juggling two DTs and have to make sure that the real one is passed on
> > to the kernel booted by U-Boot.  But I think your idea of having
> > everything configured in a single DT is flawed.
> 
> Perhaps the real issue here is that OF_PRIOR_STAGE is a build-time
> option. Perhaps it should be a run-time option, selected by the prior
> stage. If not provided, U-Bot uses its own DT?

In the case of the Apple M1 this makes no sense.  U-Boot has to use
the device tree that gets passed by m1n1.  Without the modifications
made by m1n1 some stuff simply won't work and the user experience will
suck.

> What is the flaw? Do you want to use overlays? I don't think it is
> unreasonable for U-Boot to have a DT in its own project. It has a
> defconfig, an environment and all the drivers.
> 
> >
> > > Also, where does the environment come from on these boards? Having the
> > > env in one tree and the DT in another must make things very
> > > interesting.
> >
> > What environment do you mean?  The U-Boot variables are defined in the
> > config files, not in the DT.
> 
> The U-Boot environment. Eg. here:
> 
> https://www.denx.de/wiki/DULG/UBootEnvVariables

Right.  I don't see how that is tied to the device tree.

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Heinrich,

On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Hello Simon,
>
> some boards like qemu-riscv64_defconfig do not use any device-tree at
> build time. A device-tree is only supplied at runtime by the prior boot
> stage (CONFIG_OF_PRIOR_STAGE=y).
>
> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> the device-tree.
>
> Could you, please, update the documentation to explain how adding blobs
> to the device-tree works in the different scenarios depending on the
> values of:
>
> CONFIG_OF_EMBED
> CONFIG_OF_SEPARATE
> CONFIG_OF_BOARD
> CONFIG_OF_HOSTFILE
> CONFIG_OF_PRIOR_STAGE
>
> It would be especially important to understand how one can develop a
> board independent feature which works for all of these settings.
>
> Please, describe CONFIG_OF_PRIOR_STAGE in
> doc/develop/devicetree/control.rst.

Back to this original request, I've taken a crack at this and sent out a patch.

Regards,
SImon

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Mark,

On Fri, 27 Aug 2021 at 13:52, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
>
> > From: Simon Glass <sjg@chromium.org>
> > Date: Thu, 26 Aug 2021 21:57:19 -0600
> > Cc: Heinrich Schuchardt <xypron.glpk@gmx.de>,
> >         Ilias Apalodimas <ilias.apalodimas@linaro.org>,
> >         U-Boot Mailing List <u-boot@lists.denx.de>
> > Content-Type: text/plain; charset="UTF-8"
> >
> > Hi Mark,
> >
> > On Thu, 26 Aug 2021 at 14:18, Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> > >
> > > > From: Simon Glass <sjg@chromium.org>
> > > > Date: Thu, 26 Aug 2021 13:54:49 -0600
> > > >
> > > > Hi Heinrich,
> > > >
> > > >
> > > > On Thu, 26 Aug 2021 at 01:10, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > > >
> > > > > On 8/26/21 5:15 AM, Simon Glass wrote:
> > > > > > Hi Heinrich,
> > > > > >
> > > > > > On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> > > > > >>
> > > > > >> Hello Simon,
> > > > > >>
> > > > > >> some boards like qemu-riscv64_defconfig do not use any device-tree at
> > > > > >> build time. A device-tree is only supplied at runtime by the prior boot
> > > > > >> stage (CONFIG_OF_PRIOR_STAGE=y).
> > > > > >>
> > > > > >> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> > > > > >> the device-tree.
> > > > > >>
> > > > > >> Could you, please, update the documentation to explain how adding blobs
> > > > > >> to the device-tree works in the different scenarios depending on the
> > > > > >> values of:
> > > > > >>
> > > > > >> CONFIG_OF_EMBED
> > > > > >> CONFIG_OF_SEPARATE
> > > > > >> CONFIG_OF_BOARD
> > > > > >> CONFIG_OF_HOSTFILE
> > > > > >> CONFIG_OF_PRIOR_STAGE
> > > > > >>
> > > > > >> It would be especially important to understand how one can develop a
> > > > > >> board independent feature which works for all of these settings.
> > > > > >
> > > > > > OK I will take a crack at this tomorrow.
> > > > > >
> > > > > > But I think there is a disconnect here, since the only options that
> > > > > > matter within U-Boot are OF_SEPARATE and OF_HOSTFILE, which both use a
> > > > > > u-boot.dtb file. There is nothing tricky here.
> > > > >
> > > > > The following boards use OF_PRIO_STAGE:
> > > > >
> > > > > * QEMU
> > > > > * bcm7260_defconfig
> > > > > * bcm7445_defconfig
> > > > > * ae350_rv32_defconfig
> > > > > * ae350_rv32_spl_defconfig
> > > > > * ae350_rv64_defconfig
> > > > > * ae350_rv64_spl_defconfig
> > > >
> > > > Most of these seem OK as they have an in-tree DT. But the arm and
> > > > riscv qemus and the bcm builds do not:
> > > >
> > > > bcm7260_defconfig
> > > > bcm7445_defconfig
> > > > configs/qemu_arm64_defconfig
> > > > configs/qemu_arm_defconfig
> > > > configs/qemu-ppce500_defconfig
> > > > configs/qemu-riscv32_defconfig
> > > > configs/qemu-riscv32_smode_defconfig
> > > > configs/qemu-riscv64_defconfig
> > > > configs/qemu-riscv64_smode_defconfig
> > > >
> > > > I think we are going to have to ban this. We are not really testing
> > > > the build at all, and it presumably depends on the version of qemu
> > > > that is used. It's OK to provide the DT to U-Boot as one flow, but not
> > > > to completely drop it from the tree.
> > >
> > > So you want to have a DT in the U-Boot tree that is completely unused?
> >
> > Why would I want it to be unused?
> >
> > It is used by the U-Boot build, at least.
>
> To do what?

Discoverability, documentation, for CI to check it. I cover that in
the doc I sent.

>
> > Does this all work because the qemu support has been in there for ages
> > and never changes? Otherwise I wonder about version compatibility.
>
> Well, DT bindings are supposed to be stable.  See the discussion we
> had earlier this week about the layerscape bindings...  So yes, I
> think the folks involved on both the QEMU and Linux side are committed
> to backwards compatibility here.

OK good. Then so long as U-Boot does the same, QEMU should be fine.

>
> > > > Where is the qemu source code that creates these DTs?
> > >
> > > For ARM the code is in:
> > >
> > > https://github.com/qemu/qemu/blob/master/hw/arm/virt.c
> >
> > Gosh that's 3k LOC that I didn't even imagine existed! Thank you for
> > the pointer.
> >
> > I don't see any mention in there about U-Boot. Are you sure that it is
> > passed to U-Boot, or is it just passed to Linux? If U-Boot uses it,
> > someone should make a note of that in the file.
>
> I don't know why you want to single out U-Boot and Linux.  The DT
> bindings are OS agnostic.  They work equally well for OpenBSD and
> FreeBSD.  So why wouldn't they work for U-Boot?

You are completely missing my point, sorry.

My problem with all of this is that I had no idea that it was going
on....and I'm fairly active in U-Boot on and off. I am even the FDT
maintainer in U-Boot. If I am not aware of it, how is anyone else
supposed to find out? For example I don't see mention of this in
doc/board/emulation/qemu-arm.rst

After close inspection I see a passing reference to it in the RISC-V
doc: doc/board/emulation/qemu-riscv.rst

"...and the memory node in the embedded DTB created by QEMU reflects..."

(and I note in that case that the ELF file is passed to QEMU which
doesn't seem right to me)

So however we got into this state (and I cover that somewhat in my
series). I would like to see things documented much more clearly.

>
> > > > > > The OF_BOARD business is for when the board does special things.
> > > > > > Presumably signing will do special things too. We cannot really know
> > > > > > what those things are because the board as 'opted out' of the standard
> > > > > > options.
> > > > > >
> > > > > >>
> > > > > >> Please, describe CONFIG_OF_PRIOR_STAGE in
> > > > > >> doc/develop/devicetree/control.rst.
> > > > > >
> > > > > > So I'm not allowed to delete that option? :-)
> > > > >
> > > > > No.
> > > > >
> > > > > > It seems to me to be
> > > > > > extremely sparse on documentation. We need an explanation of what it
> > > > > > means and what effect it has on the build system, U-Boot and some
> > > > > > discussion of how qemu works. It seems to have been added as part of
> > > > > > an unrelated broadcom commit. The tags were incorrect so I doubt
> > > > > > anyone noticed it. Since then it has apparently proved useful
> > > > > > elsewhere, but no one has added more docs.
> > > > > >
> > > > > > So perhaps you can help me with my doc by explaining how
> > > > > > OF_PRIOR_STAGE works in qmeu, why the DT in U-Boot cannot be used, and
> > > > > > which project actually hosts the DT that qemu provides? Armed with
> > > > > > that information, I might be able to make sense of it all.
> > > > >
> > > > > The DT provided by QEMU is not hosted anywhere. It is generated on the
> > > > > fly by QEMU in dependence of the command line arguments that are used
> > > > > for calling QEMU. The project is hosted at
> > > > >
> > > > >      https://github.com/qemu/qemu.
> > > >
> > > > 404 on that. Do you have a link to the code that actually generates the DT?
> > > >
> > > > >
> > > > > On RISC-V the address of the device-tree of the prior bootstage is
> > > > > provided in register t0.
> > > > >
> > > > > On ARM platforms QEMU places the device-tree at 0x40000000.
> > > > >
> > > > > QEMU is not the only platform where the prior boot stage supplies the
> > > > > device-tree which is to be used for booting the operating system. Any
> > > > > platform using TF-A or OpenSBI can be setup in this way.
> > > > >
> > > > > Generally it would be preferable that the prior boot stage provides the
> > > > > device-tree. But unfortunately Linux is not always backwards compatible.
> > > > >
> > > > > Don't expect the device-tree of the prior stage to contain any U-Boot
> > > > > specific quirks.
> > > >
> > > > This is my concern. I think every board in U-Boot needs at least a
> > > > basic DT, even if only a skeleton, so we have places to put things.
> > > > Any packaging at all needs a binman node, for example. U-Boot also has
> > > > its own options for certain things.
> > >
> > > Most of the options are for SPL though, and these platforms don't have
> > > that as there is other firmware that does the low-level bringup.  And
> > > these platforms typically don't need packaging.  I believe QEMU just
> > > needs the ELF binary, and that's also the case for how we boot the
> > > Apple M1 using m1n1.
> >
> > You mean that you load u-boot (the ELF file) with no DT?
>
> What actually happens is that m1n1, u-boot and the DT get concatenated
> into a single file.  m1n1 recognizes the DT, modifies it as needed and
> passes it along to the Linux kernel or U-Boot.  I suppose we could use
> binman to do this concatenation if we really wanted to and have the
> "base" device trees in the U-Boot repository.  But I expect that for
> Asahi Linux we really want to have these in the m1n1 repository.  That
> way we can add device trees for new Mac models quickly without being
> tied to the U-Boot release cycle.

Yes, please don't use ELF as it bypasses part of the U-Boot build system.

I am not worried about passing a DT to U-Boot, but there should be
something representative in the tree.

>
> > > I suppose we could have a DT for these platforms that doesn't describe
> > > the hardware and just does the config stuff.  But then you'll be
> >
> > No, I'd like a basic DT that describes the hardware and has a node for
> > each of the enabled drivres.
>
> I suppose we can have one for documentation purposes.

Yes please and thank you.

>
> > > juggling two DTs and have to make sure that the real one is passed on
> > > to the kernel booted by U-Boot.  But I think your idea of having
> > > everything configured in a single DT is flawed.
> >
> > Perhaps the real issue here is that OF_PRIOR_STAGE is a build-time
> > option. Perhaps it should be a run-time option, selected by the prior
> > stage. If not provided, U-Bot uses its own DT?
>
> In the case of the Apple M1 this makes no sense.  U-Boot has to use
> the device tree that gets passed by m1n1.  Without the modifications
> made by m1n1 some stuff simply won't work and the user experience will
> suck.

Again I think you are missing my point.

OF_PRIOR_STAGE should not mean there is no DT in U-Boot and it is
unable to produce a complete image. I am fine with passing a DT
through to U-Boot, But, as above, please put something in the U-Boot
tree and build it.

>
> > What is the flaw? Do you want to use overlays? I don't think it is
> > unreasonable for U-Boot to have a DT in its own project. It has a
> > defconfig, an environment and all the drivers.
> >
> > >
> > > > Also, where does the environment come from on these boards? Having the
> > > > env in one tree and the DT in another must make things very
> > > > interesting.
> > >
> > > What environment do you mean?  The U-Boot variables are defined in the
> > > config files, not in the DT.
> >
> > The U-Boot environment. Eg. here:
> >
> > https://www.denx.de/wiki/DULG/UBootEnvVariables
>
> Right.  I don't see how that is tied to the device tree.

My point is that the scheme here is able to update the devicetree
built into U-Boot but not the environment built into U-Boot (actually
I believe you can modify the env in DT these days). But both affect
how U-Boot operates. I suppose my larger point is that IMO none of
this appears to have been thought out very clearly, nor documented
anywhere in U-Boot. That's actually the theme of the last several
weeks for me and I really want to straighten this all out.

Regards,
Simon

Re: Usage of device-tree for blobs

[Simon Glass]

Hi Heinrich,

On Wed, 25 Aug 2021 at 02:05, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Hello Simon,
>
> some boards like qemu-riscv64_defconfig do not use any device-tree at
> build time. A device-tree is only supplied at runtime by the prior boot
> stage (CONFIG_OF_PRIOR_STAGE=y).
>
> In doc/develop/devicetree/intro.rst you suggest to put binary blobs into
> the device-tree.
>
> Could you, please, update the documentation to explain how adding blobs
> to the device-tree works in the different scenarios depending on the
> values of:
>
> CONFIG_OF_EMBED
> CONFIG_OF_SEPARATE
> CONFIG_OF_BOARD
> CONFIG_OF_HOSTFILE
> CONFIG_OF_PRIOR_STAGE
>
> It would be especially important to understand how one can develop a
> board independent feature which works for all of these settings.
>
> Please, describe CONFIG_OF_PRIOR_STAGE in
> doc/develop/devicetree/control.rst.

+Tom Rini

I have now sent documentation on this, as requested.

Please can you stop waiting, and apply the revert series that I
requested 6 weeks ago now? If this mess goes out in a release I will
be extremely unhappy.

Regards,
Simon