archive mirror
 help / color / mirror / Atom feed
From: David Gibson <>
To: Rob Herring <>
Cc: "Frank Rowand"
	"Devicetree Compiler"
	"Jon Loeliger" <jdl-CYoMK+44s/>,
	"Pantelis Antoniou"
	"Grant Likely"
	"Marek Vašut"
	"Tom Rini" <trini-OWPKS81ov/FWk0Htik3J/>,
	"Kyle Evans" <>,
	"Geert Uytterhoeven"
	"Alan Tull" <>,
	"Michael Ellerman" <mpe-Gsx/>
Subject: Re: [RFC] devicetree: new FDT format version
Date: Tue, 23 Jan 2018 23:42:32 +1100	[thread overview]
Message-ID: <20180123124232.GA14832@umbus> (raw)
In-Reply-To: <>

[-- Attachment #1: Type: text/plain, Size: 6508 bytes --]

On Mon, Jan 22, 2018 at 03:01:52PM -0600, Rob Herring wrote:
> On Mon, Jan 22, 2018 at 2:09 AM, Frank Rowand <> wrote:
> > Hi All,
> >
> > I've tried to create a decent distribution list, but I'm sure I've missed
> > someone or some important list.  Please share this with anyone you think
> > will be affected.
> >
> > I have been playing around with some thoughts for some additions to
> > the devicetree FDT aka blob format.
> >
> > I would like to get the affected parties thinking about how additions to
> > the format could improve whichever pieces of FDT related technology you
> > work on or care about.  In my opinion, the FDT format should change
> > very infrequently because of the impact on so many projects that have
> > to work together to create a final solution, plus the many many users
> > of those projects.
> A few things discussed before:
> - Adding type information Even just tagging phandles would be good.

I'm a bit dubious about this.  It would have to be "hints" only -
there's not really anyway we can put in authoritative type
information, since dtc itself doesn't really know that.  It's also
hard to see how that could be done in a way which wouldn't either a)
require very awkward parallel lookup of the data and type information
or b) not be backwards compatible (even read only).

> - Allow applying overlays by just appending to the blob. The need for
> this is somewhat gone now that libfdt can just fully apply overlays.

I'm not really sure what you want here.  I mean you could easily allow
the format to allow multiple appended overlays, and define that to
mean the overlaid result.  At some point *something* is going to have
to really do the application, so I'm not sure that it really buys you
that much.  It also makes nearly every operation on the tree in libfdt
horrible to implement, at least within the other constraints the
interface was designed around; you'll continually have to scan the
entire tree just in case some other overlay fragment has some bearing
on the thing you're looking at.  It confuses the interface too: what
does "node offset" mean if the same node could be built up from
overlay fragments at multiple offsets.

> - Move to an unflattened format so we don't have to unflatten the
> tree. Or to put it another way, extend FDT enough that the tree can be
> walked and manipulated efficiently. I think this would involve storing
> node offsets for all the nodes. Maybe that could be combined with
> storage for __symbols__?

I don't think this is feasible.  If you want an unflattened tree
you'll have to change things so much that any superficial resemblence
to dtb will just be misleading.  You're likely to increase the size in
nearly all cases (which seems to be a concern to people).  You're also
almost certain to lose the benefits the serialized approach was
written for in the first place: chiefly that you can make edits
locally, using inserts and deletes but without having to adjust
offsets or handles anywhere else in the structure.

Making a portable "libdt" that manages an in-memory unflattened tree
with pointers, and can serialize and deserialze to dtb makes sense,
IMO.  Trying to make the flattened and unflattened trees the *same*
format does not.

Basically, efficient runtime manipulation is out of scope for the dtb
format - that's not what it's for.  If you're doing non-trivial
manipulation you really should be unflattening, then doing your work,
and if necessary re-flattening afterwards.  Unfortunately, I think the
fact you can manipulate while still in flattened format (and libfdt
even makes it pretty easy to write code to do so) has meant a bunch of
projects have postponed going to an unflattening model rather past the
point they should have.

> I also am not happy that labels which were purely source level
> convenience are now part of the ABI with overlays.

Yeah :/.  I keep harping on it, but overlays really are a hack.  I
think fleshing out the "connectors" approach that has been suggested
is really a better approach than trying to make the hack less hacky.

> > So I would like you guys to consider what I send out in a day or so,
> > but I don't want to preempt your creativity by laying out the details
> > of my proposal right now.
> >
> > I have not looked at how this would impact the devicetree compilers,
> > but I have hacked together a tool to convert existing blobs to the
> > new format.  The new format is backward compatible, but transforms
> > the overlay related metadata into separate blocks and removes the
> > metadata from nodes and properties.  My current proposal leaves
> > the fragment subtrees intact - it only transforms __symbols__,
> > __fixups__, and __local_fixups__.
> >
> > Some Advantages and disadvantages of my proposal are:
> >
> > Con:
> >   - New blob version.
> >
> > Pro:
> >   - Backward compatible.  Bootloaders and kernels that can process v17 blobs
> >     will continue to work in the same manner with a v18 blob.  They will not
> >     be able to use the new v18 features.
> What does libfdt do with a v18 blob? I'd assume it was written to
> treat new versions the same as the last known version.

So, the header includes a "last compatible version".  libfdt will
attempt to work with it as long as that is <= 17, regardless of the
version field.  Assuming version >= 17 then, yes, libfdt will treat
the blob as if it was v17, the latest one it knows.

If you alter the tree, libfdt will downrev the 'version' field to the
latest one it's aware of (i.e. 17).  Current dtc will do this on any
write operation (unless they don't call fdt_rw_check_header_(), which
would be a bug).  I'd have to check, but I think older versions may
only do it at the time of fdt_open_into(), which is part of the normal
expected flow for read-write operations, but doesn't technically
*have* to be called.

The "in place write" functions (fdt_wip.c) _won't_ downrev the
version, which is arguably a bug.  Most obvious extensions would not
be broken by such in-place writes, but it's not impossible (and
something which stored type data in parallel with the properties could
be an example).

David Gibson			| I'll have my music baroque, and my code
david AT	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

  parent reply	other threads:[~2018-01-23 12:42 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-01-22  8:09 Frank Rowand
     [not found] ` <>
2018-01-22  8:14   ` Frank Rowand
     [not found]     ` <>
2018-01-22 20:08       ` Frank Rowand
2018-01-22 20:08   ` Frank Rowand
     [not found]     ` <>
2018-01-25  0:27       ` Frank Rowand
     [not found]         ` <>
2018-01-27  8:48           ` David Gibson
2018-01-29  8:08             ` Frank Rowand
     [not found]               ` <>
2018-01-29 10:56                 ` David Gibson
2018-01-30  1:29                   ` Frank Rowand
2018-01-22 21:01   ` Rob Herring
     [not found]     ` <>
2018-01-23 12:42       ` David Gibson [this message]
2018-01-23 21:17         ` Frank Rowand
     [not found]           ` <>
2018-01-24 15:47             ` Rob Herring
     [not found]               ` <>
2018-01-24 21:16                 ` Frank Rowand
     [not found]                   ` <>
2018-01-24 22:27                     ` Alan Tull
2018-01-25  0:22                     ` Frank Rowand
     [not found]                       ` <>
2018-01-25 12:29                         ` David Gibson
2018-01-25 20:01                           ` Frank Rowand
     [not found]                             ` <>
2018-01-29 18:32                               ` Grant Likely
     [not found]                                 ` <>
2018-01-29 23:15                                   ` David Gibson
2018-01-26  8:56                           ` Geert Uytterhoeven
     [not found]                             ` <>
2018-01-26  8:59                               ` Geert Uytterhoeven
2018-01-26 22:08                               ` Frank Rowand
     [not found]                                 ` <>
2018-01-27  9:00                                   ` David Gibson
2018-01-27  8:56                               ` David Gibson
2018-01-25 23:11                     ` Frank Rowand
2018-01-25 12:22                 ` David Gibson
2018-01-25  9:14             ` Marek Vasut
     [not found]               ` <>
2018-01-25 12:37                 ` David Gibson
2018-01-27 20:30                   ` Marek Vasut
     [not found]                     ` <>
2018-01-29  0:53                       ` David Gibson
2018-01-23 12:05   ` David Gibson
2018-01-23 21:28     ` Frank Rowand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180123124232.GA14832@umbus \ \ \ \ \ \ \ \
    --cc=grant.likely-s3s/ \
    --cc=jdl-CYoMK+44s/ \ \ \
    --cc=mpe-Gsx/ \
    --cc=pantelis.antoniou-OWPKS81ov/FWk0Htik3J/ \
    --cc=panto-wVdstyuyKrO8r51toPun2/ \ \
    --cc=trini-OWPKS81ov/FWk0Htik3J/ \
    --subject='Re: [RFC] devicetree: new FDT format version' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).