devicetree.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Frank Rowand <frowand.list-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
To: David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org>
Cc: Devicetree Compiler
	<devicetree-compiler-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	"devicetree-spec-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<devicetree-spec-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	Jon Loeliger <jdl-CYoMK+44s/E@public.gmane.org>,
	"devicetree-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<devicetree-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	Rob Herring <robh+dt-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>,
	Pantelis Antoniou
	<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>,
	"Pantelis
	Antomarek.vasut-Re5JQEeQqe/2sr8fMPgRzw@public.gmane.org"
	<panto-wVdstyuyKrO8r51toPun2/C9HSW9iNxf@public.gmane.org>,
	Grant Likely
	<grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org>,
	marek.vasut-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org,
	Tom Rini <trini-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>,
	Kyle Evans <kevans-HZy0K5TPuP5AfugRpC6u6w@public.gmane.org>,
	Geert Uytterhoeven
	<geert-Td1EMuHUCqxL1ZNQvxDV9g@public.gmane.org>,
	Alan Tull <atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>,
	Michael Ellerman <mpe-Gsx/Oe8HsFggBc27wqDAHg@public.gmane.org>
Subject: Re: [RFC] devicetree: new FDT format version
Date: Tue, 23 Jan 2018 13:28:59 -0800	[thread overview]
Message-ID: <1b85224d-0ee0-bf63-05da-aa4c99ff3342@gmail.com> (raw)
In-Reply-To: <20180123120500.GJ11419@umbus>

On 01/23/18 04:05, David Gibson wrote:
> On Mon, Jan 22, 2018 at 12:09:18AM -0800, Frank Rowand wrote:
>> Hi All,
>>
>> I've tried to create a decent distribution list, but I'm sure I've missed
>> someone or some important list.  Please share this with anyone you think
>> will be affected.
>>
>> I have been playing around with some thoughts for some additions to
>> the devicetree FDT aka blob format.
>>
>> I would like to get the affected parties thinking about how additions to
>> the format could improve whichever pieces of FDT related technology you
>> work on or care about.  In my opinion, the FDT format should change
>> very infrequently because of the impact on so many projects that have
>> to work together to create a final solution, plus the many many users
>> of those projects.
>>
>> So I would like you guys to consider what I send out in a day or so,
>> but I don't want to preempt your creativity by laying out the details
>> of my proposal right now.
>>
>> I have not looked at how this would impact the devicetree compilers,
>> but I have hacked together a tool to convert existing blobs to the
>> new format.  The new format is backward compatible, but transforms
>> the overlay related metadata into separate blocks and removes the
>> metadata from nodes and properties.  My current proposal leaves
>> the fragment subtrees intact - it only transforms __symbols__,
>> __fixups__, and __local_fixups__.
>>
>> Some Advantages and disadvantages of my proposal are:
>>
>> Con:
>>   - New blob version.
>>
>> Pro:
>>   - Backward compatible.  Bootloaders and kernels that can process v17 blobs
>>     will continue to work in the same manner with a v18 blob.  They will not
>>     be able to use the new v18 features.
>>
>> Pro:
>>   - If a bootloader passes a blob unmodified to a kernel, then the kernel will
>>     be able to use the new v18 features.
>>
>> Pro:
>>   - If a bootloader modifies a blob before passing it to a kernel _and_
>>     downrevs the version to v17, then the kernel will continue to work in the
>>     same manner as it works with a v17 blob.
>>
>> Con:
>>   - If a bootloader modifies a blob before passing it to a kernel _and_ fails
>>     to downrev the version to v17, then the kernel will most likely detect an
>>     error and may choose to not boot.
> 
> I have difficulty imagining any format change for which this is not
> the case.  Altering a blob and not downrevving it to the latest
> version you understand is definitely a bug.
> 
>> Pro:
>>   - A trailing magic field allows detection of a partially composed blob,
>>     where the blob is created by a multi-pass tool.
>>
>> Pro:
>>   - Validation tools can annotate the blob to indicate validation fail or
>>     validation warning.  The bootloader and kernel can choose what to do
>>     with the information.  For example, a Linux kernel might choose to
>>     taint itself if the blob is not validated or has a validation error.
>>
>> Pro:
>>   - A significant file size reduction for a blob that contains the symbol
>>     information needed by overlays.
> 
> I'm having trouble imagining how that can be while still being
> backward compatible with v17.

Mea culpa, a little bit.  The new format would not contain the __symbols__,
__fixups__, and __local_fixups__ nodes, so any overlay implementation that
is dependent upon those nodes would not be able to apply a new format
overlay FDT.

I expect that dtc will need to support the __symbols__, __fixups__, and
__local_fixups__ nodes, even if in deprecated form, for quite a while for
existing users.  If so, dtc would have to be able to place the overlay
metadata either in those nodes, or place the metadata in the new format.
I'm guessing this would maybe be based on a command line flag.

This will make more sense when I send the proposal.

-Frank


>> Pro:
>>   - A significant memory size reduction for a blob that contains the symbol
>>     information needed by overlays.
>>
>> Con:
>>   - All blobs will have a small file size increase when the symbol information
>>     needed by overlays is not included:  125 - 140 bytes or 44 bytes less
>>     than that if checksums not added.
>>
>> Pro:
>>   - Simplifies implementation of Linux kernel overlay application very early
>>     in the boot, mostly before unflattening the blob.  Early boot overlay
>>     application will be complete when the unflattenning is complete.
>>
>>     This feature _might_ _eventually_ reduce of remove the need for the
>>     bootloader to apply overlays.
>>
>> Pro:
>>   - Simplifies implementation of the Linux kernel overlay application to
>>     the live tree, post-boot.
>>
>>
>> Some data would be useful, so here it is.  I compiled all of the .dts files
>> in the Linux kernel source tree at arch/arm/boot/dts/.  The following data
>> is the size of the blob (and for Linux the amount of memory that the
>> flattened tree consumes and never frees after boot).  This is for v4.15-rc7.
>> The following sizes are in bytes.
>>
>> 'dtb no symbols' is the existing blob as currently built in the kernel tree.
>>
>> 'delta no_sym' is the extra overhead from convering 'dtb no symbols' to the
>> new format.  Again, this could be reduced by 44 bytes if the checksums
>> portion of the proposal is not implemented.
>>
>> 'delta symbols' is the existing format build in the kernel tree, but with
>> the "-@" flag provided to dtc, so that the __symbols__ node is populated.
>>
>> 'delta new fmt' is the 'delta symbols' blobs converted to the proposed
>> format.
>>
>> 'saved' is the number of bytes saved by converting to the new format,
>> when symbols metadata is included in the blob.
>>
>> The following table is sorted from most bytes saved, to least, with
>> one entry at approximately every 16 percentile:
>>
>>
>> row   dtb no   delta    delta    delta    saved
>>       symbols  no_sym   symbols  new fmt
>> ----  -------  -------  -------  -------  -------
>>  99%    90531      134    42721    15766    26955  am57xx-beagle-x15-revb1.dtb
>>  83%    44302      139    14582     5163     9419  imx6dl-tx6dl-comtft.dtb
>>  66%    26277      132    11662     4628     7034  sun6i-a31s-sinovoip-bpi-m2.dtb
>>  49%    21047      130     7328     2754     4574  imx53-qsb.dtb
>>  33%    12864      137     4305     1705     2600  kirkwood-netgear_readynas_nv+_v2.dtb
>>  16%    12009      128     2929     1520     1409  bcm911360k.dtb
>>   0%     1220      133       68      149      -81  xenvm-4.2.dtb
>>
>> Note that there are a few blobs with so little symbol metadata that the
>> proposed format increases the size of the blob.  This can be seen in the
>> tail of the distribution:
>>
>>    2537      128      593      384      209  mt6589-aquaris5.dtb
>>    2966      131      537      355      182  mt8127-moose.dtb
>>    5173      132      552      372      180  spear300-evb.dtb
>>    2473      128      485      368      117  mt6592-evb.dtb
>>    2370      135      454      343      111  cx92755_equinox.dtb
>>    2105      128      377      304       73  mt6580-evbp1.dtb
>>    2357      132      346      292       54  hip01-ca9x2.dtb
>>    1232      137      208      201        7  alphascale-asm9260-devkit.dtb
>>    2571      126      201      222      -21  alpine-db.dtb
>>    1220      133       68      149      -81  xenvm-4.2.dtb
>>
>> I will reply to this message with the complete table of all arm blobs.
>>
>> Please discuss.  :-)
>>
>> -Frank
> 

      reply	other threads:[~2018-01-23 21:28 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-01-22  8:09 Frank Rowand
     [not found] ` <b96829f9-2e8b-fdc5-5090-58591e2260cf-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-22  8:14   ` Frank Rowand
     [not found]     ` <ec68cfb3-4702-ba00-e926-3ad63b20b199-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-22 20:08       ` Frank Rowand
2018-01-22 20:08   ` Frank Rowand
     [not found]     ` <bc44f051-835d-1c8d-a928-be0fd4ef80b5-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-25  0:27       ` Frank Rowand
     [not found]         ` <b5a72db1-45b2-f21f-9afd-0991b288840e-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-27  8:48           ` David Gibson
2018-01-29  8:08             ` Frank Rowand
     [not found]               ` <2c352396-154e-ea75-c50b-0f2bfafd19da-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-29 10:56                 ` David Gibson
2018-01-30  1:29                   ` Frank Rowand
2018-01-22 21:01   ` Rob Herring
     [not found]     ` <CAL_JsqJR2y7bMw_-9TBAGWZ_kf7_sZo5qvqvRowJ8jiy=4G0Bg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-01-23 12:42       ` David Gibson
2018-01-23 21:17         ` Frank Rowand
     [not found]           ` <20328477-e511-e875-7dc4-253640f2219e-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-24 15:47             ` Rob Herring
     [not found]               ` <CAL_Jsq+fvFrGhqO0zbEUE_i23FkU=G4Z1-e0vnXHi4KbS2oK0A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-01-24 21:16                 ` Frank Rowand
     [not found]                   ` <e1bec77d-4dac-200b-34be-23573bf738f0-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-24 22:27                     ` Alan Tull
2018-01-25  0:22                     ` Frank Rowand
     [not found]                       ` <e986435b-481b-2629-7600-10d9e21ac58e-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-25 12:29                         ` David Gibson
2018-01-25 20:01                           ` Frank Rowand
     [not found]                             ` <72d30756-a963-92c9-1838-3e3f80c57e39-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-29 18:32                               ` Grant Likely
     [not found]                                 ` <CACxGe6tUB3NhNhOtqzoJfThx=RE1_=TSDQz+wQUm=sdE5JirZQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-01-29 23:15                                   ` David Gibson
2018-01-26  8:56                           ` Geert Uytterhoeven
     [not found]                             ` <CAMuHMdV0jdj90uzqMx_wtvz=-KaagJG2_UQTm1DW3gzt6cNG6A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-01-26  8:59                               ` Geert Uytterhoeven
2018-01-26 22:08                               ` Frank Rowand
     [not found]                                 ` <83008da0-7383-ba2d-a239-e11ad7d1327d-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-27  9:00                                   ` David Gibson
2018-01-27  8:56                               ` David Gibson
2018-01-25 23:11                     ` Frank Rowand
2018-01-25 12:22                 ` David Gibson
2018-01-25  9:14             ` Marek Vasut
     [not found]               ` <90983180-ae3b-5a31-9dc0-b62b978a0fba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-25 12:37                 ` David Gibson
2018-01-27 20:30                   ` Marek Vasut
     [not found]                     ` <5a943937-3b59-514b-3939-df25daea5470-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-01-29  0:53                       ` David Gibson
2018-01-23 12:05   ` David Gibson
2018-01-23 21:28     ` Frank Rowand [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1b85224d-0ee0-bf63-05da-aa4c99ff3342@gmail.com \
    --to=frowand.list-re5jqeeqqe8avxtiumwx3w@public.gmane.org \
    --cc=atull-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \
    --cc=david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org \
    --cc=devicetree-compiler-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=devicetree-spec-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=devicetree-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=geert-Td1EMuHUCqxL1ZNQvxDV9g@public.gmane.org \
    --cc=grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org \
    --cc=jdl-CYoMK+44s/E@public.gmane.org \
    --cc=kevans-HZy0K5TPuP5AfugRpC6u6w@public.gmane.org \
    --cc=marek.vasut-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org \
    --cc=mpe-Gsx/Oe8HsFggBc27wqDAHg@public.gmane.org \
    --cc=pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org \
    --cc=panto-wVdstyuyKrO8r51toPun2/C9HSW9iNxf@public.gmane.org \
    --cc=robh+dt-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org \
    --cc=trini-OWPKS81ov/FWk0Htik3J/w@public.gmane.org \
    --subject='Re: [RFC] devicetree: new FDT format version' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox