Re: [RFC] yamldt v0.5, now a DTS compiler too

[Pantelis Antoniou <pantelis.antoniou@konsulko.com>]

Hi Grant,

> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely@secretlab.ca> wrote:
> 
> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou
> <pantelis.antoniou@konsulko.com> wrote:
>> Hello again,
>> 
>> Significant progress has been made on yamldt and is now capable of
>> not only generating yaml from DTS source but also compiling DTS sources
>> and being almost fully compatible with DTC.
>> 
>> Compiling the kernel's DTBs using yamldt is as simple as using a
>> DTC=yamldt.
>> 
>> Error reporting is accurate and validation against a YAML based schema
>> works as well. In a short while I will begin posting patches with
>> fixes on bindings and DTS files in the kernel.
>> 
>> Please try it on your platform and report if you encounter any problems.
>> 
>> https://github.com/pantoniou/yamldt
>> 
>> I am eagerly awaiting for your comments.
> 
> Hi Pantelis,
> 
> This is good work. I've played around with it and I'm looking forward
> to chatting next week.
> 

Thanks. I’m looking forward to it too.

> One thing I've done is tried loading the output YAML files into
> another YAML interpreter and the current encoding causes problems.
> Specifically, in yamldt anchors/aliases are being used as a
> replacement for labels/phandles, but that conflicts with the YAML data
> model which defines a reference as a way to make a copy of the data
> appear in another part of the tree. For example, for the following
> snippit:
> 
> intc: intc@10000 {
>    #interrupt-cells = <1>;
>    compatible = "acme,intc";
>    reg = <0x10000 0x1000>;
>    gpio-controller;
> };
> 
> serial@20000 {
>    compatible = "acme,uart";
>    reg = <0x20000 0x1000>;
>    interrupt-parent = <&intc>;
>    interrupts = <5>;
> };
> 
> yamldt will encode this as:
> 
> intc@10000: &intc
>    "#interrupt-cells": 1
>    compatible: acme,intc
>    reg: [0x10000, 0x1000]
>    gpio-controller:
> 
> serial@20000:
>    compatible: acme,uart
>    reg: [0x20000, 0x1000]
>    interrupt-parent: *intc
>    interrupts: 5
> 
> But, the expected behaviour for a YAML parser is expand the alias
> '*intc' which results in the following structure:
> 
> intc@10000: &intc
>    "#interrupt-cells": 1
>    compatible: acme,intc
>    reg: [0x10000, 0x1000]
>    gpio-controller:
> 
> serial@20000:
>    compatible: acme,uart
>    reg: [0x20000, 0x1000]
>    interrupt-parent:
>        "#interrupt-cells": 1
>        compatible: acme,intc
>        reg: [0x10000, 0x1000]
>        gpio-controller:
>    interrupts: 5
> 
> See? It results in the entire interrupt controller node to appear as
> an instance under the interrupt-parent property, when the intention is
> only to create a phandle. yamldt should not redefine the behaviour of
> '*' aliases. Instead, it should use a different indicator, either
> using an explicit !phandle tag, or by replacing '*' with something
> else. I worked around it in my tests by replacing '*' with '$’.

Yes. This is expected. I don’t think pure YAML form is a good match for all
the crazy things that are possible (and actually used in practice) with DTS.

For instance there’s no way a normal YAML parser won’t choke with something like

/ {
  foo;
};

/ {
  bar;
};

Which is a common idiom in DTS files.

The decision to use the YAML anchors and references was more borne out of a desire
to sort of match conceptually the labels and references of DTS. It’s not a big
issue to switch to something different.  

If we were to force YAML/DT file to be regular YAML files parseable by everything
we’d have to rethink a lot of conventions that DT files currently use and I’m afraid
a machine translation from DTS to YAML might not be feasible then.

OTOH moving to that model might make it possible to use YAML constructs that are not
mapped at all to DTS. For instance

- foo: true
  bar: “string”

- foo: false
  bar: “another-string”

is not possible to be mapped to a DT node/property structure right now.

> 

> Plus, it would be useful to use normal YAML anchors/aliases for
> creating node templates. For example:
> 
> serial-template: &acme-uart . # The anchor for the template
>    compatible: acme,uart
>    interrupt-parent: *intc
> 
> root:
>    serial@20000:
>        <<: *acme-uart   # Alias node merged into serial@20000
>        interrupts: 5
>        reg: [0x20000, 0x1000]
>    serial@30000:
>        <<: *acme-uart   # Alias node merged into serial@30000
>        interrupts: 5
>        reg: [0x30000, 0x1000]
> 

Yes, I’m aware of this and I planned to talk to you about it :)
It can be a very powerful way to cut down the churn of DT right now.

It’s not going to be a problem for yamldt to support it (perhaps
adding a new editing tags to allow more fine grained control of the
substitions).
 
> Another problem with anchors/references is YAML seems to require the
> anchor to be defined before the reference, or at least that's what
> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding
> should be readily consumable by existing yaml implementations without
> having to do a lot of customization.
> 

I think this is only true for streamable YAML implementations; yamldt
can leave references unresolved until the emit phase (or even leave them
be as object files).

> I'm slightly concerned about using & anchors for labels because it
> seems only one anchor can be defined per node, but DTC allows multiple
> labels for a single node. This might not be an issue in practice
> though. Another implementation issue related to using & anchors is the
> YAML spec defines them as an encoding artifact, and parsers can
> discard the anchor names after parsing the YAML structure, which is a
> problem if we use something like $name to reference an anchor. The
> solution might just be that labels need to go into a special property
> so they don't disappear from the data stream.
> 

I do support multiple labels to the same node in yamldt with the
following trick, that uses an empty tree.

foo: bar: baz { frob; };

bar: &foo
  frob: true

bar: &bar
  ~: ~

Special properties should work, albeit being a little bit cludgy.

> There appears to be no place to put metadata. The root of the tree is
> the top level of the YAML structure. There isn't any provision for
> having a top level object to hold things like the memreserve map. We
> may need a namespace to use for special properties that aren't nodes
> or properties.
> 

I don’t think that’s going to be necessary. Metadata are artifacts that
only have meaning when emitting a serialized form like dtb.

yamldt fully supports /memreserve/ by simply treating a top level
/memreserve/ property as special and constructing the memreserve
entries at the emit phase.

so 

/memreserve/ 1000 10;

is simply:

/memreserve/: [ 1000 10 ]

It’s a matter of definition what properties are encoded as metadata for the
serialized form.

> The encoding differentiates between nodes and properties implicitly
> base on whether the contents are a map, or a scalar/list. This does
> mean any parser needs to do a bit more work and it may impact what can
> be done with validation (I'm not going to talk about validation in
> this email though. We'll talk next week.)

Hmm, I’m not sure I follow this. Is that related to the metadata problem.

BTW, a blog post explaining the rationale behind yamldt is going to come
up soon at our website, I’ll post the link when it does.

> 
> Cheers,
> g.

Regards

— Pantelis