* [RFC] yamldt v0.5, now a DTS compiler too @ 2017-09-28 19:58 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-09-28 19:58 UTC (permalink / raw) To: Frank Rowand Cc: Grant Likely, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hello again, Significant progress has been made on yamldt and is now capable of not only generating yaml from DTS source but also compiling DTS sources and being almost fully compatible with DTC. Compiling the kernel's DTBs using yamldt is as simple as using a DTC=yamldt. Error reporting is accurate and validation against a YAML based schema works as well. In a short while I will begin posting patches with fixes on bindings and DTS files in the kernel. Please try it on your platform and report if you encounter any problems. https://github.com/pantoniou/yamldt I am eagerly awaiting for your comments. Regards -- Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* [RFC] yamldt v0.5, now a DTS compiler too @ 2017-09-28 19:58 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-09-28 19:58 UTC (permalink / raw) To: Frank Rowand Cc: Grant Likely, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA Hello again, Significant progress has been made on yamldt and is now capable of not only generating yaml from DTS source but also compiling DTS sources and being almost fully compatible with DTC. Compiling the kernel's DTBs using yamldt is as simple as using a DTC=yamldt. Error reporting is accurate and validation against a YAML based schema works as well. In a short while I will begin posting patches with fixes on bindings and DTS files in the kernel. Please try it on your platform and report if you encounter any problems. https://github.com/pantoniou/yamldt I am eagerly awaiting for your comments. Regards -- Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-09-28 19:58 ` Pantelis Antoniou @ 2017-10-01 22:00 ` Rob Herring -1 siblings, 0 replies; 40+ messages in thread From: Rob Herring @ 2017-10-01 22:00 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > Hello again, > > Significant progress has been made on yamldt and is now capable of > not only generating yaml from DTS source but also compiling DTS sources > and being almost fully compatible with DTC. Can you quantify "almost"? > Compiling the kernel's DTBs using yamldt is as simple as using a > DTC=yamldt. Good. > > Error reporting is accurate and validation against a YAML based schema > works as well. In a short while I will begin posting patches with > fixes on bindings and DTS files in the kernel. What I would like to see is the schema format posted for review. I would also like to see the bindings for top-level compatible strings (aka boards) as an example. That's something that's simple enough that I'd think we could agree on a format and start moving towards defining board bindings that way. Rob ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-01 22:00 ` Rob Herring 0 siblings, 0 replies; 40+ messages in thread From: Rob Herring @ 2017-10-01 22:00 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > Hello again, > > Significant progress has been made on yamldt and is now capable of > not only generating yaml from DTS source but also compiling DTS sources > and being almost fully compatible with DTC. Can you quantify "almost"? > Compiling the kernel's DTBs using yamldt is as simple as using a > DTC=yamldt. Good. > > Error reporting is accurate and validation against a YAML based schema > works as well. In a short while I will begin posting patches with > fixes on bindings and DTS files in the kernel. What I would like to see is the schema format posted for review. I would also like to see the bindings for top-level compatible strings (aka boards) as an example. That's something that's simple enough that I'd think we could agree on a format and start moving towards defining board bindings that way. Rob ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-01 22:00 ` Rob Herring (?) @ 2017-10-02 7:36 ` Pantelis Antoniou -1 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-02 7:36 UTC (permalink / raw) To: Rob Herring Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel > On Oct 2, 2017, at 01:00 , Rob Herring <robherring2@gmail.com> wrote: > > On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: >> Hello again, >> >> Significant progress has been made on yamldt and is now capable of >> not only generating yaml from DTS source but also compiling DTS sources >> and being almost fully compatible with DTC. > > Can you quantify "almost”? The —sort option and /include-bin/ doesn’t work yet. It’s a day or so to add them but there are no kernel dtb files that rely on them. > >> Compiling the kernel's DTBs using yamldt is as simple as using a >> DTC=yamldt. > > Good. > >> >> Error reporting is accurate and validation against a YAML based schema >> works as well. In a short while I will begin posting patches with >> fixes on bindings and DTS files in the kernel. > > What I would like to see is the schema format posted for review. > It is based on your binding schema you’ve posted but tweaked slightly to be parseable by the YAML subset that matches the DTS style. In particular bare sequences do not map to DT at all, so: - name: foo is changed to name: foo And so on. > I would also like to see the bindings for top-level compatible strings > (aka boards) as an example. That's something that's simple enough that > I'd think we could agree on a format and start moving towards defining > board bindings that way. > I’ll see what I can do. > Rob Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-01 22:00 ` Rob Herring (?) (?) @ 2017-10-02 19:46 ` Pantelis Antoniou 2017-10-03 7:17 ` Geert Uytterhoeven 2017-10-03 13:18 ` Rob Herring -1 siblings, 2 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-02 19:46 UTC (permalink / raw) To: Rob Herring Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi Rob, On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: > On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: > > Hello again, > > > > Significant progress has been made on yamldt and is now capable of > > not only generating yaml from DTS source but also compiling DTS sources > > and being almost fully compatible with DTC. > > Can you quantify "almost"? > > > Compiling the kernel's DTBs using yamldt is as simple as using a > > DTC=yamldt. > > Good. > > > > > Error reporting is accurate and validation against a YAML based schema > > works as well. In a short while I will begin posting patches with > > fixes on bindings and DTS files in the kernel. > > What I would like to see is the schema format posted for review. > I'm including the skeleton.yaml binding which is the template for the bindings and a board-example.yaml binding for a top level binding. > I would also like to see the bindings for top-level compatible strings > (aka boards) as an example. That's something that's simple enough that > I'd think we could agree on a format and start moving towards defining > board bindings that way. > Note there is some line wrapping I'm including a link to the github repo of the files: The skeleton.yaml https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml %YAML 1.1 --- # The name of the binding is first # The anchor is put there for use by others skeleton: &skeleton version: 1 id: skel-device title: > Skeleton Device maintainer: name: Skeleton Person <skel@kernel.org> description: > The Skeleton Device binding represents the SK11 device produced by the Skeleton Corporation. The binding can also support compatible clones made by second source vendors. # The class is an optional property that declares this # binding as part of a larger set # Multiple definitions are possible class: [ device, spi-device ] # This binding inherits property characteristics from the generic # spi-slave binding # Note that the notation is standard yaml reference inherits: *spi-slave # virtual bindings do not generate checkers virtual: true # each property is defined by each name properties: # The compatible property is a reserved name. The type is always "string" # and should not be repeated device binding. compatible: category: required # required property type: strseq # is a sequence of strings description: > FX11 is a clone of the original SK11 device # v is always the name of the value of the property # np is passed to the checker and is the current # node pointer. We can access properties and call # methods that operate on them. # There can be multiple constraints, just put them # into a sequence. # Note that the BASE("skel,sk11") form from the previous # binding will have to be reworked. constraint: | anystreq(v, "skel,sk11") || anystreq(v, "faux,fx11") # The reg property is a reserved name. The type is always "int" and # should not be repeated in a device binding. Constraints are defined # only in the context of the parent node's address, size, and ranges # cells. The description is inherited from the spi-slave binding. # Note that if inheriting from a base binding this declaration may # be omitted. reg: category: required # required property type: intseq # is a sequence of integers # spi-max-frequency needs the device-specific constraint to be supplied spi-max-frequency: # this constraint is dependent on the compatible property # property containing "skel,sk11" constraint: | v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ? 10000000 : 1000000 # This property overrides the generic binding description with # a device specific description in order to mention the chip's # h/w cfg strapping pins. spi-cs-high: description: > Set if skeleton device configuration pins are set for chip select polarity high # Device specific properties don't inherit characteristic from a generic # binding so category, type, constraint, and description must be specified # if needed. skel,deprecated1: # note that the category may be declare more than one option category: [ deprecated, optional ] type: int constraint: | v >= 100000 && v <= 200000 description: > First of two deprecated properties. # There are no constraints for properties of empty type skel,deprecated2: category: deprecated type: empty description: > Second of two deprecated properties. # This example could be auto-generated rather than explicitly included # in the yaml source. # Note that the YAML example must be validated against this binding # to be an accepted entry example: dts: | sk11@0 { compatible = "skel,sk11"; reg = <0>; spi-max-frequency = <1000000>; spi-cs-high; }; yaml: | sk11@0: compatible: "skel,sk11" reg: 0 sip-max-frequency: 1000000 spi-cs-high: true --- ... And board-example.yaml https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml %YAML 1.1 --- board-example: &board-example version: 1 title: A board example using compatible and model properties maintainer: name: Skeleton Person <skel@kernel.org> class: board # this binding is selected when the compatible property constraint matches selected: "compatible" description: > A board binding example. Matches on a top-level compatible string and model. properties: compatible: category: required type: strseq description: | Compatible strings for the board example. The depth of the node must be zero, i.e. root. constraint: | get_depth(np) == 0 && ( anystreq(v, "example,evm") || anystreq(v, "example,evm2") || anystreq(v, "example,base")) model: category: required type: str description: models that this board family supports constraint: | streq(v, "Example EVM") || streq(v, "Example EVM2") example: dts: | / { compatible = "example,evm", "example,base"; model = "Example EVM"; }; yaml: | compatible: [ "example,evm", "example,base" ] ; model: "Example EVM"; As you see it's almost identical to what you've originally posted. > Rob ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-02 19:46 ` Pantelis Antoniou @ 2017-10-03 7:17 ` Geert Uytterhoeven 2017-10-03 7:33 ` Pantelis Antoniou 2017-10-03 13:18 ` Rob Herring 1 sibling, 1 reply; 40+ messages in thread From: Geert Uytterhoeven @ 2017-10-03 7:17 UTC (permalink / raw) To: Pantelis Antoniou Cc: Rob Herring, Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi Pantelis, On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > # Note that the YAML example must be validated against this binding > # to be an accepted entry Indeed ;-) > yaml: | > sk11@0: > compatible: "skel,sk11" > reg: 0 > sip-max-frequency: 1000000 s/sip-max-frequency/spi-max-frequency/ Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-03 7:17 ` Geert Uytterhoeven @ 2017-10-03 7:33 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-03 7:33 UTC (permalink / raw) To: Geert Uytterhoeven Cc: Rob Herring, Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi Geert, On Tue, 2017-10-03 at 09:17 +0200, Geert Uytterhoeven wrote: > Hi Pantelis, > > On Mon, Oct 2, 2017 at 9:46 PM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: > > # Note that the YAML example must be validated against this binding > > # to be an accepted entry > > Indeed ;-) > > > yaml: | > > sk11@0: > > compatible: "skel,sk11" > > reg: 0 > > sip-max-frequency: 1000000 > > s/sip-max-frequency/spi-max-frequency/ > Heh, obviously the auto-check of binding examples is not working yet :) > Gr{oetje,eeting}s, > > Geert > > -- > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org > > In personal conversations with technical people, I call myself a hacker. But > when I'm talking to journalists I just say "programmer" or something like that. > -- Linus Torvalds ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-02 19:46 ` Pantelis Antoniou 2017-10-03 7:17 ` Geert Uytterhoeven @ 2017-10-03 13:18 ` Rob Herring 2017-10-03 14:13 ` Pantelis Antoniou 1 sibling, 1 reply; 40+ messages in thread From: Rob Herring @ 2017-10-03 13:18 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > Hi Rob, > > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou >> <pantelis.antoniou@konsulko.com> wrote: >> > Hello again, >> > >> > Significant progress has been made on yamldt and is now capable of >> > not only generating yaml from DTS source but also compiling DTS sources >> > and being almost fully compatible with DTC. >> >> Can you quantify "almost"? >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a >> > DTC=yamldt. >> >> Good. >> >> > >> > Error reporting is accurate and validation against a YAML based schema >> > works as well. In a short while I will begin posting patches with >> > fixes on bindings and DTS files in the kernel. >> >> What I would like to see is the schema format posted for review. >> > > I'm including the skeleton.yaml binding which is the template for > the bindings and a board-example.yaml binding for a top level binding. > >> I would also like to see the bindings for top-level compatible strings >> (aka boards) as an example. That's something that's simple enough that >> I'd think we could agree on a format and start moving towards defining >> board bindings that way. >> > > Note there is some line wrapping I'm including a link > to the github repo of the files: > > > The skeleton.yaml > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml > > %YAML 1.1 > --- > # The name of the binding is first > # The anchor is put there for use by others > skeleton: &skeleton This and "id" seem redundant. > version: 1 > > id: skel-device > > title: > > Skeleton Device > > maintainer: > name: Skeleton Person <skel@kernel.org> > > description: > > The Skeleton Device binding represents the SK11 device produced by > the Skeleton Corporation. The binding can also support compatible > clones made by second source vendors. > > # The class is an optional property that declares this > # binding as part of a larger set > # Multiple definitions are possible > class: [ device, spi-device ] > > # This binding inherits property characteristics from the generic > # spi-slave binding > # Note that the notation is standard yaml reference > inherits: *spi-slave > > # virtual bindings do not generate checkers > virtual: true virtual is an overloaded term. > > # each property is defined by each name > properties: > > # The compatible property is a reserved name. The type is always > "string" > # and should not be repeated device binding. > compatible: > category: required # required property > type: strseq # is a sequence of strings > > description: > > FX11 is a clone of the original SK11 device > > # v is always the name of the value of the property > # np is passed to the checker and is the current > # node pointer. We can access properties and call > # methods that operate on them. > # There can be multiple constraints, just put them > # into a sequence. > # Note that the BASE("skel,sk11") form from the previous > # binding will have to be reworked. > constraint: | > anystreq(v, "skel,sk11") || > anystreq(v, "faux,fx11") Constraints and logic ops were certainly not decided in the last attempt and I think will be the hardest part to define. I see you are using eBPF in the checker. Is that where anystreq comes from? How would you express the ordering requirement (most significant compatible first)? > > # The reg property is a reserved name. The type is always "int" and > # should not be repeated in a device binding. Constraints are > defined > # only in the context of the parent node's address, size, and ranges > # cells. The description is inherited from the spi-slave binding. > # Note that if inheriting from a base binding this declaration may > # be omitted. > reg: > category: required # required property > type: intseq # is a sequence of integers > > # spi-max-frequency needs the device-specific constraint to be > supplied > spi-max-frequency: > # this constraint is dependent on the compatible property > # property containing "skel,sk11" > constraint: | > v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ? > 10000000 : 1000000 > > # This property overrides the generic binding description with > # a device specific description in order to mention the chip's > # h/w cfg strapping pins. > spi-cs-high: > description: > > Set if skeleton device configuration pins are set for chip > select polarity high > > # Device specific properties don't inherit characteristic from a > generic > # binding so category, type, constraint, and description must be > specified > # if needed. > skel,deprecated1: > # note that the category may be declare more than one option > category: [ deprecated, optional ] > type: int > constraint: | > v >= 100000 && v <= 200000 > description: > > First of two deprecated properties. > > # There are no constraints for properties of empty type > skel,deprecated2: > category: deprecated > type: empty > description: > > Second of two deprecated properties. > > # This example could be auto-generated rather than explicitly > included > # in the yaml source. > # Note that the YAML example must be validated against this binding > # to be an accepted entry > example: > > dts: | > sk11@0 { > compatible = "skel,sk11"; > reg = <0>; > spi-max-frequency = <1000000>; > spi-cs-high; > }; > > yaml: | > sk11@0: > compatible: "skel,sk11" > reg: 0 > sip-max-frequency: 1000000 > spi-cs-high: true > --- > ... > > And board-example.yaml > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml > > %YAML 1.1 > --- > board-example: &board-example > version: 1 > > title: A board example using compatible and model properties > > maintainer: > name: Skeleton Person <skel@kernel.org> > > class: board > > # this binding is selected when the compatible property constraint > matches > selected: "compatible" We need some way to express this must be the root node. More generally, we need to express what bindings must be a child of (think sub-devices in MFDs). Perhaps that's just a "parent" key with the value being the id/name. > > description: > > A board binding example. Matches on a top-level compatible string > and model. > > properties: > > compatible: > category: required > type: strseq > description: | > Compatible strings for the board example. > The depth of the node must be zero, i.e. root. > > constraint: | > get_depth(np) == 0 && ( Ahh, I guess this is how you are doing it. I don't think this works for any value other than 0. An MFD could be at any level. > anystreq(v, "example,evm") || > anystreq(v, "example,evm2") || > anystreq(v, "example,base")) > > model: > category: required > type: str > description: models that this board family supports > constraint: | > streq(v, "Example EVM") || > streq(v, "Example EVM2") > > example: > dts: | > / { > compatible = "example,evm", "example,base"; > model = "Example EVM"; > }; > yaml: | > compatible: [ "example,evm", "example,base" ] ; > model: "Example EVM"; I really don't want to see 2 examples. For now, it's going to be dts format. It could be converted by script later if needed. Overall, I think this format is a bit long for boards. We have something like ~1000 boards in arch/arm last I checked. I want adding a board binding to be very short and easy to review. It's often only a 1 line change. The main issue I have is it is just each SoC (or SoC family) does things their own way. > As you see it's almost identical to what you've originally posted. I barely remember and no one ever commented on it. Rob ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-03 14:13 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-03 14:13 UTC (permalink / raw) To: Rob Herring Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi Rob, On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: > On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: > > Hi Rob, > > > > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: > >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou > >> <pantelis.antoniou@konsulko.com> wrote: > >> > Hello again, > >> > > >> > Significant progress has been made on yamldt and is now capable of > >> > not only generating yaml from DTS source but also compiling DTS sources > >> > and being almost fully compatible with DTC. > >> > >> Can you quantify "almost"? > >> > >> > Compiling the kernel's DTBs using yamldt is as simple as using a > >> > DTC=yamldt. > >> > >> Good. > >> > >> > > >> > Error reporting is accurate and validation against a YAML based schema > >> > works as well. In a short while I will begin posting patches with > >> > fixes on bindings and DTS files in the kernel. > >> > >> What I would like to see is the schema format posted for review. > >> > > > > I'm including the skeleton.yaml binding which is the template for > > the bindings and a board-example.yaml binding for a top level binding. > > > >> I would also like to see the bindings for top-level compatible strings > >> (aka boards) as an example. That's something that's simple enough that > >> I'd think we could agree on a format and start moving towards defining > >> board bindings that way. > >> > > > > Note there is some line wrapping I'm including a link > > to the github repo of the files: > > > > > > The skeleton.yaml > > > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml > > > > %YAML 1.1 > > --- > > # The name of the binding is first > > # The anchor is put there for use by others > > skeleton: &skeleton > > This and "id" seem redundant. > Indeed. > > version: 1 > > > > id: skel-device > > > > title: > > > Skeleton Device > > > > maintainer: > > name: Skeleton Person <skel@kernel.org> > > > > description: > > > The Skeleton Device binding represents the SK11 device produced by > > the Skeleton Corporation. The binding can also support compatible > > clones made by second source vendors. > > > > # The class is an optional property that declares this > > # binding as part of a larger set > > # Multiple definitions are possible > > class: [ device, spi-device ] > > > > # This binding inherits property characteristics from the generic > > # spi-slave binding > > # Note that the notation is standard yaml reference > > inherits: *spi-slave > > > > # virtual bindings do not generate checkers > > virtual: true > > virtual is an overloaded term. > OK, what term should I use that this binding should not be instantiated as a checker, only be used by other bindings when inherited? > > > > # each property is defined by each name > > properties: > > > > # The compatible property is a reserved name. The type is always > > "string" > > # and should not be repeated device binding. > > compatible: > > category: required # required property > > type: strseq # is a sequence of strings > > > > description: > > > FX11 is a clone of the original SK11 device > > > > # v is always the name of the value of the property > > # np is passed to the checker and is the current > > # node pointer. We can access properties and call > > # methods that operate on them. > > # There can be multiple constraints, just put them > > # into a sequence. > > # Note that the BASE("skel,sk11") form from the previous > > # binding will have to be reworked. > > constraint: | > > anystreq(v, "skel,sk11") || > > anystreq(v, "faux,fx11") > > Constraints and logic ops were certainly not decided in the last > attempt and I think will be the hardest part to define. I see you are > using eBPF in the checker. Is that where anystreq comes from? > Yes. The ebpf environment declares a number of methods that are executed outside the ebpf sandbox. Check out https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c > How would you express the ordering requirement (most significant > compatible first)? > Err, there might be new bpf API needed there. For the first stab at the bindings problem I concentrated on getting things working and be as clear as possible. You could do something like that: orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL }) Which would check ordering too. Obviously you'd hide the weird syntax using a macro. #define ORDEREDSTRSEQ(_v, ...) \ orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL }) So you'd write the above as: ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11") > > > > # The reg property is a reserved name. The type is always "int" and > > # should not be repeated in a device binding. Constraints are > > defined > > # only in the context of the parent node's address, size, and ranges > > # cells. The description is inherited from the spi-slave binding. > > # Note that if inheriting from a base binding this declaration may > > # be omitted. > > reg: > > category: required # required property > > type: intseq # is a sequence of integers > > > > # spi-max-frequency needs the device-specific constraint to be > > supplied > > spi-max-frequency: > > # this constraint is dependent on the compatible property > > # property containing "skel,sk11" > > constraint: | > > v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ? > > 10000000 : 1000000 > > > > # This property overrides the generic binding description with > > # a device specific description in order to mention the chip's > > # h/w cfg strapping pins. > > spi-cs-high: > > description: > > > Set if skeleton device configuration pins are set for chip > > select polarity high > > > > # Device specific properties don't inherit characteristic from a > > generic > > # binding so category, type, constraint, and description must be > > specified > > # if needed. > > skel,deprecated1: > > # note that the category may be declare more than one option > > category: [ deprecated, optional ] > > type: int > > constraint: | > > v >= 100000 && v <= 200000 > > description: > > > First of two deprecated properties. > > > > # There are no constraints for properties of empty type > > skel,deprecated2: > > category: deprecated > > type: empty > > description: > > > Second of two deprecated properties. > > > > # This example could be auto-generated rather than explicitly > > included > > # in the yaml source. > > # Note that the YAML example must be validated against this binding > > # to be an accepted entry > > example: > > > > dts: | > > sk11@0 { > > compatible = "skel,sk11"; > > reg = <0>; > > spi-max-frequency = <1000000>; > > spi-cs-high; > > }; > > > > yaml: | > > sk11@0: > > compatible: "skel,sk11" > > reg: 0 > > sip-max-frequency: 1000000 > > spi-cs-high: true > > --- > > ... > > > > And board-example.yaml > > > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml > > > > %YAML 1.1 > > --- > > board-example: &board-example > > version: 1 > > > > title: A board example using compatible and model properties > > > > maintainer: > > name: Skeleton Person <skel@kernel.org> > > > > class: board > > > > # this binding is selected when the compatible property constraint > > matches > > selected: "compatible" > > We need some way to express this must be the root node. More > generally, we need to express what bindings must be a child of (think > sub-devices in MFDs). Perhaps that's just a "parent" key with the > value being the id/name. > > > > description: > > > A board binding example. Matches on a top-level compatible string > > and model. > > > > properties: > > > > compatible: > > category: required > > type: strseq > > description: | > > Compatible strings for the board example. > > The depth of the node must be zero, i.e. root. > > > > constraint: | > > get_depth(np) == 0 && ( > > Ahh, I guess this is how you are doing it. I don't think this works > for any value other than 0. An MFD could be at any level. > Well, I could've used a streq(get_name(get_parent(np)), "/") test but this is faster. It's up to you what would work best. > > anystreq(v, "example,evm") || > > anystreq(v, "example,evm2") || > > anystreq(v, "example,base")) > > > > model: > > category: required > > type: str > > description: models that this board family supports > > constraint: | > > streq(v, "Example EVM") || > > streq(v, "Example EVM2") > > > > example: > > dts: | > > / { > > compatible = "example,evm", "example,base"; > > model = "Example EVM"; > > }; > > yaml: | > > compatible: [ "example,evm", "example,base" ] ; > > model: "Example EVM"; > > I really don't want to see 2 examples. For now, it's going to be dts > format. It could be converted by script later if needed. > > Overall, I think this format is a bit long for boards. We have > something like ~1000 boards in arch/arm last I checked. I want adding > a board binding to be very short and easy to review. It's often only a > 1 line change. The main issue I have is it is just each SoC (or SoC > family) does things their own way. > Well, this is a full featured example; you could declare this a 'virtual' (or what ever you want to call it binding) and use: board-example-foo: inherits: *board-example properties: compatible: ... It is not absolutely terse, but it's still YAML. But for what is worth, those YAML files can be generated using the C preprocessor. You could define a macro that cuts the churn, albeit you lose the ability to parse them as normal YAML files. > > As you see it's almost identical to what you've originally posted. > > I barely remember and no one ever commented on it. > I mentioned that before, I wasn't CCed on it :) > Rob Regards -- Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-03 14:13 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-03 14:13 UTC (permalink / raw) To: Rob Herring Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA Hi Rob, On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: > On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou > <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > > Hi Rob, > > > > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: > >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou > >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > >> > Hello again, > >> > > >> > Significant progress has been made on yamldt and is now capable of > >> > not only generating yaml from DTS source but also compiling DTS sources > >> > and being almost fully compatible with DTC. > >> > >> Can you quantify "almost"? > >> > >> > Compiling the kernel's DTBs using yamldt is as simple as using a > >> > DTC=yamldt. > >> > >> Good. > >> > >> > > >> > Error reporting is accurate and validation against a YAML based schema > >> > works as well. In a short while I will begin posting patches with > >> > fixes on bindings and DTS files in the kernel. > >> > >> What I would like to see is the schema format posted for review. > >> > > > > I'm including the skeleton.yaml binding which is the template for > > the bindings and a board-example.yaml binding for a top level binding. > > > >> I would also like to see the bindings for top-level compatible strings > >> (aka boards) as an example. That's something that's simple enough that > >> I'd think we could agree on a format and start moving towards defining > >> board bindings that way. > >> > > > > Note there is some line wrapping I'm including a link > > to the github repo of the files: > > > > > > The skeleton.yaml > > > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml > > > > %YAML 1.1 > > --- > > # The name of the binding is first > > # The anchor is put there for use by others > > skeleton: &skeleton > > This and "id" seem redundant. > Indeed. > > version: 1 > > > > id: skel-device > > > > title: > > > Skeleton Device > > > > maintainer: > > name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> > > > > description: > > > The Skeleton Device binding represents the SK11 device produced by > > the Skeleton Corporation. The binding can also support compatible > > clones made by second source vendors. > > > > # The class is an optional property that declares this > > # binding as part of a larger set > > # Multiple definitions are possible > > class: [ device, spi-device ] > > > > # This binding inherits property characteristics from the generic > > # spi-slave binding > > # Note that the notation is standard yaml reference > > inherits: *spi-slave > > > > # virtual bindings do not generate checkers > > virtual: true > > virtual is an overloaded term. > OK, what term should I use that this binding should not be instantiated as a checker, only be used by other bindings when inherited? > > > > # each property is defined by each name > > properties: > > > > # The compatible property is a reserved name. The type is always > > "string" > > # and should not be repeated device binding. > > compatible: > > category: required # required property > > type: strseq # is a sequence of strings > > > > description: > > > FX11 is a clone of the original SK11 device > > > > # v is always the name of the value of the property > > # np is passed to the checker and is the current > > # node pointer. We can access properties and call > > # methods that operate on them. > > # There can be multiple constraints, just put them > > # into a sequence. > > # Note that the BASE("skel,sk11") form from the previous > > # binding will have to be reworked. > > constraint: | > > anystreq(v, "skel,sk11") || > > anystreq(v, "faux,fx11") > > Constraints and logic ops were certainly not decided in the last > attempt and I think will be the hardest part to define. I see you are > using eBPF in the checker. Is that where anystreq comes from? > Yes. The ebpf environment declares a number of methods that are executed outside the ebpf sandbox. Check out https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c > How would you express the ordering requirement (most significant > compatible first)? > Err, there might be new bpf API needed there. For the first stab at the bindings problem I concentrated on getting things working and be as clear as possible. You could do something like that: orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL }) Which would check ordering too. Obviously you'd hide the weird syntax using a macro. #define ORDEREDSTRSEQ(_v, ...) \ orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL }) So you'd write the above as: ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11") > > > > # The reg property is a reserved name. The type is always "int" and > > # should not be repeated in a device binding. Constraints are > > defined > > # only in the context of the parent node's address, size, and ranges > > # cells. The description is inherited from the spi-slave binding. > > # Note that if inheriting from a base binding this declaration may > > # be omitted. > > reg: > > category: required # required property > > type: intseq # is a sequence of integers > > > > # spi-max-frequency needs the device-specific constraint to be > > supplied > > spi-max-frequency: > > # this constraint is dependent on the compatible property > > # property containing "skel,sk11" > > constraint: | > > v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ? > > 10000000 : 1000000 > > > > # This property overrides the generic binding description with > > # a device specific description in order to mention the chip's > > # h/w cfg strapping pins. > > spi-cs-high: > > description: > > > Set if skeleton device configuration pins are set for chip > > select polarity high > > > > # Device specific properties don't inherit characteristic from a > > generic > > # binding so category, type, constraint, and description must be > > specified > > # if needed. > > skel,deprecated1: > > # note that the category may be declare more than one option > > category: [ deprecated, optional ] > > type: int > > constraint: | > > v >= 100000 && v <= 200000 > > description: > > > First of two deprecated properties. > > > > # There are no constraints for properties of empty type > > skel,deprecated2: > > category: deprecated > > type: empty > > description: > > > Second of two deprecated properties. > > > > # This example could be auto-generated rather than explicitly > > included > > # in the yaml source. > > # Note that the YAML example must be validated against this binding > > # to be an accepted entry > > example: > > > > dts: | > > sk11@0 { > > compatible = "skel,sk11"; > > reg = <0>; > > spi-max-frequency = <1000000>; > > spi-cs-high; > > }; > > > > yaml: | > > sk11@0: > > compatible: "skel,sk11" > > reg: 0 > > sip-max-frequency: 1000000 > > spi-cs-high: true > > --- > > ... > > > > And board-example.yaml > > > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml > > > > %YAML 1.1 > > --- > > board-example: &board-example > > version: 1 > > > > title: A board example using compatible and model properties > > > > maintainer: > > name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> > > > > class: board > > > > # this binding is selected when the compatible property constraint > > matches > > selected: "compatible" > > We need some way to express this must be the root node. More > generally, we need to express what bindings must be a child of (think > sub-devices in MFDs). Perhaps that's just a "parent" key with the > value being the id/name. > > > > description: > > > A board binding example. Matches on a top-level compatible string > > and model. > > > > properties: > > > > compatible: > > category: required > > type: strseq > > description: | > > Compatible strings for the board example. > > The depth of the node must be zero, i.e. root. > > > > constraint: | > > get_depth(np) == 0 && ( > > Ahh, I guess this is how you are doing it. I don't think this works > for any value other than 0. An MFD could be at any level. > Well, I could've used a streq(get_name(get_parent(np)), "/") test but this is faster. It's up to you what would work best. > > anystreq(v, "example,evm") || > > anystreq(v, "example,evm2") || > > anystreq(v, "example,base")) > > > > model: > > category: required > > type: str > > description: models that this board family supports > > constraint: | > > streq(v, "Example EVM") || > > streq(v, "Example EVM2") > > > > example: > > dts: | > > / { > > compatible = "example,evm", "example,base"; > > model = "Example EVM"; > > }; > > yaml: | > > compatible: [ "example,evm", "example,base" ] ; > > model: "Example EVM"; > > I really don't want to see 2 examples. For now, it's going to be dts > format. It could be converted by script later if needed. > > Overall, I think this format is a bit long for boards. We have > something like ~1000 boards in arch/arm last I checked. I want adding > a board binding to be very short and easy to review. It's often only a > 1 line change. The main issue I have is it is just each SoC (or SoC > family) does things their own way. > Well, this is a full featured example; you could declare this a 'virtual' (or what ever you want to call it binding) and use: board-example-foo: inherits: *board-example properties: compatible: ... It is not absolutely terse, but it's still YAML. But for what is worth, those YAML files can be generated using the C preprocessor. You could define a macro that cuts the churn, albeit you lose the ability to parse them as normal YAML files. > > As you see it's almost identical to what you've originally posted. > > I barely remember and no one ever commented on it. > I mentioned that before, I wasn't CCed on it :) > Rob Regards -- Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-03 14:13 ` Pantelis Antoniou (?) @ 2017-10-03 17:13 ` Rob Herring 2017-10-03 17:39 ` Pantelis Antoniou -1 siblings, 1 reply; 40+ messages in thread From: Rob Herring @ 2017-10-03 17:13 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > Hi Rob, > > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou >> <pantelis.antoniou@konsulko.com> wrote: >> > Hi Rob, >> > >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou >> >> <pantelis.antoniou@konsulko.com> wrote: >> >> > Hello again, >> >> > >> >> > Significant progress has been made on yamldt and is now capable of >> >> > not only generating yaml from DTS source but also compiling DTS sources >> >> > and being almost fully compatible with DTC. >> >> >> >> Can you quantify "almost"? >> >> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a >> >> > DTC=yamldt. >> >> >> >> Good. >> >> >> >> > >> >> > Error reporting is accurate and validation against a YAML based schema >> >> > works as well. In a short while I will begin posting patches with >> >> > fixes on bindings and DTS files in the kernel. >> >> >> >> What I would like to see is the schema format posted for review. >> >> >> > >> > I'm including the skeleton.yaml binding which is the template for >> > the bindings and a board-example.yaml binding for a top level binding. >> > >> >> I would also like to see the bindings for top-level compatible strings >> >> (aka boards) as an example. That's something that's simple enough that >> >> I'd think we could agree on a format and start moving towards defining >> >> board bindings that way. >> >> >> > >> > Note there is some line wrapping I'm including a link >> > to the github repo of the files: >> > >> > >> > The skeleton.yaml >> > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml >> > >> > %YAML 1.1 >> > --- >> > # The name of the binding is first >> > # The anchor is put there for use by others >> > skeleton: &skeleton >> >> This and "id" seem redundant. >> > > Indeed. One other thing, "skeleton" is a bit weird as a key. It can't be validated. All the other keys are standard words. I could write "skeloton" by mistake and I guess I'd only find the mistake when something inherits it. That's somewhat true with id, but we can at least check "id" is present and that it's value is unique among all other id values. > >> > version: 1 >> > >> > id: skel-device >> > >> > title: > >> > Skeleton Device >> > >> > maintainer: >> > name: Skeleton Person <skel@kernel.org> >> > >> > description: > >> > The Skeleton Device binding represents the SK11 device produced by >> > the Skeleton Corporation. The binding can also support compatible >> > clones made by second source vendors. >> > >> > # The class is an optional property that declares this >> > # binding as part of a larger set >> > # Multiple definitions are possible >> > class: [ device, spi-device ] >> > >> > # This binding inherits property characteristics from the generic >> > # spi-slave binding >> > # Note that the notation is standard yaml reference >> > inherits: *spi-slave >> > >> > # virtual bindings do not generate checkers >> > virtual: true >> >> virtual is an overloaded term. >> > > OK, what term should I use that this binding should not be instantiated > as a checker, only be used by other bindings when inherited? checks: true? I'd really like to avoid having to decide and drop this, but I don't really get why it is needed. > >> > >> > # each property is defined by each name >> > properties: >> > >> > # The compatible property is a reserved name. The type is always >> > "string" >> > # and should not be repeated device binding. >> > compatible: >> > category: required # required property >> > type: strseq # is a sequence of strings >> > >> > description: > >> > FX11 is a clone of the original SK11 device >> > >> > # v is always the name of the value of the property >> > # np is passed to the checker and is the current >> > # node pointer. We can access properties and call >> > # methods that operate on them. >> > # There can be multiple constraints, just put them >> > # into a sequence. >> > # Note that the BASE("skel,sk11") form from the previous >> > # binding will have to be reworked. >> > constraint: | >> > anystreq(v, "skel,sk11") || >> > anystreq(v, "faux,fx11") >> >> Constraints and logic ops were certainly not decided in the last >> attempt and I think will be the hardest part to define. I see you are >> using eBPF in the checker. Is that where anystreq comes from? >> > > Yes. The ebpf environment declares a number of methods that are executed > outside the ebpf sandbox. Check out > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml > https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c I looked at this some a while back. It wasn't clear to me what eBPF gives us and what you have defined on top of it. What I'm really after is documentation of the syntax and keywords here. >> How would you express the ordering requirement (most significant >> compatible first)? >> > > Err, there might be new bpf API needed there. For the first stab at > the bindings problem I concentrated on getting things working and be as > clear as possible. > > You could do something like that: > > orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL }) > > Which would check ordering too. > > Obviously you'd hide the weird syntax using a macro. > > #define ORDEREDSTRSEQ(_v, ...) \ > orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL }) > > > So you'd write the above as: > > ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11") > >> > >> > # The reg property is a reserved name. The type is always "int" and >> > # should not be repeated in a device binding. Constraints are >> > defined >> > # only in the context of the parent node's address, size, and ranges >> > # cells. The description is inherited from the spi-slave binding. >> > # Note that if inheriting from a base binding this declaration may >> > # be omitted. >> > reg: >> > category: required # required property >> > type: intseq # is a sequence of integers >> > >> > # spi-max-frequency needs the device-specific constraint to be >> > supplied >> > spi-max-frequency: >> > # this constraint is dependent on the compatible property >> > # property containing "skel,sk11" >> > constraint: | >> > v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ? >> > 10000000 : 1000000 >> > >> > # This property overrides the generic binding description with >> > # a device specific description in order to mention the chip's >> > # h/w cfg strapping pins. >> > spi-cs-high: >> > description: > >> > Set if skeleton device configuration pins are set for chip >> > select polarity high >> > >> > # Device specific properties don't inherit characteristic from a >> > generic >> > # binding so category, type, constraint, and description must be >> > specified >> > # if needed. >> > skel,deprecated1: >> > # note that the category may be declare more than one option >> > category: [ deprecated, optional ] >> > type: int >> > constraint: | >> > v >= 100000 && v <= 200000 >> > description: > >> > First of two deprecated properties. >> > >> > # There are no constraints for properties of empty type >> > skel,deprecated2: >> > category: deprecated >> > type: empty >> > description: > >> > Second of two deprecated properties. >> > >> > # This example could be auto-generated rather than explicitly >> > included >> > # in the yaml source. >> > # Note that the YAML example must be validated against this binding >> > # to be an accepted entry >> > example: >> > >> > dts: | >> > sk11@0 { >> > compatible = "skel,sk11"; >> > reg = <0>; >> > spi-max-frequency = <1000000>; >> > spi-cs-high; >> > }; >> > >> > yaml: | >> > sk11@0: >> > compatible: "skel,sk11" >> > reg: 0 >> > sip-max-frequency: 1000000 >> > spi-cs-high: true >> > --- >> > ... >> > >> > And board-example.yaml >> > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml >> > >> > %YAML 1.1 >> > --- >> > board-example: &board-example >> > version: 1 >> > >> > title: A board example using compatible and model properties >> > >> > maintainer: >> > name: Skeleton Person <skel@kernel.org> >> > >> > class: board >> > >> > # this binding is selected when the compatible property constraint >> > matches >> > selected: "compatible" >> >> We need some way to express this must be the root node. More >> generally, we need to express what bindings must be a child of (think >> sub-devices in MFDs). Perhaps that's just a "parent" key with the >> value being the id/name. > >> > >> > description: > >> > A board binding example. Matches on a top-level compatible string >> > and model. >> > >> > properties: >> > >> > compatible: >> > category: required >> > type: strseq >> > description: | >> > Compatible strings for the board example. >> > The depth of the node must be zero, i.e. root. >> > >> > constraint: | >> > get_depth(np) == 0 && ( >> >> Ahh, I guess this is how you are doing it. I don't think this works >> for any value other than 0. An MFD could be at any level. >> > > Well, I could've used a streq(get_name(get_parent(np)), "/") test but > this is faster. It's up to you what would work best. Why not just a "parent" key? >> > anystreq(v, "example,evm") || >> > anystreq(v, "example,evm2") || >> > anystreq(v, "example,base")) >> > >> > model: >> > category: required >> > type: str >> > description: models that this board family supports >> > constraint: | >> > streq(v, "Example EVM") || >> > streq(v, "Example EVM2") >> > >> > example: >> > dts: | >> > / { >> > compatible = "example,evm", "example,base"; >> > model = "Example EVM"; >> > }; >> > yaml: | >> > compatible: [ "example,evm", "example,base" ] ; >> > model: "Example EVM"; >> >> I really don't want to see 2 examples. For now, it's going to be dts >> format. It could be converted by script later if needed. >> >> Overall, I think this format is a bit long for boards. We have >> something like ~1000 boards in arch/arm last I checked. I want adding >> a board binding to be very short and easy to review. It's often only a >> 1 line change. The main issue I have is it is just each SoC (or SoC >> family) does things their own way. >> > > Well, this is a full featured example; you could declare this a > 'virtual' (or what ever you want to call it binding) and use: > > board-example-foo: > > inherits: *board-example > > properties: > compatible: ... > > It is not absolutely terse, but it's still YAML. But for what is worth, > those YAML files can be generated using the C preprocessor. You could > define a macro that cuts the churn, albeit you lose the ability to > parse them as normal YAML files. C preprocessor doesn't sound like a great option to me. Perhaps it is by SoC compatible and boards are just an addition to the constraints. That's kind of how things are organized already. Rob ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-03 17:39 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-03 17:39 UTC (permalink / raw) To: Rob Herring Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi Rob, On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote: > On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: > > Hi Rob, > > > > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: > >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou > >> <pantelis.antoniou@konsulko.com> wrote: > >> > Hi Rob, > >> > > >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: > >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou > >> >> <pantelis.antoniou@konsulko.com> wrote: > >> >> > Hello again, > >> >> > > >> >> > Significant progress has been made on yamldt and is now capable of > >> >> > not only generating yaml from DTS source but also compiling DTS sources > >> >> > and being almost fully compatible with DTC. > >> >> > >> >> Can you quantify "almost"? > >> >> > >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a > >> >> > DTC=yamldt. > >> >> > >> >> Good. > >> >> > >> >> > > >> >> > Error reporting is accurate and validation against a YAML based schema > >> >> > works as well. In a short while I will begin posting patches with > >> >> > fixes on bindings and DTS files in the kernel. > >> >> > >> >> What I would like to see is the schema format posted for review. > >> >> > >> > > >> > I'm including the skeleton.yaml binding which is the template for > >> > the bindings and a board-example.yaml binding for a top level binding. > >> > > >> >> I would also like to see the bindings for top-level compatible strings > >> >> (aka boards) as an example. That's something that's simple enough that > >> >> I'd think we could agree on a format and start moving towards defining > >> >> board bindings that way. > >> >> > >> > > >> > Note there is some line wrapping I'm including a link > >> > to the github repo of the files: > >> > > >> > > >> > The skeleton.yaml > >> > > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml > >> > > >> > %YAML 1.1 > >> > --- > >> > # The name of the binding is first > >> > # The anchor is put there for use by others > >> > skeleton: &skeleton > >> > >> This and "id" seem redundant. > >> > > > > Indeed. > > One other thing, "skeleton" is a bit weird as a key. It can't be > validated. All the other keys are standard words. I could write > "skeloton" by mistake and I guess I'd only find the mistake when > something inherits it. That's somewhat true with id, but we can at > least check "id" is present and that it's value is unique among all > other id values. > We can keep id and check that it matches the name of the enclosing node. That way you can be sure that there's no error. But it's a bit weird cause this is similar to declaring a function name with a typo. You won't find out until you use it. > > > >> > version: 1 > >> > > >> > id: skel-device > >> > > >> > title: > > >> > Skeleton Device > >> > > >> > maintainer: > >> > name: Skeleton Person <skel@kernel.org> > >> > > >> > description: > > >> > The Skeleton Device binding represents the SK11 device produced by > >> > the Skeleton Corporation. The binding can also support compatible > >> > clones made by second source vendors. > >> > > >> > # The class is an optional property that declares this > >> > # binding as part of a larger set > >> > # Multiple definitions are possible > >> > class: [ device, spi-device ] > >> > > >> > # This binding inherits property characteristics from the generic > >> > # spi-slave binding > >> > # Note that the notation is standard yaml reference > >> > inherits: *spi-slave > >> > > >> > # virtual bindings do not generate checkers > >> > virtual: true > >> > >> virtual is an overloaded term. > >> > > > > OK, what term should I use that this binding should not be instantiated > > as a checker, only be used by other bindings when inherited? > > checks: true? > > I'd really like to avoid having to decide and drop this, but I don't > really get why it is needed. > It is needed because otherwise checker filters will be generated for the template bindings that while they won't be executed they will be compiled and take up space in the schema. > > > >> > > >> > # each property is defined by each name > >> > properties: > >> > > >> > # The compatible property is a reserved name. The type is always > >> > "string" > >> > # and should not be repeated device binding. > >> > compatible: > >> > category: required # required property > >> > type: strseq # is a sequence of strings > >> > > >> > description: > > >> > FX11 is a clone of the original SK11 device > >> > > >> > # v is always the name of the value of the property > >> > # np is passed to the checker and is the current > >> > # node pointer. We can access properties and call > >> > # methods that operate on them. > >> > # There can be multiple constraints, just put them > >> > # into a sequence. > >> > # Note that the BASE("skel,sk11") form from the previous > >> > # binding will have to be reworked. > >> > constraint: | > >> > anystreq(v, "skel,sk11") || > >> > anystreq(v, "faux,fx11") > >> > >> Constraints and logic ops were certainly not decided in the last > >> attempt and I think will be the hardest part to define. I see you are > >> using eBPF in the checker. Is that where anystreq comes from? > >> > > > > Yes. The ebpf environment declares a number of methods that are executed > > outside the ebpf sandbox. Check out > > > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml > > https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c > > I looked at this some a while back. It wasn't clear to me what eBPF > gives us and what you have defined on top of it. What I'm really after > is documentation of the syntax and keywords here. > eBPF is portable, can be serialized after compiling in the schema file and can be executed in the kernel. By stripping out all documentation related properties and nodes keeping only the compiled filters you can generate a dtb blob that passed to kernel can be used for verification of all runtime changes in the kernel's live tree. eBPF is enforcing an execution model that is 'safe' so we can be sure that no foul play is possible. That means that you can a) run it at boot-time, verifying the dtb blob passed by the bootloader for errors (potentially disabling devices that their nodes fail) and b) run it when applying overlays to reject any that result in an invalid tree. One other use that I'm thinking that might be possible too, like binding version check between what the driver is compiled against and what the passed blob is without using an explicit version property. The constraint format ofcourse is plain C, you can generate a .so using the host compiler and execute that if you are looking for performance. Syntax and keywords of the internals is in a bit of flux right now, since the binding format is not finalized yet. I'd be happy to discuss the internal at ELCE. > >> How would you express the ordering requirement (most significant > >> compatible first)? > >> > > > > Err, there might be new bpf API needed there. For the first stab at > > the bindings problem I concentrated on getting things working and be as > > clear as possible. > > > > You could do something like that: > > > > orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL }) > > > > Which would check ordering too. > > > > Obviously you'd hide the weird syntax using a macro. > > > > #define ORDEREDSTRSEQ(_v, ...) \ > > orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL }) > > > > > > So you'd write the above as: > > > > ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11") > > > >> > > >> > # The reg property is a reserved name. The type is always "int" and > >> > # should not be repeated in a device binding. Constraints are > >> > defined > >> > # only in the context of the parent node's address, size, and ranges > >> > # cells. The description is inherited from the spi-slave binding. > >> > # Note that if inheriting from a base binding this declaration may > >> > # be omitted. > >> > reg: > >> > category: required # required property > >> > type: intseq # is a sequence of integers > >> > > >> > # spi-max-frequency needs the device-specific constraint to be > >> > supplied > >> > spi-max-frequency: > >> > # this constraint is dependent on the compatible property > >> > # property containing "skel,sk11" > >> > constraint: | > >> > v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ? > >> > 10000000 : 1000000 > >> > > >> > # This property overrides the generic binding description with > >> > # a device specific description in order to mention the chip's > >> > # h/w cfg strapping pins. > >> > spi-cs-high: > >> > description: > > >> > Set if skeleton device configuration pins are set for chip > >> > select polarity high > >> > > >> > # Device specific properties don't inherit characteristic from a > >> > generic > >> > # binding so category, type, constraint, and description must be > >> > specified > >> > # if needed. > >> > skel,deprecated1: > >> > # note that the category may be declare more than one option > >> > category: [ deprecated, optional ] > >> > type: int > >> > constraint: | > >> > v >= 100000 && v <= 200000 > >> > description: > > >> > First of two deprecated properties. > >> > > >> > # There are no constraints for properties of empty type > >> > skel,deprecated2: > >> > category: deprecated > >> > type: empty > >> > description: > > >> > Second of two deprecated properties. > >> > > >> > # This example could be auto-generated rather than explicitly > >> > included > >> > # in the yaml source. > >> > # Note that the YAML example must be validated against this binding > >> > # to be an accepted entry > >> > example: > >> > > >> > dts: | > >> > sk11@0 { > >> > compatible = "skel,sk11"; > >> > reg = <0>; > >> > spi-max-frequency = <1000000>; > >> > spi-cs-high; > >> > }; > >> > > >> > yaml: | > >> > sk11@0: > >> > compatible: "skel,sk11" > >> > reg: 0 > >> > sip-max-frequency: 1000000 > >> > spi-cs-high: true > >> > --- > >> > ... > >> > > >> > And board-example.yaml > >> > > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml > >> > > >> > %YAML 1.1 > >> > --- > >> > board-example: &board-example > >> > version: 1 > >> > > >> > title: A board example using compatible and model properties > >> > > >> > maintainer: > >> > name: Skeleton Person <skel@kernel.org> > >> > > >> > class: board > >> > > >> > # this binding is selected when the compatible property constraint > >> > matches > >> > selected: "compatible" > >> > >> We need some way to express this must be the root node. More > >> generally, we need to express what bindings must be a child of (think > >> sub-devices in MFDs). Perhaps that's just a "parent" key with the > >> value being the id/name. > > > >> > > >> > description: > > >> > A board binding example. Matches on a top-level compatible string > >> > and model. > >> > > >> > properties: > >> > > >> > compatible: > >> > category: required > >> > type: strseq > >> > description: | > >> > Compatible strings for the board example. > >> > The depth of the node must be zero, i.e. root. > >> > > >> > constraint: | > >> > get_depth(np) == 0 && ( > >> > >> Ahh, I guess this is how you are doing it. I don't think this works > >> for any value other than 0. An MFD could be at any level. > >> > > > > Well, I could've used a streq(get_name(get_parent(np)), "/") test but > > this is faster. It's up to you what would work best. > > Why not just a "parent" key? > Because the property/type prolog code would increase even for properties that don't need the parent key. While generating the parent key only for constraints that need it keeps the filter size smaller. Of course if parent ends up being used for most constraints the balancing act changes. > >> > anystreq(v, "example,evm") || > >> > anystreq(v, "example,evm2") || > >> > anystreq(v, "example,base")) > >> > > >> > model: > >> > category: required > >> > type: str > >> > description: models that this board family supports > >> > constraint: | > >> > streq(v, "Example EVM") || > >> > streq(v, "Example EVM2") > >> > > >> > example: > >> > dts: | > >> > / { > >> > compatible = "example,evm", "example,base"; > >> > model = "Example EVM"; > >> > }; > >> > yaml: | > >> > compatible: [ "example,evm", "example,base" ] ; > >> > model: "Example EVM"; > >> > >> I really don't want to see 2 examples. For now, it's going to be dts > >> format. It could be converted by script later if needed. > >> > >> Overall, I think this format is a bit long for boards. We have > >> something like ~1000 boards in arch/arm last I checked. I want adding > >> a board binding to be very short and easy to review. It's often only a > >> 1 line change. The main issue I have is it is just each SoC (or SoC > >> family) does things their own way. > >> > > > > Well, this is a full featured example; you could declare this a > > 'virtual' (or what ever you want to call it binding) and use: > > > > board-example-foo: > > > > inherits: *board-example > > > > properties: > > compatible: ... > > > > It is not absolutely terse, but it's still YAML. But for what is worth, > > those YAML files can be generated using the C preprocessor. You could > > define a macro that cuts the churn, albeit you lose the ability to > > parse them as normal YAML files. > > C preprocessor doesn't sound like a great option to me. > > Perhaps it is by SoC compatible and boards are just an addition to the > constraints. That's kind of how things are organized already. > The code base supports multiple constraints so we could have multiple constraint properties. How would you like your SoC example to work ideally? > Rob Regards -- Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-03 17:39 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-03 17:39 UTC (permalink / raw) To: Rob Herring Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA Hi Rob, On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote: > On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou > <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > > Hi Rob, > > > > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: > >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou > >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > >> > Hi Rob, > >> > > >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: > >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou > >> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > >> >> > Hello again, > >> >> > > >> >> > Significant progress has been made on yamldt and is now capable of > >> >> > not only generating yaml from DTS source but also compiling DTS sources > >> >> > and being almost fully compatible with DTC. > >> >> > >> >> Can you quantify "almost"? > >> >> > >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a > >> >> > DTC=yamldt. > >> >> > >> >> Good. > >> >> > >> >> > > >> >> > Error reporting is accurate and validation against a YAML based schema > >> >> > works as well. In a short while I will begin posting patches with > >> >> > fixes on bindings and DTS files in the kernel. > >> >> > >> >> What I would like to see is the schema format posted for review. > >> >> > >> > > >> > I'm including the skeleton.yaml binding which is the template for > >> > the bindings and a board-example.yaml binding for a top level binding. > >> > > >> >> I would also like to see the bindings for top-level compatible strings > >> >> (aka boards) as an example. That's something that's simple enough that > >> >> I'd think we could agree on a format and start moving towards defining > >> >> board bindings that way. > >> >> > >> > > >> > Note there is some line wrapping I'm including a link > >> > to the github repo of the files: > >> > > >> > > >> > The skeleton.yaml > >> > > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml > >> > > >> > %YAML 1.1 > >> > --- > >> > # The name of the binding is first > >> > # The anchor is put there for use by others > >> > skeleton: &skeleton > >> > >> This and "id" seem redundant. > >> > > > > Indeed. > > One other thing, "skeleton" is a bit weird as a key. It can't be > validated. All the other keys are standard words. I could write > "skeloton" by mistake and I guess I'd only find the mistake when > something inherits it. That's somewhat true with id, but we can at > least check "id" is present and that it's value is unique among all > other id values. > We can keep id and check that it matches the name of the enclosing node. That way you can be sure that there's no error. But it's a bit weird cause this is similar to declaring a function name with a typo. You won't find out until you use it. > > > >> > version: 1 > >> > > >> > id: skel-device > >> > > >> > title: > > >> > Skeleton Device > >> > > >> > maintainer: > >> > name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> > >> > > >> > description: > > >> > The Skeleton Device binding represents the SK11 device produced by > >> > the Skeleton Corporation. The binding can also support compatible > >> > clones made by second source vendors. > >> > > >> > # The class is an optional property that declares this > >> > # binding as part of a larger set > >> > # Multiple definitions are possible > >> > class: [ device, spi-device ] > >> > > >> > # This binding inherits property characteristics from the generic > >> > # spi-slave binding > >> > # Note that the notation is standard yaml reference > >> > inherits: *spi-slave > >> > > >> > # virtual bindings do not generate checkers > >> > virtual: true > >> > >> virtual is an overloaded term. > >> > > > > OK, what term should I use that this binding should not be instantiated > > as a checker, only be used by other bindings when inherited? > > checks: true? > > I'd really like to avoid having to decide and drop this, but I don't > really get why it is needed. > It is needed because otherwise checker filters will be generated for the template bindings that while they won't be executed they will be compiled and take up space in the schema. > > > >> > > >> > # each property is defined by each name > >> > properties: > >> > > >> > # The compatible property is a reserved name. The type is always > >> > "string" > >> > # and should not be repeated device binding. > >> > compatible: > >> > category: required # required property > >> > type: strseq # is a sequence of strings > >> > > >> > description: > > >> > FX11 is a clone of the original SK11 device > >> > > >> > # v is always the name of the value of the property > >> > # np is passed to the checker and is the current > >> > # node pointer. We can access properties and call > >> > # methods that operate on them. > >> > # There can be multiple constraints, just put them > >> > # into a sequence. > >> > # Note that the BASE("skel,sk11") form from the previous > >> > # binding will have to be reworked. > >> > constraint: | > >> > anystreq(v, "skel,sk11") || > >> > anystreq(v, "faux,fx11") > >> > >> Constraints and logic ops were certainly not decided in the last > >> attempt and I think will be the hardest part to define. I see you are > >> using eBPF in the checker. Is that where anystreq comes from? > >> > > > > Yes. The ebpf environment declares a number of methods that are executed > > outside the ebpf sandbox. Check out > > > > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml > > https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c > > I looked at this some a while back. It wasn't clear to me what eBPF > gives us and what you have defined on top of it. What I'm really after > is documentation of the syntax and keywords here. > eBPF is portable, can be serialized after compiling in the schema file and can be executed in the kernel. By stripping out all documentation related properties and nodes keeping only the compiled filters you can generate a dtb blob that passed to kernel can be used for verification of all runtime changes in the kernel's live tree. eBPF is enforcing an execution model that is 'safe' so we can be sure that no foul play is possible. That means that you can a) run it at boot-time, verifying the dtb blob passed by the bootloader for errors (potentially disabling devices that their nodes fail) and b) run it when applying overlays to reject any that result in an invalid tree. One other use that I'm thinking that might be possible too, like binding version check between what the driver is compiled against and what the passed blob is without using an explicit version property. The constraint format ofcourse is plain C, you can generate a .so using the host compiler and execute that if you are looking for performance. Syntax and keywords of the internals is in a bit of flux right now, since the binding format is not finalized yet. I'd be happy to discuss the internal at ELCE. > >> How would you express the ordering requirement (most significant > >> compatible first)? > >> > > > > Err, there might be new bpf API needed there. For the first stab at > > the bindings problem I concentrated on getting things working and be as > > clear as possible. > > > > You could do something like that: > > > > orderedstrseq(v, (const char *[]){ "skel,sk11", "faux,fx11", NULL }) > > > > Which would check ordering too. > > > > Obviously you'd hide the weird syntax using a macro. > > > > #define ORDEREDSTRSEQ(_v, ...) \ > > orderedstrseq(_v, (const char *[]){ __VA_ARGS__ , NULL }) > > > > > > So you'd write the above as: > > > > ORDEREDSTRSEQ(v, "skel,sk11", "faux,fx11") > > > >> > > >> > # The reg property is a reserved name. The type is always "int" and > >> > # should not be repeated in a device binding. Constraints are > >> > defined > >> > # only in the context of the parent node's address, size, and ranges > >> > # cells. The description is inherited from the spi-slave binding. > >> > # Note that if inheriting from a base binding this declaration may > >> > # be omitted. > >> > reg: > >> > category: required # required property > >> > type: intseq # is a sequence of integers > >> > > >> > # spi-max-frequency needs the device-specific constraint to be > >> > supplied > >> > spi-max-frequency: > >> > # this constraint is dependent on the compatible property > >> > # property containing "skel,sk11" > >> > constraint: | > >> > v <= anystreq(get_strseq(np, "compatible"), "skel,sk11") ? > >> > 10000000 : 1000000 > >> > > >> > # This property overrides the generic binding description with > >> > # a device specific description in order to mention the chip's > >> > # h/w cfg strapping pins. > >> > spi-cs-high: > >> > description: > > >> > Set if skeleton device configuration pins are set for chip > >> > select polarity high > >> > > >> > # Device specific properties don't inherit characteristic from a > >> > generic > >> > # binding so category, type, constraint, and description must be > >> > specified > >> > # if needed. > >> > skel,deprecated1: > >> > # note that the category may be declare more than one option > >> > category: [ deprecated, optional ] > >> > type: int > >> > constraint: | > >> > v >= 100000 && v <= 200000 > >> > description: > > >> > First of two deprecated properties. > >> > > >> > # There are no constraints for properties of empty type > >> > skel,deprecated2: > >> > category: deprecated > >> > type: empty > >> > description: > > >> > Second of two deprecated properties. > >> > > >> > # This example could be auto-generated rather than explicitly > >> > included > >> > # in the yaml source. > >> > # Note that the YAML example must be validated against this binding > >> > # to be an accepted entry > >> > example: > >> > > >> > dts: | > >> > sk11@0 { > >> > compatible = "skel,sk11"; > >> > reg = <0>; > >> > spi-max-frequency = <1000000>; > >> > spi-cs-high; > >> > }; > >> > > >> > yaml: | > >> > sk11@0: > >> > compatible: "skel,sk11" > >> > reg: 0 > >> > sip-max-frequency: 1000000 > >> > spi-cs-high: true > >> > --- > >> > ... > >> > > >> > And board-example.yaml > >> > > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/board-example.yaml > >> > > >> > %YAML 1.1 > >> > --- > >> > board-example: &board-example > >> > version: 1 > >> > > >> > title: A board example using compatible and model properties > >> > > >> > maintainer: > >> > name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> > >> > > >> > class: board > >> > > >> > # this binding is selected when the compatible property constraint > >> > matches > >> > selected: "compatible" > >> > >> We need some way to express this must be the root node. More > >> generally, we need to express what bindings must be a child of (think > >> sub-devices in MFDs). Perhaps that's just a "parent" key with the > >> value being the id/name. > > > >> > > >> > description: > > >> > A board binding example. Matches on a top-level compatible string > >> > and model. > >> > > >> > properties: > >> > > >> > compatible: > >> > category: required > >> > type: strseq > >> > description: | > >> > Compatible strings for the board example. > >> > The depth of the node must be zero, i.e. root. > >> > > >> > constraint: | > >> > get_depth(np) == 0 && ( > >> > >> Ahh, I guess this is how you are doing it. I don't think this works > >> for any value other than 0. An MFD could be at any level. > >> > > > > Well, I could've used a streq(get_name(get_parent(np)), "/") test but > > this is faster. It's up to you what would work best. > > Why not just a "parent" key? > Because the property/type prolog code would increase even for properties that don't need the parent key. While generating the parent key only for constraints that need it keeps the filter size smaller. Of course if parent ends up being used for most constraints the balancing act changes. > >> > anystreq(v, "example,evm") || > >> > anystreq(v, "example,evm2") || > >> > anystreq(v, "example,base")) > >> > > >> > model: > >> > category: required > >> > type: str > >> > description: models that this board family supports > >> > constraint: | > >> > streq(v, "Example EVM") || > >> > streq(v, "Example EVM2") > >> > > >> > example: > >> > dts: | > >> > / { > >> > compatible = "example,evm", "example,base"; > >> > model = "Example EVM"; > >> > }; > >> > yaml: | > >> > compatible: [ "example,evm", "example,base" ] ; > >> > model: "Example EVM"; > >> > >> I really don't want to see 2 examples. For now, it's going to be dts > >> format. It could be converted by script later if needed. > >> > >> Overall, I think this format is a bit long for boards. We have > >> something like ~1000 boards in arch/arm last I checked. I want adding > >> a board binding to be very short and easy to review. It's often only a > >> 1 line change. The main issue I have is it is just each SoC (or SoC > >> family) does things their own way. > >> > > > > Well, this is a full featured example; you could declare this a > > 'virtual' (or what ever you want to call it binding) and use: > > > > board-example-foo: > > > > inherits: *board-example > > > > properties: > > compatible: ... > > > > It is not absolutely terse, but it's still YAML. But for what is worth, > > those YAML files can be generated using the C preprocessor. You could > > define a macro that cuts the churn, albeit you lose the ability to > > parse them as normal YAML files. > > C preprocessor doesn't sound like a great option to me. > > Perhaps it is by SoC compatible and boards are just an addition to the > constraints. That's kind of how things are organized already. > The code base supports multiple constraints so we could have multiple constraint properties. How would you like your SoC example to work ideally? > Rob Regards -- Pantelis -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-03 17:39 ` Pantelis Antoniou @ 2017-10-06 13:55 ` Rob Herring -1 siblings, 0 replies; 40+ messages in thread From: Rob Herring @ 2017-10-06 13:55 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > Hi Rob, > > On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote: >> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou >> <pantelis.antoniou@konsulko.com> wrote: >> > Hi Rob, >> > >> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: >> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou >> >> <pantelis.antoniou@konsulko.com> wrote: >> >> > Hi Rob, >> >> > >> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: >> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou >> >> >> <pantelis.antoniou@konsulko.com> wrote: >> >> >> > Hello again, >> >> >> > >> >> >> > Significant progress has been made on yamldt and is now capable of >> >> >> > not only generating yaml from DTS source but also compiling DTS sources >> >> >> > and being almost fully compatible with DTC. >> >> >> >> >> >> Can you quantify "almost"? >> >> >> >> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a >> >> >> > DTC=yamldt. >> >> >> >> >> >> Good. >> >> >> >> >> >> > >> >> >> > Error reporting is accurate and validation against a YAML based schema >> >> >> > works as well. In a short while I will begin posting patches with >> >> >> > fixes on bindings and DTS files in the kernel. >> >> >> >> >> >> What I would like to see is the schema format posted for review. >> >> >> >> >> > >> >> > I'm including the skeleton.yaml binding which is the template for >> >> > the bindings and a board-example.yaml binding for a top level binding. >> >> > >> >> >> I would also like to see the bindings for top-level compatible strings >> >> >> (aka boards) as an example. That's something that's simple enough that >> >> >> I'd think we could agree on a format and start moving towards defining >> >> >> board bindings that way. >> >> >> >> >> > >> >> > Note there is some line wrapping I'm including a link >> >> > to the github repo of the files: >> >> > >> >> > >> >> > The skeleton.yaml >> >> > >> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml >> >> > >> >> > %YAML 1.1 >> >> > --- >> >> > # The name of the binding is first >> >> > # The anchor is put there for use by others >> >> > skeleton: &skeleton >> >> >> >> This and "id" seem redundant. >> >> >> > >> > Indeed. >> >> One other thing, "skeleton" is a bit weird as a key. It can't be >> validated. All the other keys are standard words. I could write >> "skeloton" by mistake and I guess I'd only find the mistake when >> something inherits it. That's somewhat true with id, but we can at >> least check "id" is present and that it's value is unique among all >> other id values. >> > > We can keep id and check that it matches the name of the enclosing node. > That way you can be sure that there's no error. > > But it's a bit weird cause this is similar to declaring a function name > with a typo. You won't find out until you use it. > >> > >> >> > version: 1 >> >> > >> >> > id: skel-device >> >> > >> >> > title: > >> >> > Skeleton Device >> >> > >> >> > maintainer: >> >> > name: Skeleton Person <skel@kernel.org> >> >> > >> >> > description: > >> >> > The Skeleton Device binding represents the SK11 device produced by >> >> > the Skeleton Corporation. The binding can also support compatible >> >> > clones made by second source vendors. >> >> > >> >> > # The class is an optional property that declares this >> >> > # binding as part of a larger set >> >> > # Multiple definitions are possible >> >> > class: [ device, spi-device ] >> >> > >> >> > # This binding inherits property characteristics from the generic >> >> > # spi-slave binding >> >> > # Note that the notation is standard yaml reference >> >> > inherits: *spi-slave >> >> > >> >> > # virtual bindings do not generate checkers >> >> > virtual: true >> >> >> >> virtual is an overloaded term. >> >> >> > >> > OK, what term should I use that this binding should not be instantiated >> > as a checker, only be used by other bindings when inherited? >> >> checks: true? >> >> I'd really like to avoid having to decide and drop this, but I don't >> really get why it is needed. >> > > It is needed because otherwise checker filters will be generated for > the template bindings that while they won't be executed they will be > compiled and take up space in the schema. Size is not a problem I have. That can come later if needed. >> >> > # each property is defined by each name >> >> > properties: >> >> > >> >> > # The compatible property is a reserved name. The type is always >> >> > "string" >> >> > # and should not be repeated device binding. >> >> > compatible: >> >> > category: required # required property >> >> > type: strseq # is a sequence of strings >> >> > >> >> > description: > >> >> > FX11 is a clone of the original SK11 device >> >> > >> >> > # v is always the name of the value of the property >> >> > # np is passed to the checker and is the current >> >> > # node pointer. We can access properties and call >> >> > # methods that operate on them. >> >> > # There can be multiple constraints, just put them >> >> > # into a sequence. >> >> > # Note that the BASE("skel,sk11") form from the previous >> >> > # binding will have to be reworked. >> >> > constraint: | >> >> > anystreq(v, "skel,sk11") || >> >> > anystreq(v, "faux,fx11") >> >> >> >> Constraints and logic ops were certainly not decided in the last >> >> attempt and I think will be the hardest part to define. I see you are >> >> using eBPF in the checker. Is that where anystreq comes from? >> >> >> > >> > Yes. The ebpf environment declares a number of methods that are executed >> > outside the ebpf sandbox. Check out >> > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml >> > https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c >> >> I looked at this some a while back. It wasn't clear to me what eBPF >> gives us and what you have defined on top of it. What I'm really after >> is documentation of the syntax and keywords here. >> > > eBPF is portable, can be serialized after compiling in the schema file > and can be executed in the kernel. Executing in the kernel is a non-goal for me. > By stripping out all documentation related properties and nodes keeping > only the compiled filters you can generate a dtb blob that passed to > kernel can be used for verification of all runtime changes in the > kernel's live tree. eBPF is enforcing an execution model that is 'safe' > so we can be sure that no foul play is possible. Humm, if you wanted to ensure dtb's are safe, I'd think that we just sign them like you would for the kernel or modules. > That means that you can a) run it at boot-time, verifying the dtb blob > passed by the bootloader for errors (potentially disabling devices > that their nodes fail) and b) run it when applying overlays to reject > any that result in an invalid tree. Let's get verification at build time working first, then we can worry about something like this. > One other use that I'm thinking that might be possible too, like > binding version check between what the driver is compiled against and > what the passed blob is without using an explicit version property. > > The constraint format ofcourse is plain C, you can generate a .so using > the host compiler and execute that if you are looking for performance. > > Syntax and keywords of the internals is in a bit of flux right now, > since the binding format is not finalized yet. I'd be happy to > discuss the internal at ELCE. I won't be there. I'm just trying to understand what eBPF defines if anything and what's up for discussion. [...] >> >> > constraint: | >> >> > get_depth(np) == 0 && ( >> >> >> >> Ahh, I guess this is how you are doing it. I don't think this works >> >> for any value other than 0. An MFD could be at any level. >> >> >> > >> > Well, I could've used a streq(get_name(get_parent(np)), "/") test but >> > this is faster. It's up to you what would work best. >> >> Why not just a "parent" key? >> > > Because the property/type prolog code would increase even for properties > that don't need the parent key. While generating the parent key only for > constraints that need it keeps the filter size smaller. One could argue that category and type are also constraints, but you don't express them as constraints. I think constraints should be limited to constraints on the value of the property which are not generically expressible (like type). I haven't fully thought through whether there's other cases. Another problem is this constraint is not really a constraint for compatible as you have it, but for the whole binding. [...] >> >> Overall, I think this format is a bit long for boards. We have >> >> something like ~1000 boards in arch/arm last I checked. I want adding >> >> a board binding to be very short and easy to review. It's often only a >> >> 1 line change. The main issue I have is it is just each SoC (or SoC >> >> family) does things their own way. >> >> >> > >> > Well, this is a full featured example; you could declare this a >> > 'virtual' (or what ever you want to call it binding) and use: >> > >> > board-example-foo: >> > >> > inherits: *board-example >> > >> > properties: >> > compatible: ... >> > >> > It is not absolutely terse, but it's still YAML. But for what is worth, >> > those YAML files can be generated using the C preprocessor. You could >> > define a macro that cuts the churn, albeit you lose the ability to >> > parse them as normal YAML files. >> >> C preprocessor doesn't sound like a great option to me. >> >> Perhaps it is by SoC compatible and boards are just an addition to the >> constraints. That's kind of how things are organized already. >> > > The code base supports multiple constraints so we could have multiple > constraint properties. How would you like your SoC example to > work ideally? Not sure exactly other than meeting the requirements that I gave. Rob ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-06 13:55 ` Rob Herring 0 siblings, 0 replies; 40+ messages in thread From: Rob Herring @ 2017-10-06 13:55 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > Hi Rob, > > On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote: >> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >> > Hi Rob, >> > >> > On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: >> >> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou >> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >> >> > Hi Rob, >> >> > >> >> > On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: >> >> >> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou >> >> >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >> >> >> > Hello again, >> >> >> > >> >> >> > Significant progress has been made on yamldt and is now capable of >> >> >> > not only generating yaml from DTS source but also compiling DTS sources >> >> >> > and being almost fully compatible with DTC. >> >> >> >> >> >> Can you quantify "almost"? >> >> >> >> >> >> > Compiling the kernel's DTBs using yamldt is as simple as using a >> >> >> > DTC=yamldt. >> >> >> >> >> >> Good. >> >> >> >> >> >> > >> >> >> > Error reporting is accurate and validation against a YAML based schema >> >> >> > works as well. In a short while I will begin posting patches with >> >> >> > fixes on bindings and DTS files in the kernel. >> >> >> >> >> >> What I would like to see is the schema format posted for review. >> >> >> >> >> > >> >> > I'm including the skeleton.yaml binding which is the template for >> >> > the bindings and a board-example.yaml binding for a top level binding. >> >> > >> >> >> I would also like to see the bindings for top-level compatible strings >> >> >> (aka boards) as an example. That's something that's simple enough that >> >> >> I'd think we could agree on a format and start moving towards defining >> >> >> board bindings that way. >> >> >> >> >> > >> >> > Note there is some line wrapping I'm including a link >> >> > to the github repo of the files: >> >> > >> >> > >> >> > The skeleton.yaml >> >> > >> >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml >> >> > >> >> > %YAML 1.1 >> >> > --- >> >> > # The name of the binding is first >> >> > # The anchor is put there for use by others >> >> > skeleton: &skeleton >> >> >> >> This and "id" seem redundant. >> >> >> > >> > Indeed. >> >> One other thing, "skeleton" is a bit weird as a key. It can't be >> validated. All the other keys are standard words. I could write >> "skeloton" by mistake and I guess I'd only find the mistake when >> something inherits it. That's somewhat true with id, but we can at >> least check "id" is present and that it's value is unique among all >> other id values. >> > > We can keep id and check that it matches the name of the enclosing node. > That way you can be sure that there's no error. > > But it's a bit weird cause this is similar to declaring a function name > with a typo. You won't find out until you use it. > >> > >> >> > version: 1 >> >> > >> >> > id: skel-device >> >> > >> >> > title: > >> >> > Skeleton Device >> >> > >> >> > maintainer: >> >> > name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> >> >> > >> >> > description: > >> >> > The Skeleton Device binding represents the SK11 device produced by >> >> > the Skeleton Corporation. The binding can also support compatible >> >> > clones made by second source vendors. >> >> > >> >> > # The class is an optional property that declares this >> >> > # binding as part of a larger set >> >> > # Multiple definitions are possible >> >> > class: [ device, spi-device ] >> >> > >> >> > # This binding inherits property characteristics from the generic >> >> > # spi-slave binding >> >> > # Note that the notation is standard yaml reference >> >> > inherits: *spi-slave >> >> > >> >> > # virtual bindings do not generate checkers >> >> > virtual: true >> >> >> >> virtual is an overloaded term. >> >> >> > >> > OK, what term should I use that this binding should not be instantiated >> > as a checker, only be used by other bindings when inherited? >> >> checks: true? >> >> I'd really like to avoid having to decide and drop this, but I don't >> really get why it is needed. >> > > It is needed because otherwise checker filters will be generated for > the template bindings that while they won't be executed they will be > compiled and take up space in the schema. Size is not a problem I have. That can come later if needed. >> >> > # each property is defined by each name >> >> > properties: >> >> > >> >> > # The compatible property is a reserved name. The type is always >> >> > "string" >> >> > # and should not be repeated device binding. >> >> > compatible: >> >> > category: required # required property >> >> > type: strseq # is a sequence of strings >> >> > >> >> > description: > >> >> > FX11 is a clone of the original SK11 device >> >> > >> >> > # v is always the name of the value of the property >> >> > # np is passed to the checker and is the current >> >> > # node pointer. We can access properties and call >> >> > # methods that operate on them. >> >> > # There can be multiple constraints, just put them >> >> > # into a sequence. >> >> > # Note that the BASE("skel,sk11") form from the previous >> >> > # binding will have to be reworked. >> >> > constraint: | >> >> > anystreq(v, "skel,sk11") || >> >> > anystreq(v, "faux,fx11") >> >> >> >> Constraints and logic ops were certainly not decided in the last >> >> attempt and I think will be the hardest part to define. I see you are >> >> using eBPF in the checker. Is that where anystreq comes from? >> >> >> > >> > Yes. The ebpf environment declares a number of methods that are executed >> > outside the ebpf sandbox. Check out >> > >> > https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml >> > https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c >> >> I looked at this some a while back. It wasn't clear to me what eBPF >> gives us and what you have defined on top of it. What I'm really after >> is documentation of the syntax and keywords here. >> > > eBPF is portable, can be serialized after compiling in the schema file > and can be executed in the kernel. Executing in the kernel is a non-goal for me. > By stripping out all documentation related properties and nodes keeping > only the compiled filters you can generate a dtb blob that passed to > kernel can be used for verification of all runtime changes in the > kernel's live tree. eBPF is enforcing an execution model that is 'safe' > so we can be sure that no foul play is possible. Humm, if you wanted to ensure dtb's are safe, I'd think that we just sign them like you would for the kernel or modules. > That means that you can a) run it at boot-time, verifying the dtb blob > passed by the bootloader for errors (potentially disabling devices > that their nodes fail) and b) run it when applying overlays to reject > any that result in an invalid tree. Let's get verification at build time working first, then we can worry about something like this. > One other use that I'm thinking that might be possible too, like > binding version check between what the driver is compiled against and > what the passed blob is without using an explicit version property. > > The constraint format ofcourse is plain C, you can generate a .so using > the host compiler and execute that if you are looking for performance. > > Syntax and keywords of the internals is in a bit of flux right now, > since the binding format is not finalized yet. I'd be happy to > discuss the internal at ELCE. I won't be there. I'm just trying to understand what eBPF defines if anything and what's up for discussion. [...] >> >> > constraint: | >> >> > get_depth(np) == 0 && ( >> >> >> >> Ahh, I guess this is how you are doing it. I don't think this works >> >> for any value other than 0. An MFD could be at any level. >> >> >> > >> > Well, I could've used a streq(get_name(get_parent(np)), "/") test but >> > this is faster. It's up to you what would work best. >> >> Why not just a "parent" key? >> > > Because the property/type prolog code would increase even for properties > that don't need the parent key. While generating the parent key only for > constraints that need it keeps the filter size smaller. One could argue that category and type are also constraints, but you don't express them as constraints. I think constraints should be limited to constraints on the value of the property which are not generically expressible (like type). I haven't fully thought through whether there's other cases. Another problem is this constraint is not really a constraint for compatible as you have it, but for the whole binding. [...] >> >> Overall, I think this format is a bit long for boards. We have >> >> something like ~1000 boards in arch/arm last I checked. I want adding >> >> a board binding to be very short and easy to review. It's often only a >> >> 1 line change. The main issue I have is it is just each SoC (or SoC >> >> family) does things their own way. >> >> >> > >> > Well, this is a full featured example; you could declare this a >> > 'virtual' (or what ever you want to call it binding) and use: >> > >> > board-example-foo: >> > >> > inherits: *board-example >> > >> > properties: >> > compatible: ... >> > >> > It is not absolutely terse, but it's still YAML. But for what is worth, >> > those YAML files can be generated using the C preprocessor. You could >> > define a macro that cuts the churn, albeit you lose the ability to >> > parse them as normal YAML files. >> >> C preprocessor doesn't sound like a great option to me. >> >> Perhaps it is by SoC compatible and boards are just an addition to the >> constraints. That's kind of how things are organized already. >> > > The code base supports multiple constraints so we could have multiple > constraint properties. How would you like your SoC example to > work ideally? Not sure exactly other than meeting the requirements that I gave. Rob -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-06 13:55 ` Rob Herring (?) @ 2017-10-07 10:23 ` Pantelis Antoniou 2017-10-08 23:08 ` Frank Rowand -1 siblings, 1 reply; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-07 10:23 UTC (permalink / raw) To: Rob Herring Cc: Frank Rowand, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi Rob, > On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: > > On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: >> Hi Rob, >> >> On Tue, 2017-10-03 at 12:13 -0500, Rob Herring wrote: >>> On Tue, Oct 3, 2017 at 9:13 AM, Pantelis Antoniou >>> <pantelis.antoniou@konsulko.com> wrote: >>>> Hi Rob, >>>> >>>> On Tue, 2017-10-03 at 08:18 -0500, Rob Herring wrote: >>>>> On Mon, Oct 2, 2017 at 2:46 PM, Pantelis Antoniou >>>>> <pantelis.antoniou@konsulko.com> wrote: >>>>>> Hi Rob, >>>>>> >>>>>> On Sun, 2017-10-01 at 17:00 -0500, Rob Herring wrote: >>>>>>> On Thu, Sep 28, 2017 at 2:58 PM, Pantelis Antoniou >>>>>>> <pantelis.antoniou@konsulko.com> wrote: >>>>>>>> Hello again, >>>>>>>> >>>>>>>> Significant progress has been made on yamldt and is now capable of >>>>>>>> not only generating yaml from DTS source but also compiling DTS sources >>>>>>>> and being almost fully compatible with DTC. >>>>>>> >>>>>>> Can you quantify "almost"? >>>>>>> >>>>>>>> Compiling the kernel's DTBs using yamldt is as simple as using a >>>>>>>> DTC=yamldt. >>>>>>> >>>>>>> Good. >>>>>>> >>>>>>>> >>>>>>>> Error reporting is accurate and validation against a YAML based schema >>>>>>>> works as well. In a short while I will begin posting patches with >>>>>>>> fixes on bindings and DTS files in the kernel. >>>>>>> >>>>>>> What I would like to see is the schema format posted for review. >>>>>>> >>>>>> >>>>>> I'm including the skeleton.yaml binding which is the template for >>>>>> the bindings and a board-example.yaml binding for a top level binding. >>>>>> >>>>>>> I would also like to see the bindings for top-level compatible strings >>>>>>> (aka boards) as an example. That's something that's simple enough that >>>>>>> I'd think we could agree on a format and start moving towards defining >>>>>>> board bindings that way. >>>>>>> >>>>>> >>>>>> Note there is some line wrapping I'm including a link >>>>>> to the github repo of the files: >>>>>> >>>>>> >>>>>> The skeleton.yaml >>>>>> >>>>>> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/bindings/skeleton.yaml >>>>>> >>>>>> %YAML 1.1 >>>>>> --- >>>>>> # The name of the binding is first >>>>>> # The anchor is put there for use by others >>>>>> skeleton: &skeleton >>>>> >>>>> This and "id" seem redundant. >>>>> >>>> >>>> Indeed. >>> >>> One other thing, "skeleton" is a bit weird as a key. It can't be >>> validated. All the other keys are standard words. I could write >>> "skeloton" by mistake and I guess I'd only find the mistake when >>> something inherits it. That's somewhat true with id, but we can at >>> least check "id" is present and that it's value is unique among all >>> other id values. >>> >> >> We can keep id and check that it matches the name of the enclosing node. >> That way you can be sure that there's no error. >> >> But it's a bit weird cause this is similar to declaring a function name >> with a typo. You won't find out until you use it. >> >>>> >>>>>> version: 1 >>>>>> >>>>>> id: skel-device >>>>>> >>>>>> title: > >>>>>> Skeleton Device >>>>>> >>>>>> maintainer: >>>>>> name: Skeleton Person <skel@kernel.org> >>>>>> >>>>>> description: > >>>>>> The Skeleton Device binding represents the SK11 device produced by >>>>>> the Skeleton Corporation. The binding can also support compatible >>>>>> clones made by second source vendors. >>>>>> >>>>>> # The class is an optional property that declares this >>>>>> # binding as part of a larger set >>>>>> # Multiple definitions are possible >>>>>> class: [ device, spi-device ] >>>>>> >>>>>> # This binding inherits property characteristics from the generic >>>>>> # spi-slave binding >>>>>> # Note that the notation is standard yaml reference >>>>>> inherits: *spi-slave >>>>>> >>>>>> # virtual bindings do not generate checkers >>>>>> virtual: true >>>>> >>>>> virtual is an overloaded term. >>>>> >>>> >>>> OK, what term should I use that this binding should not be instantiated >>>> as a checker, only be used by other bindings when inherited? >>> >>> checks: true? >>> >>> I'd really like to avoid having to decide and drop this, but I don't >>> really get why it is needed. >>> >> >> It is needed because otherwise checker filters will be generated for >> the template bindings that while they won't be executed they will be >> compiled and take up space in the schema. > > Size is not a problem I have. That can come later if needed. > >>>>>> # each property is defined by each name >>>>>> properties: >>>>>> >>>>>> # The compatible property is a reserved name. The type is always >>>>>> "string" >>>>>> # and should not be repeated device binding. >>>>>> compatible: >>>>>> category: required # required property >>>>>> type: strseq # is a sequence of strings >>>>>> >>>>>> description: > >>>>>> FX11 is a clone of the original SK11 device >>>>>> >>>>>> # v is always the name of the value of the property >>>>>> # np is passed to the checker and is the current >>>>>> # node pointer. We can access properties and call >>>>>> # methods that operate on them. >>>>>> # There can be multiple constraints, just put them >>>>>> # into a sequence. >>>>>> # Note that the BASE("skel,sk11") form from the previous >>>>>> # binding will have to be reworked. >>>>>> constraint: | >>>>>> anystreq(v, "skel,sk11") || >>>>>> anystreq(v, "faux,fx11") >>>>> >>>>> Constraints and logic ops were certainly not decided in the last >>>>> attempt and I think will be the hardest part to define. I see you are >>>>> using eBPF in the checker. Is that where anystreq comes from? >>>>> >>>> >>>> Yes. The ebpf environment declares a number of methods that are executed >>>> outside the ebpf sandbox. Check out >>>> >>>> https://raw.githubusercontent.com/pantoniou/yamldt/master/validate/schema/codegen.yaml >>>> https://github.com/pantoniou/yamldt/blob/master/ebpf_dt.c >>> >>> I looked at this some a while back. It wasn't clear to me what eBPF >>> gives us and what you have defined on top of it. What I'm really after >>> is documentation of the syntax and keywords here. >>> >> >> eBPF is portable, can be serialized after compiling in the schema file >> and can be executed in the kernel. > > Executing in the kernel is a non-goal for me. > >> By stripping out all documentation related properties and nodes keeping >> only the compiled filters you can generate a dtb blob that passed to >> kernel can be used for verification of all runtime changes in the >> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >> so we can be sure that no foul play is possible. > > Humm, if you wanted to ensure dtb's are safe, I'd think that we just > sign them like you would for the kernel or modules. > That’s a problem when deploying; the runtime validation is for making sure developers don’t get bogged down chasing problems when working on their own platforms/drivers. We have absolutely zero checks for stopping badly configured DT blobs hanging the kernel. With runtime validation a bug that might take a few days to figure out can be cut down to a few minutes. >> That means that you can a) run it at boot-time, verifying the dtb blob >> passed by the bootloader for errors (potentially disabling devices >> that their nodes fail) and b) run it when applying overlays to reject >> any that result in an invalid tree. > > Let's get verification at build time working first, then we can worry > about something like this. > >> One other use that I'm thinking that might be possible too, like >> binding version check between what the driver is compiled against and >> what the passed blob is without using an explicit version property. >> >> The constraint format ofcourse is plain C, you can generate a .so using >> the host compiler and execute that if you are looking for performance. >> >> Syntax and keywords of the internals is in a bit of flux right now, >> since the binding format is not finalized yet. I'd be happy to >> discuss the internal at ELCE. > > I won't be there. > > I'm just trying to understand what eBPF defines if anything and what's > up for discussion. The eBPF defines an API that the constraints use to perform validation. Anything that can be expressed as a C method can be in the eBPF API. > > [...] > >>>>>> constraint: | >>>>>> get_depth(np) == 0 && ( >>>>> >>>>> Ahh, I guess this is how you are doing it. I don't think this works >>>>> for any value other than 0. An MFD could be at any level. >>>>> >>>> >>>> Well, I could've used a streq(get_name(get_parent(np)), "/") test but >>>> this is faster. It's up to you what would work best. >>> >>> Why not just a "parent" key? >>> >> >> Because the property/type prolog code would increase even for properties >> that don't need the parent key. While generating the parent key only for >> constraints that need it keeps the filter size smaller. > > One could argue that category and type are also constraints, but you > don't express them as constraints. I think constraints should be > limited to constraints on the value of the property which are not > generically expressible (like type). I haven't fully thought through > whether there's other cases. > Both type and category constraints work in what I’ve posted, using the eBPF APIs. Taking for instance the enabled status constraint from ‘device-enabled.yaml’ > %YAML 1.1 > --- > device-enabled: > title: Contraint for enabled devices > > class: constraint > virtual: true > > properties: > status: &device-status-enabled > category: optional > type: str > description: Marks device state as enabled > constraint: | > !exists || streq(v, "okay") || streq(v, "ok”) > It generates the following code fragment. > { > uint64_t flags; > const char *v = get_str(np, "status", &flags); > const bool badtype = !!(flags & BADTYPE); > const bool exists = !!(flags & EXISTS); > > if (badtype) > return -BADTYPE_BASE - 100002; > > if (!exists) > goto skip_100002; > > /* for status from device-compatible rule */ > if (!( > !exists || streq(v, "okay") || streq(v, "ok") > )) > return -PROPC_BASE - 100002; > skip_100002: > do { } while(0); /* fix goto that requires a statement */ > > } > Both type checking (BADTYPE) and category (EXISTS) flags are returned and checked. > Another problem is this constraint is not really a constraint for > compatible as you have it, but for the whole binding. > What exactly is the difference in actual use? AFAIKT is our use of bindings and special meaning of the compatible property. > [...] > >>>>> Overall, I think this format is a bit long for boards. We have >>>>> something like ~1000 boards in arch/arm last I checked. I want adding >>>>> a board binding to be very short and easy to review. It's often only a >>>>> 1 line change. The main issue I have is it is just each SoC (or SoC >>>>> family) does things their own way. >>>>> >>>> >>>> Well, this is a full featured example; you could declare this a >>>> 'virtual' (or what ever you want to call it binding) and use: >>>> >>>> board-example-foo: >>>> >>>> inherits: *board-example >>>> >>>> properties: >>>> compatible: ... >>>> >>>> It is not absolutely terse, but it's still YAML. But for what is worth, >>>> those YAML files can be generated using the C preprocessor. You could >>>> define a macro that cuts the churn, albeit you lose the ability to >>>> parse them as normal YAML files. >>> >>> C preprocessor doesn't sound like a great option to me. >>> >>> Perhaps it is by SoC compatible and boards are just an addition to the >>> constraints. That's kind of how things are organized already. >>> >> >> The code base supports multiple constraints so we could have multiple >> constraint properties. How would you like your SoC example to >> work ideally? > > Not sure exactly other than meeting the requirements that I gave. > > Rob Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-08 23:08 ` Frank Rowand 0 siblings, 0 replies; 40+ messages in thread From: Frank Rowand @ 2017-10-08 23:08 UTC (permalink / raw) To: Pantelis Antoniou, Rob Herring Cc: Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel On 10/07/17 03:23, Pantelis Antoniou wrote: > Hi Rob, > >> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: >> >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou >> <pantelis.antoniou@konsulko.com> wrote: >>> Hi Rob, < snip > >>> eBPF is portable, can be serialized after compiling in the schema file >>> and can be executed in the kernel. >> >> Executing in the kernel is a non-goal for me. Executing in the kernel is an anti-goal for me. We are trying to reduce the device tree footprint inside the kernel, not increase it. 99.99% of the validation should be possible statically, in the compile phase. >>> By stripping out all documentation related properties and nodes keeping >>> only the compiled filters you can generate a dtb blob that passed to >>> kernel can be used for verification of all runtime changes in the >>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >>> so we can be sure that no foul play is possible. Run time changes can be assumed correct (short of bugs in the overlay application code), if the base tree is validated, the overlay is validated, and the interface between the live tree and the overlay is a connector. >> Humm, if you wanted to ensure dtb's are safe, I'd think that we just >> sign them like you would for the kernel or modules. >> > > That’s a problem when deploying; the runtime validation is for making sure > developers don’t get bogged down chasing problems when working on their > own platforms/drivers. > > We have absolutely zero checks for stopping badly configured DT blobs > hanging the kernel. With runtime validation a bug that might take a few > days to figure out can be cut down to a few minutes. Same reply as above. >>> That means that you can a) run it at boot-time, verifying the dtb blob >>> passed by the bootloader for errors (potentially disabling devices >>> that their nodes fail) and b) run it when applying overlays to reject >>> any that result in an invalid tree. >> >> Let's get verification at build time working first, then we can worry >> about something like this. < snip > -Frank ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-08 23:08 ` Frank Rowand 0 siblings, 0 replies; 40+ messages in thread From: Frank Rowand @ 2017-10-08 23:08 UTC (permalink / raw) To: Pantelis Antoniou, Rob Herring Cc: Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA On 10/07/17 03:23, Pantelis Antoniou wrote: > Hi Rob, > >> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >> >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >>> Hi Rob, < snip > >>> eBPF is portable, can be serialized after compiling in the schema file >>> and can be executed in the kernel. >> >> Executing in the kernel is a non-goal for me. Executing in the kernel is an anti-goal for me. We are trying to reduce the device tree footprint inside the kernel, not increase it. 99.99% of the validation should be possible statically, in the compile phase. >>> By stripping out all documentation related properties and nodes keeping >>> only the compiled filters you can generate a dtb blob that passed to >>> kernel can be used for verification of all runtime changes in the >>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >>> so we can be sure that no foul play is possible. Run time changes can be assumed correct (short of bugs in the overlay application code), if the base tree is validated, the overlay is validated, and the interface between the live tree and the overlay is a connector. >> Humm, if you wanted to ensure dtb's are safe, I'd think that we just >> sign them like you would for the kernel or modules. >> > > That’s a problem when deploying; the runtime validation is for making sure > developers don’t get bogged down chasing problems when working on their > own platforms/drivers. > > We have absolutely zero checks for stopping badly configured DT blobs > hanging the kernel. With runtime validation a bug that might take a few > days to figure out can be cut down to a few minutes. Same reply as above. >>> That means that you can a) run it at boot-time, verifying the dtb blob >>> passed by the bootloader for errors (potentially disabling devices >>> that their nodes fail) and b) run it when applying overlays to reject >>> any that result in an invalid tree. >> >> Let's get verification at build time working first, then we can worry >> about something like this. < snip > -Frank ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-09 0:00 ` David Gibson 0 siblings, 0 replies; 40+ messages in thread From: David Gibson @ 2017-10-09 0:00 UTC (permalink / raw) To: Frank Rowand Cc: Pantelis Antoniou, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel [-- Attachment #1: Type: text/plain, Size: 1973 bytes --] On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: > On 10/07/17 03:23, Pantelis Antoniou wrote: > > Hi Rob, > > > >> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: > >> > >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou > >> <pantelis.antoniou@konsulko.com> wrote: > >>> Hi Rob, > > < snip > > > >>> eBPF is portable, can be serialized after compiling in the schema file > >>> and can be executed in the kernel. > >> > >> Executing in the kernel is a non-goal for me. > > Executing in the kernel is an anti-goal for me. > > We are trying to reduce the device tree footprint inside the kernel, > not increase it. > > 99.99% of the validation should be possible statically, in the compile > phase. > > > >>> By stripping out all documentation related properties and nodes keeping > >>> only the compiled filters you can generate a dtb blob that passed to > >>> kernel can be used for verification of all runtime changes in the > >>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' > >>> so we can be sure that no foul play is possible. > > Run time changes can be assumed correct (short of bugs in the overlay > application code), if the base tree is validated, the overlay is validated, > and the interface between the live tree and the overlay is a > connector. In addition, no amount of schema validation can really protect the kernel from a bad DT. Even if the schemas can 100% verify that the DT is "syntactically" correct, which is ambitious, it can't protect against a DT which is in the right form, but contains information that is simply wrong for the hardware in question. That can stuff the kernel at least as easily as an incorrectly formatted DT. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-09 0:00 ` David Gibson 0 siblings, 0 replies; 40+ messages in thread From: David Gibson @ 2017-10-09 0:00 UTC (permalink / raw) To: Frank Rowand Cc: Pantelis Antoniou, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA [-- Attachment #1: Type: text/plain, Size: 2030 bytes --] On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: > On 10/07/17 03:23, Pantelis Antoniou wrote: > > Hi Rob, > > > >> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >> > >> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou > >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > >>> Hi Rob, > > < snip > > > >>> eBPF is portable, can be serialized after compiling in the schema file > >>> and can be executed in the kernel. > >> > >> Executing in the kernel is a non-goal for me. > > Executing in the kernel is an anti-goal for me. > > We are trying to reduce the device tree footprint inside the kernel, > not increase it. > > 99.99% of the validation should be possible statically, in the compile > phase. > > > >>> By stripping out all documentation related properties and nodes keeping > >>> only the compiled filters you can generate a dtb blob that passed to > >>> kernel can be used for verification of all runtime changes in the > >>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' > >>> so we can be sure that no foul play is possible. > > Run time changes can be assumed correct (short of bugs in the overlay > application code), if the base tree is validated, the overlay is validated, > and the interface between the live tree and the overlay is a > connector. In addition, no amount of schema validation can really protect the kernel from a bad DT. Even if the schemas can 100% verify that the DT is "syntactically" correct, which is ambitious, it can't protect against a DT which is in the right form, but contains information that is simply wrong for the hardware in question. That can stuff the kernel at least as easily as an incorrectly formatted DT. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-09 15:07 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-09 15:07 UTC (permalink / raw) To: David Gibson Cc: Frank Rowand, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi David, > On Oct 9, 2017, at 03:00 , David Gibson <david@gibson.dropbear.id.au> wrote: > > On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: >> On 10/07/17 03:23, Pantelis Antoniou wrote: >>> Hi Rob, >>> >>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: >>>> >>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou >>>> <pantelis.antoniou@konsulko.com> wrote: >>>>> Hi Rob, >> >> < snip > >> >>>>> eBPF is portable, can be serialized after compiling in the schema file >>>>> and can be executed in the kernel. >>>> >>>> Executing in the kernel is a non-goal for me. >> >> Executing in the kernel is an anti-goal for me. >> >> We are trying to reduce the device tree footprint inside the kernel, >> not increase it. >> >> 99.99% of the validation should be possible statically, in the compile >> phase. >> >> >>>>> By stripping out all documentation related properties and nodes keeping >>>>> only the compiled filters you can generate a dtb blob that passed to >>>>> kernel can be used for verification of all runtime changes in the >>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >>>>> so we can be sure that no foul play is possible. >> >> Run time changes can be assumed correct (short of bugs in the overlay >> application code), if the base tree is validated, the overlay is validated, >> and the interface between the live tree and the overlay is a >> connector. > > In addition, no amount of schema validation can really protect the > kernel from a bad DT. Even if the schemas can 100% verify that the DT > is "syntactically" correct, which is ambitious, it can't protect > against a DT which is in the right form, but contains information that > is simply wrong for the hardware in question. That can stuff the > kernel at least as easily as an incorrectly formatted DT. > I disagree. There are multiple levels of validation. For now we’re only talking about binding validation. There can be SoC level validation, board level validation, revision level validation and finally application specific validation. Binding validation is making sure properties/nodes follow the binding document. For instance that for a foo device there’s a mandatory interrupt property. Simplified interrupt = <X>; Binding validation would ‘catch’ errors like assigning a string or not having the interrupt property available. SoC level validation would list the available interrupt number that a given SoC would support for that device. For example that interrupt can only take the values 10 or 99 in a given SoC. Board level validation would narrow this down even further to a value of 10 for a given board model. Similar revision level validation would place further restriction on the allowed configuration. Finally application specific validation could place restriction based on the intended application that piece of hardware is used for. For instance devices that should not exceed a given power budget would have restrictions on the clock frequency of the processor or bus frequencies etc. > -- > David Gibson | I'll have my music baroque, and my code > david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ > | _way_ _around_! > http://www.ozlabs.org/~dgibson Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-09 15:07 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-09 15:07 UTC (permalink / raw) To: David Gibson Cc: Frank Rowand, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA Hi David, > On Oct 9, 2017, at 03:00 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote: > > On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: >> On 10/07/17 03:23, Pantelis Antoniou wrote: >>> Hi Rob, >>> >>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >>>> >>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou >>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >>>>> Hi Rob, >> >> < snip > >> >>>>> eBPF is portable, can be serialized after compiling in the schema file >>>>> and can be executed in the kernel. >>>> >>>> Executing in the kernel is a non-goal for me. >> >> Executing in the kernel is an anti-goal for me. >> >> We are trying to reduce the device tree footprint inside the kernel, >> not increase it. >> >> 99.99% of the validation should be possible statically, in the compile >> phase. >> >> >>>>> By stripping out all documentation related properties and nodes keeping >>>>> only the compiled filters you can generate a dtb blob that passed to >>>>> kernel can be used for verification of all runtime changes in the >>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >>>>> so we can be sure that no foul play is possible. >> >> Run time changes can be assumed correct (short of bugs in the overlay >> application code), if the base tree is validated, the overlay is validated, >> and the interface between the live tree and the overlay is a >> connector. > > In addition, no amount of schema validation can really protect the > kernel from a bad DT. Even if the schemas can 100% verify that the DT > is "syntactically" correct, which is ambitious, it can't protect > against a DT which is in the right form, but contains information that > is simply wrong for the hardware in question. That can stuff the > kernel at least as easily as an incorrectly formatted DT. > I disagree. There are multiple levels of validation. For now we’re only talking about binding validation. There can be SoC level validation, board level validation, revision level validation and finally application specific validation. Binding validation is making sure properties/nodes follow the binding document. For instance that for a foo device there’s a mandatory interrupt property. Simplified interrupt = <X>; Binding validation would ‘catch’ errors like assigning a string or not having the interrupt property available. SoC level validation would list the available interrupt number that a given SoC would support for that device. For example that interrupt can only take the values 10 or 99 in a given SoC. Board level validation would narrow this down even further to a value of 10 for a given board model. Similar revision level validation would place further restriction on the allowed configuration. Finally application specific validation could place restriction based on the intended application that piece of hardware is used for. For instance devices that should not exceed a given power budget would have restrictions on the clock frequency of the processor or bus frequencies etc. > -- > David Gibson | I'll have my music baroque, and my code > david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ > | _way_ _around_! > http://www.ozlabs.org/~dgibson Regards — Pantelis -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-09 15:07 ` Pantelis Antoniou (?) @ 2017-10-10 1:50 ` David Gibson 2017-10-10 15:19 ` Pantelis Antoniou -1 siblings, 1 reply; 40+ messages in thread From: David Gibson @ 2017-10-10 1:50 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel [-- Attachment #1: Type: text/plain, Size: 4066 bytes --] On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote: > Hi David, > > > On Oct 9, 2017, at 03:00 , David Gibson <david@gibson.dropbear.id.au> wrote: > > > > On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: > >> On 10/07/17 03:23, Pantelis Antoniou wrote: > >>> Hi Rob, > >>> > >>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: > >>>> > >>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou > >>>> <pantelis.antoniou@konsulko.com> wrote: > >>>>> Hi Rob, > >> > >> < snip > > >> > >>>>> eBPF is portable, can be serialized after compiling in the schema file > >>>>> and can be executed in the kernel. > >>>> > >>>> Executing in the kernel is a non-goal for me. > >> > >> Executing in the kernel is an anti-goal for me. > >> > >> We are trying to reduce the device tree footprint inside the kernel, > >> not increase it. > >> > >> 99.99% of the validation should be possible statically, in the compile > >> phase. > >> > >> > >>>>> By stripping out all documentation related properties and nodes keeping > >>>>> only the compiled filters you can generate a dtb blob that passed to > >>>>> kernel can be used for verification of all runtime changes in the > >>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' > >>>>> so we can be sure that no foul play is possible. > >> > >> Run time changes can be assumed correct (short of bugs in the overlay > >> application code), if the base tree is validated, the overlay is validated, > >> and the interface between the live tree and the overlay is a > >> connector. > > > > In addition, no amount of schema validation can really protect the > > kernel from a bad DT. Even if the schemas can 100% verify that the DT > > is "syntactically" correct, which is ambitious, it can't protect > > against a DT which is in the right form, but contains information that > > is simply wrong for the hardware in question. That can stuff the > > kernel at least as easily as an incorrectly formatted DT. > > > > I disagree. > > There are multiple levels of validation. For now we’re only talking about > binding validation. There can be SoC level validation, board level validation, > revision level validation and finally application specific validation. > > Binding validation is making sure properties/nodes follow the binding document. > For instance that for a foo device there’s a mandatory interrupt property. > > Simplified > > interrupt = <X>; > > Binding validation would ‘catch’ errors like assigning a string or not having the > interrupt property available. > > SoC level validation would list the available interrupt number that a given > SoC would support for that device. > > For example that interrupt can only take the values 10 or 99 in a given SoC. > > Board level validation would narrow this down even further to a value of 10 for > a given board model. > Similar revision level validation would place further restriction on the allowed > configuration. > > Finally application specific validation could place restriction based on the intended > application that piece of hardware is used for. For instance devices that should not > exceed a given power budget would have restrictions on the clock frequency of the processor > or bus frequencies etc. This doesn't help. In order to do this, the validator would need information that's essentially equivalent to the content of DT, at which point there's no point to the DT at all - and you're left with the problem of validating the information that the validator has. Fundamentally a validator that's useful *cannot* tell the difference between a correct tree and one which _could_ be correct for some theoretical hardware, but isn't for this particular hardware. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-10 15:19 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-10 15:19 UTC (permalink / raw) To: David Gibson Cc: Frank Rowand, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi David, > On Oct 10, 2017, at 04:50 , David Gibson <david@gibson.dropbear.id.au> wrote: > > On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote: >> Hi David, >> >>> On Oct 9, 2017, at 03:00 , David Gibson <david@gibson.dropbear.id.au> wrote: >>> >>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: >>>> On 10/07/17 03:23, Pantelis Antoniou wrote: >>>>> Hi Rob, >>>>> >>>>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: >>>>>> >>>>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou >>>>>> <pantelis.antoniou@konsulko.com> wrote: >>>>>>> Hi Rob, >>>> >>>> < snip > >>>> >>>>>>> eBPF is portable, can be serialized after compiling in the schema file >>>>>>> and can be executed in the kernel. >>>>>> >>>>>> Executing in the kernel is a non-goal for me. >>>> >>>> Executing in the kernel is an anti-goal for me. >>>> >>>> We are trying to reduce the device tree footprint inside the kernel, >>>> not increase it. >>>> >>>> 99.99% of the validation should be possible statically, in the compile >>>> phase. >>>> >>>> >>>>>>> By stripping out all documentation related properties and nodes keeping >>>>>>> only the compiled filters you can generate a dtb blob that passed to >>>>>>> kernel can be used for verification of all runtime changes in the >>>>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >>>>>>> so we can be sure that no foul play is possible. >>>> >>>> Run time changes can be assumed correct (short of bugs in the overlay >>>> application code), if the base tree is validated, the overlay is validated, >>>> and the interface between the live tree and the overlay is a >>>> connector. >>> >>> In addition, no amount of schema validation can really protect the >>> kernel from a bad DT. Even if the schemas can 100% verify that the DT >>> is "syntactically" correct, which is ambitious, it can't protect >>> against a DT which is in the right form, but contains information that >>> is simply wrong for the hardware in question. That can stuff the >>> kernel at least as easily as an incorrectly formatted DT. >>> >> >> I disagree. >> >> There are multiple levels of validation. For now we’re only talking about >> binding validation. There can be SoC level validation, board level validation, >> revision level validation and finally application specific validation. >> >> Binding validation is making sure properties/nodes follow the binding document. >> For instance that for a foo device there’s a mandatory interrupt property. >> >> Simplified >> >> interrupt = <X>; >> >> Binding validation would ‘catch’ errors like assigning a string or not having the >> interrupt property available. >> >> SoC level validation would list the available interrupt number that a given >> SoC would support for that device. >> >> For example that interrupt can only take the values 10 or 99 in a given SoC. >> >> Board level validation would narrow this down even further to a value of 10 for >> a given board model. > >> Similar revision level validation would place further restriction on the allowed >> configuration. >> >> Finally application specific validation could place restriction based on the intended >> application that piece of hardware is used for. For instance devices that should not >> exceed a given power budget would have restrictions on the clock frequency of the processor >> or bus frequencies etc. > > This doesn't help. In order to do this, the validator would need > information that's essentially equivalent to the content of DT, at > which point there's no point to the DT at all - and you're left with > the problem of validating the information that the validator has. > That would be the case if hardware IP only has a single way to be configured. The industry standard nowadays is picking reusable IP blocks and integrating them together in an SoC. The driver and the binding is common for every platform that uses that block, but the allowed configuration varies according to what the hardware people uses in a given instance. > Fundamentally a validator that's useful *cannot* tell the difference > between a correct tree and one which _could_ be correct for some > theoretical hardware, but isn't for this particular hardware. > That’s why there’s reason for a nested hierarchy of bindings IMO. Completeness of validation schemes can be a differentiating factor when choosing parts for hardware design. They would sure cut down development time. > -- > David Gibson | I'll have my music baroque, and my code > david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ > | _way_ _around_! > http://www.ozlabs.org/~dgibson Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-10 15:19 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-10 15:19 UTC (permalink / raw) To: David Gibson Cc: Frank Rowand, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA Hi David, > On Oct 10, 2017, at 04:50 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote: > > On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote: >> Hi David, >> >>> On Oct 9, 2017, at 03:00 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote: >>> >>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: >>>> On 10/07/17 03:23, Pantelis Antoniou wrote: >>>>> Hi Rob, >>>>> >>>>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: >>>>>> >>>>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou >>>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >>>>>>> Hi Rob, >>>> >>>> < snip > >>>> >>>>>>> eBPF is portable, can be serialized after compiling in the schema file >>>>>>> and can be executed in the kernel. >>>>>> >>>>>> Executing in the kernel is a non-goal for me. >>>> >>>> Executing in the kernel is an anti-goal for me. >>>> >>>> We are trying to reduce the device tree footprint inside the kernel, >>>> not increase it. >>>> >>>> 99.99% of the validation should be possible statically, in the compile >>>> phase. >>>> >>>> >>>>>>> By stripping out all documentation related properties and nodes keeping >>>>>>> only the compiled filters you can generate a dtb blob that passed to >>>>>>> kernel can be used for verification of all runtime changes in the >>>>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >>>>>>> so we can be sure that no foul play is possible. >>>> >>>> Run time changes can be assumed correct (short of bugs in the overlay >>>> application code), if the base tree is validated, the overlay is validated, >>>> and the interface between the live tree and the overlay is a >>>> connector. >>> >>> In addition, no amount of schema validation can really protect the >>> kernel from a bad DT. Even if the schemas can 100% verify that the DT >>> is "syntactically" correct, which is ambitious, it can't protect >>> against a DT which is in the right form, but contains information that >>> is simply wrong for the hardware in question. That can stuff the >>> kernel at least as easily as an incorrectly formatted DT. >>> >> >> I disagree. >> >> There are multiple levels of validation. For now we’re only talking about >> binding validation. There can be SoC level validation, board level validation, >> revision level validation and finally application specific validation. >> >> Binding validation is making sure properties/nodes follow the binding document. >> For instance that for a foo device there’s a mandatory interrupt property. >> >> Simplified >> >> interrupt = <X>; >> >> Binding validation would ‘catch’ errors like assigning a string or not having the >> interrupt property available. >> >> SoC level validation would list the available interrupt number that a given >> SoC would support for that device. >> >> For example that interrupt can only take the values 10 or 99 in a given SoC. >> >> Board level validation would narrow this down even further to a value of 10 for >> a given board model. > >> Similar revision level validation would place further restriction on the allowed >> configuration. >> >> Finally application specific validation could place restriction based on the intended >> application that piece of hardware is used for. For instance devices that should not >> exceed a given power budget would have restrictions on the clock frequency of the processor >> or bus frequencies etc. > > This doesn't help. In order to do this, the validator would need > information that's essentially equivalent to the content of DT, at > which point there's no point to the DT at all - and you're left with > the problem of validating the information that the validator has. > That would be the case if hardware IP only has a single way to be configured. The industry standard nowadays is picking reusable IP blocks and integrating them together in an SoC. The driver and the binding is common for every platform that uses that block, but the allowed configuration varies according to what the hardware people uses in a given instance. > Fundamentally a validator that's useful *cannot* tell the difference > between a correct tree and one which _could_ be correct for some > theoretical hardware, but isn't for this particular hardware. > That’s why there’s reason for a nested hierarchy of bindings IMO. Completeness of validation schemes can be a differentiating factor when choosing parts for hardware design. They would sure cut down development time. > -- > David Gibson | I'll have my music baroque, and my code > david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ > | _way_ _around_! > http://www.ozlabs.org/~dgibson Regards — Pantelis -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-11 3:49 ` David Gibson 0 siblings, 0 replies; 40+ messages in thread From: David Gibson @ 2017-10-11 3:49 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel [-- Attachment #1: Type: text/plain, Size: 5768 bytes --] On Tue, Oct 10, 2017 at 06:19:03PM +0300, Pantelis Antoniou wrote: > Hi David, > > > On Oct 10, 2017, at 04:50 , David Gibson <david@gibson.dropbear.id.au> wrote: > > > > On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote: > >> Hi David, > >> > >>> On Oct 9, 2017, at 03:00 , David Gibson <david@gibson.dropbear.id.au> wrote: > >>> > >>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: > >>>> On 10/07/17 03:23, Pantelis Antoniou wrote: > >>>>> Hi Rob, > >>>>> > >>>>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: > >>>>>> > >>>>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou > >>>>>> <pantelis.antoniou@konsulko.com> wrote: > >>>>>>> Hi Rob, > >>>> > >>>> < snip > > >>>> > >>>>>>> eBPF is portable, can be serialized after compiling in the schema file > >>>>>>> and can be executed in the kernel. > >>>>>> > >>>>>> Executing in the kernel is a non-goal for me. > >>>> > >>>> Executing in the kernel is an anti-goal for me. > >>>> > >>>> We are trying to reduce the device tree footprint inside the kernel, > >>>> not increase it. > >>>> > >>>> 99.99% of the validation should be possible statically, in the compile > >>>> phase. > >>>> > >>>> > >>>>>>> By stripping out all documentation related properties and nodes keeping > >>>>>>> only the compiled filters you can generate a dtb blob that passed to > >>>>>>> kernel can be used for verification of all runtime changes in the > >>>>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' > >>>>>>> so we can be sure that no foul play is possible. > >>>> > >>>> Run time changes can be assumed correct (short of bugs in the overlay > >>>> application code), if the base tree is validated, the overlay is validated, > >>>> and the interface between the live tree and the overlay is a > >>>> connector. > >>> > >>> In addition, no amount of schema validation can really protect the > >>> kernel from a bad DT. Even if the schemas can 100% verify that the DT > >>> is "syntactically" correct, which is ambitious, it can't protect > >>> against a DT which is in the right form, but contains information that > >>> is simply wrong for the hardware in question. That can stuff the > >>> kernel at least as easily as an incorrectly formatted DT. > >>> > >> > >> I disagree. > >> > >> There are multiple levels of validation. For now we’re only talking about > >> binding validation. There can be SoC level validation, board level validation, > >> revision level validation and finally application specific validation. > >> > >> Binding validation is making sure properties/nodes follow the binding document. > >> For instance that for a foo device there’s a mandatory interrupt property. > >> > >> Simplified > >> > >> interrupt = <X>; > >> > >> Binding validation would ‘catch’ errors like assigning a string or not having the > >> interrupt property available. > >> > >> SoC level validation would list the available interrupt number that a given > >> SoC would support for that device. > >> > >> For example that interrupt can only take the values 10 or 99 in a given SoC. > >> > >> Board level validation would narrow this down even further to a value of 10 for > >> a given board model. > > > >> Similar revision level validation would place further restriction on the allowed > >> configuration. > >> > >> Finally application specific validation could place restriction based on the intended > >> application that piece of hardware is used for. For instance devices that should not > >> exceed a given power budget would have restrictions on the clock frequency of the processor > >> or bus frequencies etc. > > > > This doesn't help. In order to do this, the validator would need > > information that's essentially equivalent to the content of DT, at > > which point there's no point to the DT at all - and you're left with > > the problem of validating the information that the validator has. > > That would be the case if hardware IP only has a single way to be configured. Right, and if if there's more than one way, then the validator can't possibly tell whether the DT has the right one. DTs must always come from a trusted source, because if they don't, then you don't need the DT in the first place (you could build your own). > The industry standard nowadays is picking reusable IP blocks and integrating them > together in an SoC. The driver and the binding is common for every platform that uses > that block, but the allowed configuration varies according to what the hardware > people uses in a given instance. > > Fundamentally a validator that's useful *cannot* tell the difference > > between a correct tree and one which _could_ be correct for some > > theoretical hardware, but isn't for this particular hardware. > > That’s why there’s reason for a nested hierarchy of bindings IMO. Nothing about how you structure the validation can change the basic fact that there are only two possibilities. Either: a) You know the hardware structure independent of the DT, in which case the DT is pointless or b) You don't know everything about the hardware without the DT, in which case you can't know if the DT is right for this hardware > Completeness of validation schemes can be a differentiating factor when > choosing parts for hardware design. They would sure cut down development time. > > > > Regards > > — Pantelis > -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-11 3:49 ` David Gibson 0 siblings, 0 replies; 40+ messages in thread From: David Gibson @ 2017-10-11 3:49 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, Rob Herring, Grant Likely, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA [-- Attachment #1: Type: text/plain, Size: 5881 bytes --] On Tue, Oct 10, 2017 at 06:19:03PM +0300, Pantelis Antoniou wrote: > Hi David, > > > On Oct 10, 2017, at 04:50 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote: > > > > On Mon, Oct 09, 2017 at 06:07:28PM +0300, Pantelis Antoniou wrote: > >> Hi David, > >> > >>> On Oct 9, 2017, at 03:00 , David Gibson <david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote: > >>> > >>> On Sun, Oct 08, 2017 at 04:08:03PM -0700, Frank Rowand wrote: > >>>> On 10/07/17 03:23, Pantelis Antoniou wrote: > >>>>> Hi Rob, > >>>>> > >>>>>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote: > >>>>>> > >>>>>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou > >>>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > >>>>>>> Hi Rob, > >>>> > >>>> < snip > > >>>> > >>>>>>> eBPF is portable, can be serialized after compiling in the schema file > >>>>>>> and can be executed in the kernel. > >>>>>> > >>>>>> Executing in the kernel is a non-goal for me. > >>>> > >>>> Executing in the kernel is an anti-goal for me. > >>>> > >>>> We are trying to reduce the device tree footprint inside the kernel, > >>>> not increase it. > >>>> > >>>> 99.99% of the validation should be possible statically, in the compile > >>>> phase. > >>>> > >>>> > >>>>>>> By stripping out all documentation related properties and nodes keeping > >>>>>>> only the compiled filters you can generate a dtb blob that passed to > >>>>>>> kernel can be used for verification of all runtime changes in the > >>>>>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' > >>>>>>> so we can be sure that no foul play is possible. > >>>> > >>>> Run time changes can be assumed correct (short of bugs in the overlay > >>>> application code), if the base tree is validated, the overlay is validated, > >>>> and the interface between the live tree and the overlay is a > >>>> connector. > >>> > >>> In addition, no amount of schema validation can really protect the > >>> kernel from a bad DT. Even if the schemas can 100% verify that the DT > >>> is "syntactically" correct, which is ambitious, it can't protect > >>> against a DT which is in the right form, but contains information that > >>> is simply wrong for the hardware in question. That can stuff the > >>> kernel at least as easily as an incorrectly formatted DT. > >>> > >> > >> I disagree. > >> > >> There are multiple levels of validation. For now we’re only talking about > >> binding validation. There can be SoC level validation, board level validation, > >> revision level validation and finally application specific validation. > >> > >> Binding validation is making sure properties/nodes follow the binding document. > >> For instance that for a foo device there’s a mandatory interrupt property. > >> > >> Simplified > >> > >> interrupt = <X>; > >> > >> Binding validation would ‘catch’ errors like assigning a string or not having the > >> interrupt property available. > >> > >> SoC level validation would list the available interrupt number that a given > >> SoC would support for that device. > >> > >> For example that interrupt can only take the values 10 or 99 in a given SoC. > >> > >> Board level validation would narrow this down even further to a value of 10 for > >> a given board model. > > > >> Similar revision level validation would place further restriction on the allowed > >> configuration. > >> > >> Finally application specific validation could place restriction based on the intended > >> application that piece of hardware is used for. For instance devices that should not > >> exceed a given power budget would have restrictions on the clock frequency of the processor > >> or bus frequencies etc. > > > > This doesn't help. In order to do this, the validator would need > > information that's essentially equivalent to the content of DT, at > > which point there's no point to the DT at all - and you're left with > > the problem of validating the information that the validator has. > > That would be the case if hardware IP only has a single way to be configured. Right, and if if there's more than one way, then the validator can't possibly tell whether the DT has the right one. DTs must always come from a trusted source, because if they don't, then you don't need the DT in the first place (you could build your own). > The industry standard nowadays is picking reusable IP blocks and integrating them > together in an SoC. The driver and the binding is common for every platform that uses > that block, but the allowed configuration varies according to what the hardware > people uses in a given instance. > > Fundamentally a validator that's useful *cannot* tell the difference > > between a correct tree and one which _could_ be correct for some > > theoretical hardware, but isn't for this particular hardware. > > That’s why there’s reason for a nested hierarchy of bindings IMO. Nothing about how you structure the validation can change the basic fact that there are only two possibilities. Either: a) You know the hardware structure independent of the DT, in which case the DT is pointless or b) You don't know everything about the hardware without the DT, in which case you can't know if the DT is right for this hardware > Completeness of validation schemes can be a differentiating factor when > choosing parts for hardware design. They would sure cut down development time. > > > > Regards > > — Pantelis > -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-08 23:08 ` Frank Rowand (?) (?) @ 2017-10-09 14:59 ` Pantelis Antoniou -1 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-09 14:59 UTC (permalink / raw) To: Frank Rowand Cc: Rob Herring, Grant Likely, David Gibson, Tom Rini, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, linux-kernel Hi Frank, > On Oct 9, 2017, at 02:08 , Frank Rowand <frowand.list@gmail.com> wrote: > > On 10/07/17 03:23, Pantelis Antoniou wrote: >> Hi Rob, >> >>> On Oct 6, 2017, at 16:55 , Rob Herring <robherring2@gmail.com> wrote: >>> >>> On Tue, Oct 3, 2017 at 12:39 PM, Pantelis Antoniou >>> <pantelis.antoniou@konsulko.com> wrote: >>>> Hi Rob, > > < snip > > >>>> eBPF is portable, can be serialized after compiling in the schema file >>>> and can be executed in the kernel. >>> >>> Executing in the kernel is a non-goal for me. > > Executing in the kernel is an anti-goal for me. > > We are trying to reduce the device tree footprint inside the kernel, > not increase it. > > 99.99% of the validation should be possible statically, in the compile > phase. > That’s not possible when you dynamically alter the tree at runtime. > >>>> By stripping out all documentation related properties and nodes keeping >>>> only the compiled filters you can generate a dtb blob that passed to >>>> kernel can be used for verification of all runtime changes in the >>>> kernel's live tree. eBPF is enforcing an execution model that is 'safe' >>>> so we can be sure that no foul play is possible. > > Run time changes can be assumed correct (short of bugs in the overlay > application code), if the base tree is validated, the overlay is validated, > and the interface between the live tree and the overlay is a connector. > You can validate the base tree statically but not the overlays. In fact a large percentage of overlay usage simply modifies a status property to turn on or off a device. There is nothing to validate in such case. The portable connector is still a long ways off and I haven’t seen anything that could handle something trickier that a few GPIOs and I2C or SPI busses. My goal is something that works covering all the cases without any surprising gotchas. > >>> Humm, if you wanted to ensure dtb's are safe, I'd think that we just >>> sign them like you would for the kernel or modules. >>> >> >> That’s a problem when deploying; the runtime validation is for making sure >> developers don’t get bogged down chasing problems when working on their >> own platforms/drivers. >> >> We have absolutely zero checks for stopping badly configured DT blobs >> hanging the kernel. With runtime validation a bug that might take a few >> days to figure out can be cut down to a few minutes. > > Same reply as above. > > >>>> That means that you can a) run it at boot-time, verifying the dtb blob >>>> passed by the bootloader for errors (potentially disabling devices >>>> that their nodes fail) and b) run it when applying overlays to reject >>>> any that result in an invalid tree. >>> >>> Let's get verification at build time working first, then we can worry >>> about something like this. > > < snip > > Right, let’s get build verification working first. > -Frank Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-09-28 19:58 ` Pantelis Antoniou @ 2017-10-20 17:46 ` Grant Likely -1 siblings, 0 replies; 40+ messages in thread From: Grant Likely @ 2017-10-20 17:46 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, Linux Kernel Mailing List On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > Hello again, > > Significant progress has been made on yamldt and is now capable of > not only generating yaml from DTS source but also compiling DTS sources > and being almost fully compatible with DTC. > > Compiling the kernel's DTBs using yamldt is as simple as using a > DTC=yamldt. > > Error reporting is accurate and validation against a YAML based schema > works as well. In a short while I will begin posting patches with > fixes on bindings and DTS files in the kernel. > > Please try it on your platform and report if you encounter any problems. > > https://github.com/pantoniou/yamldt > > I am eagerly awaiting for your comments. Hi Pantelis, This is good work. I've played around with it and I'm looking forward to chatting next week. One thing I've done is tried loading the output YAML files into another YAML interpreter and the current encoding causes problems. Specifically, in yamldt anchors/aliases are being used as a replacement for labels/phandles, but that conflicts with the YAML data model which defines a reference as a way to make a copy of the data appear in another part of the tree. For example, for the following snippit: intc: intc@10000 { #interrupt-cells = <1>; compatible = "acme,intc"; reg = <0x10000 0x1000>; gpio-controller; }; serial@20000 { compatible = "acme,uart"; reg = <0x20000 0x1000>; interrupt-parent = <&intc>; interrupts = <5>; }; yamldt will encode this as: intc@10000: &intc "#interrupt-cells": 1 compatible: acme,intc reg: [0x10000, 0x1000] gpio-controller: serial@20000: compatible: acme,uart reg: [0x20000, 0x1000] interrupt-parent: *intc interrupts: 5 But, the expected behaviour for a YAML parser is expand the alias '*intc' which results in the following structure: intc@10000: &intc "#interrupt-cells": 1 compatible: acme,intc reg: [0x10000, 0x1000] gpio-controller: serial@20000: compatible: acme,uart reg: [0x20000, 0x1000] interrupt-parent: "#interrupt-cells": 1 compatible: acme,intc reg: [0x10000, 0x1000] gpio-controller: interrupts: 5 See? It results in the entire interrupt controller node to appear as an instance under the interrupt-parent property, when the intention is only to create a phandle. yamldt should not redefine the behaviour of '*' aliases. Instead, it should use a different indicator, either using an explicit !phandle tag, or by replacing '*' with something else. I worked around it in my tests by replacing '*' with '$'. Plus, it would be useful to use normal YAML anchors/aliases for creating node templates. For example: serial-template: &acme-uart . # The anchor for the template compatible: acme,uart interrupt-parent: *intc root: serial@20000: <<: *acme-uart # Alias node merged into serial@20000 interrupts: 5 reg: [0x20000, 0x1000] serial@30000: <<: *acme-uart # Alias node merged into serial@30000 interrupts: 5 reg: [0x30000, 0x1000] Another problem with anchors/references is YAML seems to require the anchor to be defined before the reference, or at least that's what pyyaml and ruamel both expect. Regardless, The chosen YAML encoding should be readily consumable by existing yaml implementations without having to do a lot of customization. I'm slightly concerned about using & anchors for labels because it seems only one anchor can be defined per node, but DTC allows multiple labels for a single node. This might not be an issue in practice though. Another implementation issue related to using & anchors is the YAML spec defines them as an encoding artifact, and parsers can discard the anchor names after parsing the YAML structure, which is a problem if we use something like $name to reference an anchor. The solution might just be that labels need to go into a special property so they don't disappear from the data stream. There appears to be no place to put metadata. The root of the tree is the top level of the YAML structure. There isn't any provision for having a top level object to hold things like the memreserve map. We may need a namespace to use for special properties that aren't nodes or properties. The encoding differentiates between nodes and properties implicitly base on whether the contents are a map, or a scalar/list. This does mean any parser needs to do a bit more work and it may impact what can be done with validation (I'm not going to talk about validation in this email though. We'll talk next week.) Cheers, g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-20 17:46 ` Grant Likely 0 siblings, 0 replies; 40+ messages in thread From: Grant Likely @ 2017-10-20 17:46 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, Linux Kernel Mailing List On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > Hello again, > > Significant progress has been made on yamldt and is now capable of > not only generating yaml from DTS source but also compiling DTS sources > and being almost fully compatible with DTC. > > Compiling the kernel's DTBs using yamldt is as simple as using a > DTC=yamldt. > > Error reporting is accurate and validation against a YAML based schema > works as well. In a short while I will begin posting patches with > fixes on bindings and DTS files in the kernel. > > Please try it on your platform and report if you encounter any problems. > > https://github.com/pantoniou/yamldt > > I am eagerly awaiting for your comments. Hi Pantelis, This is good work. I've played around with it and I'm looking forward to chatting next week. One thing I've done is tried loading the output YAML files into another YAML interpreter and the current encoding causes problems. Specifically, in yamldt anchors/aliases are being used as a replacement for labels/phandles, but that conflicts with the YAML data model which defines a reference as a way to make a copy of the data appear in another part of the tree. For example, for the following snippit: intc: intc@10000 { #interrupt-cells = <1>; compatible = "acme,intc"; reg = <0x10000 0x1000>; gpio-controller; }; serial@20000 { compatible = "acme,uart"; reg = <0x20000 0x1000>; interrupt-parent = <&intc>; interrupts = <5>; }; yamldt will encode this as: intc@10000: &intc "#interrupt-cells": 1 compatible: acme,intc reg: [0x10000, 0x1000] gpio-controller: serial@20000: compatible: acme,uart reg: [0x20000, 0x1000] interrupt-parent: *intc interrupts: 5 But, the expected behaviour for a YAML parser is expand the alias '*intc' which results in the following structure: intc@10000: &intc "#interrupt-cells": 1 compatible: acme,intc reg: [0x10000, 0x1000] gpio-controller: serial@20000: compatible: acme,uart reg: [0x20000, 0x1000] interrupt-parent: "#interrupt-cells": 1 compatible: acme,intc reg: [0x10000, 0x1000] gpio-controller: interrupts: 5 See? It results in the entire interrupt controller node to appear as an instance under the interrupt-parent property, when the intention is only to create a phandle. yamldt should not redefine the behaviour of '*' aliases. Instead, it should use a different indicator, either using an explicit !phandle tag, or by replacing '*' with something else. I worked around it in my tests by replacing '*' with '$'. Plus, it would be useful to use normal YAML anchors/aliases for creating node templates. For example: serial-template: &acme-uart . # The anchor for the template compatible: acme,uart interrupt-parent: *intc root: serial@20000: <<: *acme-uart # Alias node merged into serial@20000 interrupts: 5 reg: [0x20000, 0x1000] serial@30000: <<: *acme-uart # Alias node merged into serial@30000 interrupts: 5 reg: [0x30000, 0x1000] Another problem with anchors/references is YAML seems to require the anchor to be defined before the reference, or at least that's what pyyaml and ruamel both expect. Regardless, The chosen YAML encoding should be readily consumable by existing yaml implementations without having to do a lot of customization. I'm slightly concerned about using & anchors for labels because it seems only one anchor can be defined per node, but DTC allows multiple labels for a single node. This might not be an issue in practice though. Another implementation issue related to using & anchors is the YAML spec defines them as an encoding artifact, and parsers can discard the anchor names after parsing the YAML structure, which is a problem if we use something like $name to reference an anchor. The solution might just be that labels need to go into a special property so they don't disappear from the data stream. There appears to be no place to put metadata. The root of the tree is the top level of the YAML structure. There isn't any provision for having a top level object to hold things like the memreserve map. We may need a namespace to use for special properties that aren't nodes or properties. The encoding differentiates between nodes and properties implicitly base on whether the contents are a map, or a scalar/list. This does mean any parser needs to do a bit more work and it may impact what can be done with validation (I'm not going to talk about validation in this email though. We'll talk next week.) Cheers, g. -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too 2017-10-20 17:46 ` Grant Likely (?) @ 2017-10-20 19:16 ` Pantelis Antoniou 2017-10-22 17:54 ` Grant Likely -1 siblings, 1 reply; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-20 19:16 UTC (permalink / raw) To: Grant Likely Cc: Frank Rowand, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, Linux Kernel Mailing List Hi Grant, > On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely@secretlab.ca> wrote: > > On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: >> Hello again, >> >> Significant progress has been made on yamldt and is now capable of >> not only generating yaml from DTS source but also compiling DTS sources >> and being almost fully compatible with DTC. >> >> Compiling the kernel's DTBs using yamldt is as simple as using a >> DTC=yamldt. >> >> Error reporting is accurate and validation against a YAML based schema >> works as well. In a short while I will begin posting patches with >> fixes on bindings and DTS files in the kernel. >> >> Please try it on your platform and report if you encounter any problems. >> >> https://github.com/pantoniou/yamldt >> >> I am eagerly awaiting for your comments. > > Hi Pantelis, > > This is good work. I've played around with it and I'm looking forward > to chatting next week. > Thanks. I’m looking forward to it too. > One thing I've done is tried loading the output YAML files into > another YAML interpreter and the current encoding causes problems. > Specifically, in yamldt anchors/aliases are being used as a > replacement for labels/phandles, but that conflicts with the YAML data > model which defines a reference as a way to make a copy of the data > appear in another part of the tree. For example, for the following > snippit: > > intc: intc@10000 { > #interrupt-cells = <1>; > compatible = "acme,intc"; > reg = <0x10000 0x1000>; > gpio-controller; > }; > > serial@20000 { > compatible = "acme,uart"; > reg = <0x20000 0x1000>; > interrupt-parent = <&intc>; > interrupts = <5>; > }; > > yamldt will encode this as: > > intc@10000: &intc > "#interrupt-cells": 1 > compatible: acme,intc > reg: [0x10000, 0x1000] > gpio-controller: > > serial@20000: > compatible: acme,uart > reg: [0x20000, 0x1000] > interrupt-parent: *intc > interrupts: 5 > > But, the expected behaviour for a YAML parser is expand the alias > '*intc' which results in the following structure: > > intc@10000: &intc > "#interrupt-cells": 1 > compatible: acme,intc > reg: [0x10000, 0x1000] > gpio-controller: > > serial@20000: > compatible: acme,uart > reg: [0x20000, 0x1000] > interrupt-parent: > "#interrupt-cells": 1 > compatible: acme,intc > reg: [0x10000, 0x1000] > gpio-controller: > interrupts: 5 > > See? It results in the entire interrupt controller node to appear as > an instance under the interrupt-parent property, when the intention is > only to create a phandle. yamldt should not redefine the behaviour of > '*' aliases. Instead, it should use a different indicator, either > using an explicit !phandle tag, or by replacing '*' with something > else. I worked around it in my tests by replacing '*' with '$’. Yes. This is expected. I don’t think pure YAML form is a good match for all the crazy things that are possible (and actually used in practice) with DTS. For instance there’s no way a normal YAML parser won’t choke with something like / { foo; }; / { bar; }; Which is a common idiom in DTS files. The decision to use the YAML anchors and references was more borne out of a desire to sort of match conceptually the labels and references of DTS. It’s not a big issue to switch to something different. If we were to force YAML/DT file to be regular YAML files parseable by everything we’d have to rethink a lot of conventions that DT files currently use and I’m afraid a machine translation from DTS to YAML might not be feasible then. OTOH moving to that model might make it possible to use YAML constructs that are not mapped at all to DTS. For instance - foo: true bar: “string” - foo: false bar: “another-string” is not possible to be mapped to a DT node/property structure right now. > > Plus, it would be useful to use normal YAML anchors/aliases for > creating node templates. For example: > > serial-template: &acme-uart . # The anchor for the template > compatible: acme,uart > interrupt-parent: *intc > > root: > serial@20000: > <<: *acme-uart # Alias node merged into serial@20000 > interrupts: 5 > reg: [0x20000, 0x1000] > serial@30000: > <<: *acme-uart # Alias node merged into serial@30000 > interrupts: 5 > reg: [0x30000, 0x1000] > Yes, I’m aware of this and I planned to talk to you about it :) It can be a very powerful way to cut down the churn of DT right now. It’s not going to be a problem for yamldt to support it (perhaps adding a new editing tags to allow more fine grained control of the substitions). > Another problem with anchors/references is YAML seems to require the > anchor to be defined before the reference, or at least that's what > pyyaml and ruamel both expect. Regardless, The chosen YAML encoding > should be readily consumable by existing yaml implementations without > having to do a lot of customization. > I think this is only true for streamable YAML implementations; yamldt can leave references unresolved until the emit phase (or even leave them be as object files). > I'm slightly concerned about using & anchors for labels because it > seems only one anchor can be defined per node, but DTC allows multiple > labels for a single node. This might not be an issue in practice > though. Another implementation issue related to using & anchors is the > YAML spec defines them as an encoding artifact, and parsers can > discard the anchor names after parsing the YAML structure, which is a > problem if we use something like $name to reference an anchor. The > solution might just be that labels need to go into a special property > so they don't disappear from the data stream. > I do support multiple labels to the same node in yamldt with the following trick, that uses an empty tree. foo: bar: baz { frob; }; bar: &foo frob: true bar: &bar ~: ~ Special properties should work, albeit being a little bit cludgy. > There appears to be no place to put metadata. The root of the tree is > the top level of the YAML structure. There isn't any provision for > having a top level object to hold things like the memreserve map. We > may need a namespace to use for special properties that aren't nodes > or properties. > I don’t think that’s going to be necessary. Metadata are artifacts that only have meaning when emitting a serialized form like dtb. yamldt fully supports /memreserve/ by simply treating a top level /memreserve/ property as special and constructing the memreserve entries at the emit phase. so /memreserve/ 1000 10; is simply: /memreserve/: [ 1000 10 ] It’s a matter of definition what properties are encoded as metadata for the serialized form. > The encoding differentiates between nodes and properties implicitly > base on whether the contents are a map, or a scalar/list. This does > mean any parser needs to do a bit more work and it may impact what can > be done with validation (I'm not going to talk about validation in > this email though. We'll talk next week.) Hmm, I’m not sure I follow this. Is that related to the metadata problem. BTW, a blog post explaining the rationale behind yamldt is going to come up soon at our website, I’ll post the link when it does. > > Cheers, > g. Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-22 17:54 ` Grant Likely 0 siblings, 0 replies; 40+ messages in thread From: Grant Likely @ 2017-10-22 17:54 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, Linux Kernel Mailing List On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > Hi Grant, > >> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely@secretlab.ca> wrote: >> >> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou >> <pantelis.antoniou@konsulko.com> wrote: >>> Hello again, >>> >>> Significant progress has been made on yamldt and is now capable of >>> not only generating yaml from DTS source but also compiling DTS sources >>> and being almost fully compatible with DTC. >>> >>> Compiling the kernel's DTBs using yamldt is as simple as using a >>> DTC=yamldt. >>> >>> Error reporting is accurate and validation against a YAML based schema >>> works as well. In a short while I will begin posting patches with >>> fixes on bindings and DTS files in the kernel. >>> >>> Please try it on your platform and report if you encounter any problems. >>> >>> https://github.com/pantoniou/yamldt >>> >>> I am eagerly awaiting for your comments. >> >> Hi Pantelis, >> >> This is good work. I've played around with it and I'm looking forward >> to chatting next week. >> > > Thanks. I’m looking forward to it too. > >> One thing I've done is tried loading the output YAML files into >> another YAML interpreter and the current encoding causes problems. >> Specifically, in yamldt anchors/aliases are being used as a >> replacement for labels/phandles, but that conflicts with the YAML data >> model which defines a reference as a way to make a copy of the data >> appear in another part of the tree. For example, for the following >> snippit: >> >> intc: intc@10000 { >> #interrupt-cells = <1>; >> compatible = "acme,intc"; >> reg = <0x10000 0x1000>; >> gpio-controller; >> }; >> >> serial@20000 { >> compatible = "acme,uart"; >> reg = <0x20000 0x1000>; >> interrupt-parent = <&intc>; >> interrupts = <5>; >> }; >> >> yamldt will encode this as: >> >> intc@10000: &intc >> "#interrupt-cells": 1 >> compatible: acme,intc >> reg: [0x10000, 0x1000] >> gpio-controller: >> >> serial@20000: >> compatible: acme,uart >> reg: [0x20000, 0x1000] >> interrupt-parent: *intc >> interrupts: 5 >> >> But, the expected behaviour for a YAML parser is expand the alias >> '*intc' which results in the following structure: >> >> intc@10000: &intc >> "#interrupt-cells": 1 >> compatible: acme,intc >> reg: [0x10000, 0x1000] >> gpio-controller: >> >> serial@20000: >> compatible: acme,uart >> reg: [0x20000, 0x1000] >> interrupt-parent: >> "#interrupt-cells": 1 >> compatible: acme,intc >> reg: [0x10000, 0x1000] >> gpio-controller: >> interrupts: 5 >> >> See? It results in the entire interrupt controller node to appear as >> an instance under the interrupt-parent property, when the intention is >> only to create a phandle. yamldt should not redefine the behaviour of >> '*' aliases. Instead, it should use a different indicator, either >> using an explicit !phandle tag, or by replacing '*' with something >> else. I worked around it in my tests by replacing '*' with '$’. > > Yes. This is expected. I don’t think pure YAML form is a good match for all > the crazy things that are possible (and actually used in practice) with DTS. I don’t think it is as dire as that. The DTS structure is not complex and I think can easily be mapped into pure YAML. But, it does require treating the DTS structure separately from its semantic meaning. For example, the grammar of nodes, properties and labels easily maps to pure YAML, but phandles and overlay trees have semantic meaning that must be resolved by DT specific code. I’ll respond to you’re specific examples below... > > For instance there’s no way a normal YAML parser won’t choke with something like > > / { > foo; > }; > > / { > bar; > }; > > Which is a common idiom in DTS files. That’s only true when the top level of nodes is encoded as a map, but it shouldn’t be. It should be a list instead, for two reasons. First, order matters for the top level. Subsequent top level trees have to be applied in order to fully resolve the tree, but a map doesn’t preserve the ordering. Second, as you rightly point out, the same name can be used for multiple top level trees. So, the encoding should be a list with each list entry containing the node path {/path,&ref}, and the node data. One possible way to do this is with a tuple: treedata: - - / - foo: - - / - bar: Another option is to add a special property in the node to contain the name: treedata: - $path: / foo: - $path: / bar: Personally, I prefer the special property approach. That would also be a good way to encode labels > The decision to use the YAML anchors and references was more borne out of a desire > to sort of match conceptually the labels and references of DTS. It’s not a big > issue to switch to something different. I think it would be a good idea. One reason for defining a YAML encoding is to be able to use existing tools to work with the data. Changing the meaning of anchors and aliases breaks those tools immediately. > If we were to force YAML/DT file to be regular YAML files parseable by everything > we’d have to rethink a lot of conventions that DT files currently use and I’m afraid > a machine translation from DTS to YAML might not be feasible then. > > OTOH moving to that model might make it possible to use YAML constructs that are not > mapped at all to DTS. For instance > > - foo: true > bar: “string” > > - foo: false > bar: “another-string” > > is not possible to be mapped to a DT node/property structure right now. I’m not following. Are you saying DT has no way to encode a list of nodes? What use case are you imagining? > >> > >> Plus, it would be useful to use normal YAML anchors/aliases for >> creating node templates. For example: >> >> serial-template: &acme-uart . # The anchor for the template >> compatible: acme,uart >> interrupt-parent: *intc >> >> root: >> serial@20000: >> <<: *acme-uart # Alias node merged into serial@20000 >> interrupts: 5 >> reg: [0x20000, 0x1000] >> serial@30000: >> <<: *acme-uart # Alias node merged into serial@30000 >> interrupts: 5 >> reg: [0x30000, 0x1000] >> > > Yes, I’m aware of this and I planned to talk to you about it :) > It can be a very powerful way to cut down the churn of DT right now. > > It’s not going to be a problem for yamldt to support it (perhaps > adding a new editing tags to allow more fine grained control of the > substitions). > >> Another problem with anchors/references is YAML seems to require the >> anchor to be defined before the reference, or at least that's what >> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding >> should be readily consumable by existing yaml implementations without >> having to do a lot of customization. >> > > I think this is only true for streamable YAML implementations; yamldt > can leave references unresolved until the emit phase (or even leave them > be as object files). it is possible (probable) that I’m just not familiar with how to change that behaviour in pyyaml and ruamel. However, the point is moot if we can agree to preserve pure YAML semantics. >> I'm slightly concerned about using & anchors for labels because it >> seems only one anchor can be defined per node, but DTC allows multiple >> labels for a single node. This might not be an issue in practice >> though. Another implementation issue related to using & anchors is the >> YAML spec defines them as an encoding artifact, and parsers can >> discard the anchor names after parsing the YAML structure, which is a >> problem if we use something like $name to reference an anchor. The >> solution might just be that labels need to go into a special property >> so they don't disappear from the data stream. >> > > I do support multiple labels to the same node in yamldt with the > following trick, that uses an empty tree. > > foo: bar: baz { frob; }; > > bar: &foo > frob: true > > bar: &bar > ~: ~ > > Special properties should work, albeit being a little bit cludgy. With special properties, it might look like this: baz: $labels: [foo, bar] frob: > >> There appears to be no place to put metadata. The root of the tree is >> the top level of the YAML structure. There isn't any provision for >> having a top level object to hold things like the memreserve map. We >> may need a namespace to use for special properties that aren't nodes >> or properties. >> > > I don’t think that’s going to be necessary. Metadata are artifacts that > only have meaning when emitting a serialized form like dtb. > > yamldt fully supports /memreserve/ by simply treating a top level > /memreserve/ property as special and constructing the memreserve > entries at the emit phase. > > so > > /memreserve/ 1000 10; > > is simply: > > /memreserve/: [ 1000 10 ] > > It’s a matter of definition what properties are encoded as metadata for the > serialized form. > >> The encoding differentiates between nodes and properties implicitly >> base on whether the contents are a map, or a scalar/list. This does >> mean any parser needs to do a bit more work and it may impact what can >> be done with validation (I'm not going to talk about validation in >> this email though. We'll talk next week.) > > Hmm, I’m not sure I follow this. Is that related to the metadata problem. It has to do with iterating over properties or nodes. Nodes and props are mixed together, and any code processing them has to explicitly check the type to differentiate between the two. The alternative would be to collect all properties under a single key: parent: $props: foo: hello bar: world child1: {} child2: {} > > BTW, a blog post explaining the rationale behind yamldt is going to come > up soon at our website, I’ll post the link when it does. Cool. I see it got posted, but didn’t get a chance to read it before I got on this airplane. Will try to read this evening. (BTW, I’m arriving at about 11:00 tonight) Cheers, g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-22 17:54 ` Grant Likely 0 siblings, 0 replies; 40+ messages in thread From: Grant Likely @ 2017-10-22 17:54 UTC (permalink / raw) To: Pantelis Antoniou Cc: Frank Rowand, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, Linux Kernel Mailing List On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > Hi Grant, > >> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote: >> >> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >>> Hello again, >>> >>> Significant progress has been made on yamldt and is now capable of >>> not only generating yaml from DTS source but also compiling DTS sources >>> and being almost fully compatible with DTC. >>> >>> Compiling the kernel's DTBs using yamldt is as simple as using a >>> DTC=yamldt. >>> >>> Error reporting is accurate and validation against a YAML based schema >>> works as well. In a short while I will begin posting patches with >>> fixes on bindings and DTS files in the kernel. >>> >>> Please try it on your platform and report if you encounter any problems. >>> >>> https://github.com/pantoniou/yamldt >>> >>> I am eagerly awaiting for your comments. >> >> Hi Pantelis, >> >> This is good work. I've played around with it and I'm looking forward >> to chatting next week. >> > > Thanks. I’m looking forward to it too. > >> One thing I've done is tried loading the output YAML files into >> another YAML interpreter and the current encoding causes problems. >> Specifically, in yamldt anchors/aliases are being used as a >> replacement for labels/phandles, but that conflicts with the YAML data >> model which defines a reference as a way to make a copy of the data >> appear in another part of the tree. For example, for the following >> snippit: >> >> intc: intc@10000 { >> #interrupt-cells = <1>; >> compatible = "acme,intc"; >> reg = <0x10000 0x1000>; >> gpio-controller; >> }; >> >> serial@20000 { >> compatible = "acme,uart"; >> reg = <0x20000 0x1000>; >> interrupt-parent = <&intc>; >> interrupts = <5>; >> }; >> >> yamldt will encode this as: >> >> intc@10000: &intc >> "#interrupt-cells": 1 >> compatible: acme,intc >> reg: [0x10000, 0x1000] >> gpio-controller: >> >> serial@20000: >> compatible: acme,uart >> reg: [0x20000, 0x1000] >> interrupt-parent: *intc >> interrupts: 5 >> >> But, the expected behaviour for a YAML parser is expand the alias >> '*intc' which results in the following structure: >> >> intc@10000: &intc >> "#interrupt-cells": 1 >> compatible: acme,intc >> reg: [0x10000, 0x1000] >> gpio-controller: >> >> serial@20000: >> compatible: acme,uart >> reg: [0x20000, 0x1000] >> interrupt-parent: >> "#interrupt-cells": 1 >> compatible: acme,intc >> reg: [0x10000, 0x1000] >> gpio-controller: >> interrupts: 5 >> >> See? It results in the entire interrupt controller node to appear as >> an instance under the interrupt-parent property, when the intention is >> only to create a phandle. yamldt should not redefine the behaviour of >> '*' aliases. Instead, it should use a different indicator, either >> using an explicit !phandle tag, or by replacing '*' with something >> else. I worked around it in my tests by replacing '*' with '$’. > > Yes. This is expected. I don’t think pure YAML form is a good match for all > the crazy things that are possible (and actually used in practice) with DTS. I don’t think it is as dire as that. The DTS structure is not complex and I think can easily be mapped into pure YAML. But, it does require treating the DTS structure separately from its semantic meaning. For example, the grammar of nodes, properties and labels easily maps to pure YAML, but phandles and overlay trees have semantic meaning that must be resolved by DT specific code. I’ll respond to you’re specific examples below... > > For instance there’s no way a normal YAML parser won’t choke with something like > > / { > foo; > }; > > / { > bar; > }; > > Which is a common idiom in DTS files. That’s only true when the top level of nodes is encoded as a map, but it shouldn’t be. It should be a list instead, for two reasons. First, order matters for the top level. Subsequent top level trees have to be applied in order to fully resolve the tree, but a map doesn’t preserve the ordering. Second, as you rightly point out, the same name can be used for multiple top level trees. So, the encoding should be a list with each list entry containing the node path {/path,&ref}, and the node data. One possible way to do this is with a tuple: treedata: - - / - foo: - - / - bar: Another option is to add a special property in the node to contain the name: treedata: - $path: / foo: - $path: / bar: Personally, I prefer the special property approach. That would also be a good way to encode labels > The decision to use the YAML anchors and references was more borne out of a desire > to sort of match conceptually the labels and references of DTS. It’s not a big > issue to switch to something different. I think it would be a good idea. One reason for defining a YAML encoding is to be able to use existing tools to work with the data. Changing the meaning of anchors and aliases breaks those tools immediately. > If we were to force YAML/DT file to be regular YAML files parseable by everything > we’d have to rethink a lot of conventions that DT files currently use and I’m afraid > a machine translation from DTS to YAML might not be feasible then. > > OTOH moving to that model might make it possible to use YAML constructs that are not > mapped at all to DTS. For instance > > - foo: true > bar: “string” > > - foo: false > bar: “another-string” > > is not possible to be mapped to a DT node/property structure right now. I’m not following. Are you saying DT has no way to encode a list of nodes? What use case are you imagining? > >> > >> Plus, it would be useful to use normal YAML anchors/aliases for >> creating node templates. For example: >> >> serial-template: &acme-uart . # The anchor for the template >> compatible: acme,uart >> interrupt-parent: *intc >> >> root: >> serial@20000: >> <<: *acme-uart # Alias node merged into serial@20000 >> interrupts: 5 >> reg: [0x20000, 0x1000] >> serial@30000: >> <<: *acme-uart # Alias node merged into serial@30000 >> interrupts: 5 >> reg: [0x30000, 0x1000] >> > > Yes, I’m aware of this and I planned to talk to you about it :) > It can be a very powerful way to cut down the churn of DT right now. > > It’s not going to be a problem for yamldt to support it (perhaps > adding a new editing tags to allow more fine grained control of the > substitions). > >> Another problem with anchors/references is YAML seems to require the >> anchor to be defined before the reference, or at least that's what >> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding >> should be readily consumable by existing yaml implementations without >> having to do a lot of customization. >> > > I think this is only true for streamable YAML implementations; yamldt > can leave references unresolved until the emit phase (or even leave them > be as object files). it is possible (probable) that I’m just not familiar with how to change that behaviour in pyyaml and ruamel. However, the point is moot if we can agree to preserve pure YAML semantics. >> I'm slightly concerned about using & anchors for labels because it >> seems only one anchor can be defined per node, but DTC allows multiple >> labels for a single node. This might not be an issue in practice >> though. Another implementation issue related to using & anchors is the >> YAML spec defines them as an encoding artifact, and parsers can >> discard the anchor names after parsing the YAML structure, which is a >> problem if we use something like $name to reference an anchor. The >> solution might just be that labels need to go into a special property >> so they don't disappear from the data stream. >> > > I do support multiple labels to the same node in yamldt with the > following trick, that uses an empty tree. > > foo: bar: baz { frob; }; > > bar: &foo > frob: true > > bar: &bar > ~: ~ > > Special properties should work, albeit being a little bit cludgy. With special properties, it might look like this: baz: $labels: [foo, bar] frob: > >> There appears to be no place to put metadata. The root of the tree is >> the top level of the YAML structure. There isn't any provision for >> having a top level object to hold things like the memreserve map. We >> may need a namespace to use for special properties that aren't nodes >> or properties. >> > > I don’t think that’s going to be necessary. Metadata are artifacts that > only have meaning when emitting a serialized form like dtb. > > yamldt fully supports /memreserve/ by simply treating a top level > /memreserve/ property as special and constructing the memreserve > entries at the emit phase. > > so > > /memreserve/ 1000 10; > > is simply: > > /memreserve/: [ 1000 10 ] > > It’s a matter of definition what properties are encoded as metadata for the > serialized form. > >> The encoding differentiates between nodes and properties implicitly >> base on whether the contents are a map, or a scalar/list. This does >> mean any parser needs to do a bit more work and it may impact what can >> be done with validation (I'm not going to talk about validation in >> this email though. We'll talk next week.) > > Hmm, I’m not sure I follow this. Is that related to the metadata problem. It has to do with iterating over properties or nodes. Nodes and props are mixed together, and any code processing them has to explicitly check the type to differentiate between the two. The alternative would be to collect all properties under a single key: parent: $props: foo: hello bar: world child1: {} child2: {} > > BTW, a blog post explaining the rationale behind yamldt is going to come > up soon at our website, I’ll post the link when it does. Cool. I see it got posted, but didn’t get a chance to read it before I got on this airplane. Will try to read this evening. (BTW, I’m arriving at about 11:00 tonight) Cheers, g. -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-22 18:23 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-22 18:23 UTC (permalink / raw) To: Grant Likely Cc: Frank Rowand, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree, Linux Kernel Mailing List Hi Grant, > On Oct 22, 2017, at 19:54 , Grant Likely <grant.likely@secretlab.ca> wrote: > > On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou > <pantelis.antoniou@konsulko.com> wrote: >> Hi Grant, >> >>> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely@secretlab.ca> wrote: >>> >>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou >>> <pantelis.antoniou@konsulko.com> wrote: >>>> Hello again, >>>> >>>> Significant progress has been made on yamldt and is now capable of >>>> not only generating yaml from DTS source but also compiling DTS sources >>>> and being almost fully compatible with DTC. >>>> >>>> Compiling the kernel's DTBs using yamldt is as simple as using a >>>> DTC=yamldt. >>>> >>>> Error reporting is accurate and validation against a YAML based schema >>>> works as well. In a short while I will begin posting patches with >>>> fixes on bindings and DTS files in the kernel. >>>> >>>> Please try it on your platform and report if you encounter any problems. >>>> >>>> https://github.com/pantoniou/yamldt >>>> >>>> I am eagerly awaiting for your comments. >>> >>> Hi Pantelis, >>> >>> This is good work. I've played around with it and I'm looking forward >>> to chatting next week. >>> >> >> Thanks. I’m looking forward to it too. >> >>> One thing I've done is tried loading the output YAML files into >>> another YAML interpreter and the current encoding causes problems. >>> Specifically, in yamldt anchors/aliases are being used as a >>> replacement for labels/phandles, but that conflicts with the YAML data >>> model which defines a reference as a way to make a copy of the data >>> appear in another part of the tree. For example, for the following >>> snippit: >>> >>> intc: intc@10000 { >>> #interrupt-cells = <1>; >>> compatible = "acme,intc"; >>> reg = <0x10000 0x1000>; >>> gpio-controller; >>> }; >>> >>> serial@20000 { >>> compatible = "acme,uart"; >>> reg = <0x20000 0x1000>; >>> interrupt-parent = <&intc>; >>> interrupts = <5>; >>> }; >>> >>> yamldt will encode this as: >>> >>> intc@10000: &intc >>> "#interrupt-cells": 1 >>> compatible: acme,intc >>> reg: [0x10000, 0x1000] >>> gpio-controller: >>> >>> serial@20000: >>> compatible: acme,uart >>> reg: [0x20000, 0x1000] >>> interrupt-parent: *intc >>> interrupts: 5 >>> >>> But, the expected behaviour for a YAML parser is expand the alias >>> '*intc' which results in the following structure: >>> >>> intc@10000: &intc >>> "#interrupt-cells": 1 >>> compatible: acme,intc >>> reg: [0x10000, 0x1000] >>> gpio-controller: >>> >>> serial@20000: >>> compatible: acme,uart >>> reg: [0x20000, 0x1000] >>> interrupt-parent: >>> "#interrupt-cells": 1 >>> compatible: acme,intc >>> reg: [0x10000, 0x1000] >>> gpio-controller: >>> interrupts: 5 >>> >>> See? It results in the entire interrupt controller node to appear as >>> an instance under the interrupt-parent property, when the intention is >>> only to create a phandle. yamldt should not redefine the behaviour of >>> '*' aliases. Instead, it should use a different indicator, either >>> using an explicit !phandle tag, or by replacing '*' with something >>> else. I worked around it in my tests by replacing '*' with '$’. >> >> Yes. This is expected. I don’t think pure YAML form is a good match for all >> the crazy things that are possible (and actually used in practice) with DTS. > > I don’t think it is as dire as that. The DTS structure is not complex > and I think can easily be mapped into pure YAML. But, it does require > treating the DTS structure separately from its semantic meaning. For > example, the grammar of nodes, properties and labels easily maps to > pure YAML, but phandles and overlay trees have semantic meaning that > must be resolved by DT specific code. I’ll respond to you’re specific > examples below… > We are in complete agreement here. Single nodes and properties map to YAML perfectly. It’s the complex way that we build the complete DTB that stress the YAML structures. >> >> For instance there’s no way a normal YAML parser won’t choke with something like >> >> / { >> foo; >> }; >> >> / { >> bar; >> }; >> >> Which is a common idiom in DTS files. > > That’s only true when the top level of nodes is encoded as a map, but > it shouldn’t be. It should be a list instead, for two reasons. First, > order matters for the top level. Subsequent top level trees have to be > applied in order to fully resolve the tree, but a map doesn’t preserve > the ordering. Second, as you rightly point out, the same name can be > used for multiple top level trees. So, the encoding should be a list > with each list entry containing the node path {/path,&ref}, and the > node data. One possible way to do this is with a tuple: > > treedata: > - - / > - foo: > - - / > - bar: > > Another option is to add a special property in the node to contain the name: > > treedata: > - $path: / > foo: > - $path: / > bar: > > Personally, I prefer the special property approach. That would also be > a good way to encode labels > The YAML bare sequence is problematic; I’ll explain below. >> The decision to use the YAML anchors and references was more borne out of a desire >> to sort of match conceptually the labels and references of DTS. It’s not a big >> issue to switch to something different. > > I think it would be a good idea. One reason for defining a YAML > encoding is to be able to use existing tools to work with the data. > Changing the meaning of anchors and aliases breaks those tools > immediately. > Hmm, I see what you mean. Semantically this mapping is more correct but it does present problems to non-aware tools. >> If we were to force YAML/DT file to be regular YAML files parseable by everything >> we’d have to rethink a lot of conventions that DT files currently use and I’m afraid >> a machine translation from DTS to YAML might not be feasible then. >> >> OTOH moving to that model might make it possible to use YAML constructs that are not >> mapped at all to DTS. For instance >> >> - foo: true >> bar: “string” >> >> - foo: false >> bar: “another-string” >> >> is not possible to be mapped to a DT node/property structure right now. > > I’m not following. Are you saying DT has no way to encode a list of > nodes? What use case are you imagining? > This threw me off too at first. The problem is when you try to convert this into a live tree structure. The YAML encoding is +SEQ +MAP =VAL :foo =VAL :true =VAL :bar =VAL "string -MAP +MAP =VAL :foo =VAL :false =VAL :bar =VAL "another-string -MAP -SEQ So it’s a sequence of MAPs. In a live tree DTS form would be noname-property = { foo=true; bar = “string”; }, { foo = false; bar=“another-string”; }; We don’t have the capability as right now to express something like this. Namely properties containing nodes, and the root of the live tree not being a node. >> >>> >> >>> Plus, it would be useful to use normal YAML anchors/aliases for >>> creating node templates. For example: >>> >>> serial-template: &acme-uart . # The anchor for the template >>> compatible: acme,uart >>> interrupt-parent: *intc >>> >>> root: >>> serial@20000: >>> <<: *acme-uart # Alias node merged into serial@20000 >>> interrupts: 5 >>> reg: [0x20000, 0x1000] >>> serial@30000: >>> <<: *acme-uart # Alias node merged into serial@30000 >>> interrupts: 5 >>> reg: [0x30000, 0x1000] >>> >> >> Yes, I’m aware of this and I planned to talk to you about it :) >> It can be a very powerful way to cut down the churn of DT right now. >> >> It’s not going to be a problem for yamldt to support it (perhaps >> adding a new editing tags to allow more fine grained control of the >> substitions). >> >>> Another problem with anchors/references is YAML seems to require the >>> anchor to be defined before the reference, or at least that's what >>> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding >>> should be readily consumable by existing yaml implementations without >>> having to do a lot of customization. >>> >> >> I think this is only true for streamable YAML implementations; yamldt >> can leave references unresolved until the emit phase (or even leave them >> be as object files). > > it is possible (probable) that I’m just not familiar with how to > change that behaviour in pyyaml and ruamel. However, the point is moot > if we can agree to preserve pure YAML semantics. > Agreed. >>> I'm slightly concerned about using & anchors for labels because it >>> seems only one anchor can be defined per node, but DTC allows multiple >>> labels for a single node. This might not be an issue in practice >>> though. Another implementation issue related to using & anchors is the >>> YAML spec defines them as an encoding artifact, and parsers can >>> discard the anchor names after parsing the YAML structure, which is a >>> problem if we use something like $name to reference an anchor. The >>> solution might just be that labels need to go into a special property >>> so they don't disappear from the data stream. >>> >> >> I do support multiple labels to the same node in yamldt with the >> following trick, that uses an empty tree. >> >> foo: bar: baz { frob; }; >> >> bar: &foo >> frob: true >> >> bar: &bar >> ~: ~ >> >> Special properties should work, albeit being a little bit cludgy. > > With special properties, it might look like this: > > baz: > $labels: [foo, bar] > frob: > Yeah, that would work. >> >>> There appears to be no place to put metadata. The root of the tree is >>> the top level of the YAML structure. There isn't any provision for >>> having a top level object to hold things like the memreserve map. We >>> may need a namespace to use for special properties that aren't nodes >>> or properties. >>> >> >> I don’t think that’s going to be necessary. Metadata are artifacts that >> only have meaning when emitting a serialized form like dtb. >> >> yamldt fully supports /memreserve/ by simply treating a top level >> /memreserve/ property as special and constructing the memreserve >> entries at the emit phase. >> >> so >> >> /memreserve/ 1000 10; >> >> is simply: >> >> /memreserve/: [ 1000 10 ] >> >> It’s a matter of definition what properties are encoded as metadata for the >> serialized form. >> >>> The encoding differentiates between nodes and properties implicitly >>> base on whether the contents are a map, or a scalar/list. This does >>> mean any parser needs to do a bit more work and it may impact what can >>> be done with validation (I'm not going to talk about validation in >>> this email though. We'll talk next week.) >> >> Hmm, I’m not sure I follow this. Is that related to the metadata problem. > > It has to do with iterating over properties or nodes. Nodes and props > are mixed together, and any code processing them has to explicitly > check the type to differentiate between the two. The alternative would > be to collect all properties under a single key: > > parent: > $props: > foo: hello > bar: world > child1: {} > child2: {} > That’s no problem. It does look weird though. I am using the same parser for parsing bindings and this might make the files a little bit weird. Keep in mind that we have two YAML formats to parse; the hardware description and the bindings describing them. >> >> BTW, a blog post explaining the rationale behind yamldt is going to come >> up soon at our website, I’ll post the link when it does. > > Cool. I see it got posted, but didn’t get a chance to read it before I > got on this airplane. Will try to read this evening. > > (BTW, I’m arriving at about 11:00 tonight) > Have a safe flight, you’ll get to beat me up tomorrow :) > Cheers, > g. Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-22 18:23 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-22 18:23 UTC (permalink / raw) To: Grant Likely Cc: Frank Rowand, David Gibson, Tom Rini, Rob Herring, Franklin S Cooper Jr, Matt Porter, Simon Glass, Phil Elwell, Geert Uytterhoeven, Marek Vasut, Devicetree Compiler, devicetree-u79uwXL29TY76Z2rM5mHXA, Linux Kernel Mailing List Hi Grant, > On Oct 22, 2017, at 19:54 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote: > > On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou > <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >> Hi Grant, >> >>> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote: >>> >>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou >>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >>>> Hello again, >>>> >>>> Significant progress has been made on yamldt and is now capable of >>>> not only generating yaml from DTS source but also compiling DTS sources >>>> and being almost fully compatible with DTC. >>>> >>>> Compiling the kernel's DTBs using yamldt is as simple as using a >>>> DTC=yamldt. >>>> >>>> Error reporting is accurate and validation against a YAML based schema >>>> works as well. In a short while I will begin posting patches with >>>> fixes on bindings and DTS files in the kernel. >>>> >>>> Please try it on your platform and report if you encounter any problems. >>>> >>>> https://github.com/pantoniou/yamldt >>>> >>>> I am eagerly awaiting for your comments. >>> >>> Hi Pantelis, >>> >>> This is good work. I've played around with it and I'm looking forward >>> to chatting next week. >>> >> >> Thanks. I’m looking forward to it too. >> >>> One thing I've done is tried loading the output YAML files into >>> another YAML interpreter and the current encoding causes problems. >>> Specifically, in yamldt anchors/aliases are being used as a >>> replacement for labels/phandles, but that conflicts with the YAML data >>> model which defines a reference as a way to make a copy of the data >>> appear in another part of the tree. For example, for the following >>> snippit: >>> >>> intc: intc@10000 { >>> #interrupt-cells = <1>; >>> compatible = "acme,intc"; >>> reg = <0x10000 0x1000>; >>> gpio-controller; >>> }; >>> >>> serial@20000 { >>> compatible = "acme,uart"; >>> reg = <0x20000 0x1000>; >>> interrupt-parent = <&intc>; >>> interrupts = <5>; >>> }; >>> >>> yamldt will encode this as: >>> >>> intc@10000: &intc >>> "#interrupt-cells": 1 >>> compatible: acme,intc >>> reg: [0x10000, 0x1000] >>> gpio-controller: >>> >>> serial@20000: >>> compatible: acme,uart >>> reg: [0x20000, 0x1000] >>> interrupt-parent: *intc >>> interrupts: 5 >>> >>> But, the expected behaviour for a YAML parser is expand the alias >>> '*intc' which results in the following structure: >>> >>> intc@10000: &intc >>> "#interrupt-cells": 1 >>> compatible: acme,intc >>> reg: [0x10000, 0x1000] >>> gpio-controller: >>> >>> serial@20000: >>> compatible: acme,uart >>> reg: [0x20000, 0x1000] >>> interrupt-parent: >>> "#interrupt-cells": 1 >>> compatible: acme,intc >>> reg: [0x10000, 0x1000] >>> gpio-controller: >>> interrupts: 5 >>> >>> See? It results in the entire interrupt controller node to appear as >>> an instance under the interrupt-parent property, when the intention is >>> only to create a phandle. yamldt should not redefine the behaviour of >>> '*' aliases. Instead, it should use a different indicator, either >>> using an explicit !phandle tag, or by replacing '*' with something >>> else. I worked around it in my tests by replacing '*' with '$’. >> >> Yes. This is expected. I don’t think pure YAML form is a good match for all >> the crazy things that are possible (and actually used in practice) with DTS. > > I don’t think it is as dire as that. The DTS structure is not complex > and I think can easily be mapped into pure YAML. But, it does require > treating the DTS structure separately from its semantic meaning. For > example, the grammar of nodes, properties and labels easily maps to > pure YAML, but phandles and overlay trees have semantic meaning that > must be resolved by DT specific code. I’ll respond to you’re specific > examples below… > We are in complete agreement here. Single nodes and properties map to YAML perfectly. It’s the complex way that we build the complete DTB that stress the YAML structures. >> >> For instance there’s no way a normal YAML parser won’t choke with something like >> >> / { >> foo; >> }; >> >> / { >> bar; >> }; >> >> Which is a common idiom in DTS files. > > That’s only true when the top level of nodes is encoded as a map, but > it shouldn’t be. It should be a list instead, for two reasons. First, > order matters for the top level. Subsequent top level trees have to be > applied in order to fully resolve the tree, but a map doesn’t preserve > the ordering. Second, as you rightly point out, the same name can be > used for multiple top level trees. So, the encoding should be a list > with each list entry containing the node path {/path,&ref}, and the > node data. One possible way to do this is with a tuple: > > treedata: > - - / > - foo: > - - / > - bar: > > Another option is to add a special property in the node to contain the name: > > treedata: > - $path: / > foo: > - $path: / > bar: > > Personally, I prefer the special property approach. That would also be > a good way to encode labels > The YAML bare sequence is problematic; I’ll explain below. >> The decision to use the YAML anchors and references was more borne out of a desire >> to sort of match conceptually the labels and references of DTS. It’s not a big >> issue to switch to something different. > > I think it would be a good idea. One reason for defining a YAML > encoding is to be able to use existing tools to work with the data. > Changing the meaning of anchors and aliases breaks those tools > immediately. > Hmm, I see what you mean. Semantically this mapping is more correct but it does present problems to non-aware tools. >> If we were to force YAML/DT file to be regular YAML files parseable by everything >> we’d have to rethink a lot of conventions that DT files currently use and I’m afraid >> a machine translation from DTS to YAML might not be feasible then. >> >> OTOH moving to that model might make it possible to use YAML constructs that are not >> mapped at all to DTS. For instance >> >> - foo: true >> bar: “string” >> >> - foo: false >> bar: “another-string” >> >> is not possible to be mapped to a DT node/property structure right now. > > I’m not following. Are you saying DT has no way to encode a list of > nodes? What use case are you imagining? > This threw me off too at first. The problem is when you try to convert this into a live tree structure. The YAML encoding is +SEQ +MAP =VAL :foo =VAL :true =VAL :bar =VAL "string -MAP +MAP =VAL :foo =VAL :false =VAL :bar =VAL "another-string -MAP -SEQ So it’s a sequence of MAPs. In a live tree DTS form would be noname-property = { foo=true; bar = “string”; }, { foo = false; bar=“another-string”; }; We don’t have the capability as right now to express something like this. Namely properties containing nodes, and the root of the live tree not being a node. >> >>> >> >>> Plus, it would be useful to use normal YAML anchors/aliases for >>> creating node templates. For example: >>> >>> serial-template: &acme-uart . # The anchor for the template >>> compatible: acme,uart >>> interrupt-parent: *intc >>> >>> root: >>> serial@20000: >>> <<: *acme-uart # Alias node merged into serial@20000 >>> interrupts: 5 >>> reg: [0x20000, 0x1000] >>> serial@30000: >>> <<: *acme-uart # Alias node merged into serial@30000 >>> interrupts: 5 >>> reg: [0x30000, 0x1000] >>> >> >> Yes, I’m aware of this and I planned to talk to you about it :) >> It can be a very powerful way to cut down the churn of DT right now. >> >> It’s not going to be a problem for yamldt to support it (perhaps >> adding a new editing tags to allow more fine grained control of the >> substitions). >> >>> Another problem with anchors/references is YAML seems to require the >>> anchor to be defined before the reference, or at least that's what >>> pyyaml and ruamel both expect. Regardless, The chosen YAML encoding >>> should be readily consumable by existing yaml implementations without >>> having to do a lot of customization. >>> >> >> I think this is only true for streamable YAML implementations; yamldt >> can leave references unresolved until the emit phase (or even leave them >> be as object files). > > it is possible (probable) that I’m just not familiar with how to > change that behaviour in pyyaml and ruamel. However, the point is moot > if we can agree to preserve pure YAML semantics. > Agreed. >>> I'm slightly concerned about using & anchors for labels because it >>> seems only one anchor can be defined per node, but DTC allows multiple >>> labels for a single node. This might not be an issue in practice >>> though. Another implementation issue related to using & anchors is the >>> YAML spec defines them as an encoding artifact, and parsers can >>> discard the anchor names after parsing the YAML structure, which is a >>> problem if we use something like $name to reference an anchor. The >>> solution might just be that labels need to go into a special property >>> so they don't disappear from the data stream. >>> >> >> I do support multiple labels to the same node in yamldt with the >> following trick, that uses an empty tree. >> >> foo: bar: baz { frob; }; >> >> bar: &foo >> frob: true >> >> bar: &bar >> ~: ~ >> >> Special properties should work, albeit being a little bit cludgy. > > With special properties, it might look like this: > > baz: > $labels: [foo, bar] > frob: > Yeah, that would work. >> >>> There appears to be no place to put metadata. The root of the tree is >>> the top level of the YAML structure. There isn't any provision for >>> having a top level object to hold things like the memreserve map. We >>> may need a namespace to use for special properties that aren't nodes >>> or properties. >>> >> >> I don’t think that’s going to be necessary. Metadata are artifacts that >> only have meaning when emitting a serialized form like dtb. >> >> yamldt fully supports /memreserve/ by simply treating a top level >> /memreserve/ property as special and constructing the memreserve >> entries at the emit phase. >> >> so >> >> /memreserve/ 1000 10; >> >> is simply: >> >> /memreserve/: [ 1000 10 ] >> >> It’s a matter of definition what properties are encoded as metadata for the >> serialized form. >> >>> The encoding differentiates between nodes and properties implicitly >>> base on whether the contents are a map, or a scalar/list. This does >>> mean any parser needs to do a bit more work and it may impact what can >>> be done with validation (I'm not going to talk about validation in >>> this email though. We'll talk next week.) >> >> Hmm, I’m not sure I follow this. Is that related to the metadata problem. > > It has to do with iterating over properties or nodes. Nodes and props > are mixed together, and any code processing them has to explicitly > check the type to differentiate between the two. The alternative would > be to collect all properties under a single key: > > parent: > $props: > foo: hello > bar: world > child1: {} > child2: {} > That’s no problem. It does look weird though. I am using the same parser for parsing bindings and this might make the files a little bit weird. Keep in mind that we have two YAML formats to parse; the hardware description and the bindings describing them. >> >> BTW, a blog post explaining the rationale behind yamldt is going to come >> up soon at our website, I’ll post the link when it does. > > Cool. I see it got posted, but didn’t get a chance to read it before I > got on this airplane. Will try to read this evening. > > (BTW, I’m arriving at about 11:00 tonight) > Have a safe flight, you’ll get to beat me up tomorrow :) > Cheers, > g. Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-22 19:13 ` Grant Likely 0 siblings, 0 replies; 40+ messages in thread From: Grant Likely @ 2017-10-22 19:13 UTC (permalink / raw) To: Pantelis Antoniou Cc: David Gibson, Devicetree Compiler, Frank Rowand, Franklin S Cooper Jr, Geert Uytterhoeven, Linux Kernel Mailing List, Marek Vasut, Matt Porter, Phil Elwell, Rob Herring, Simon Glass, Tom Rini, devicetree On Sun, Oct 22, 2017 at 7:23 PM, Pantelis Antoniou <pantelis.antoniou@konsulko.com> wrote: > Hi Grant, > >> On Oct 22, 2017, at 19:54 , Grant Likely <grant.likely@secretlab.ca> wrote: >> >> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou >> <pantelis.antoniou@konsulko.com> wrote: >>> Hi Grant, >>> >>>> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely@secretlab.ca> wrote: >>>> >>>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou >>>> <pantelis.antoniou@konsulko.com> wrote: [...] >>> Yes. This is expected. I don’t think pure YAML form is a good match for all >>> the crazy things that are possible (and actually used in practice) with DTS. >> >> I don’t think it is as dire as that. The DTS structure is not complex >> and I think can easily be mapped into pure YAML. But, it does require >> treating the DTS structure separately from its semantic meaning. For >> example, the grammar of nodes, properties and labels easily maps to >> pure YAML, but phandles and overlay trees have semantic meaning that >> must be resolved by DT specific code. I’ll respond to you’re specific >> examples below… >> > > We are in complete agreement here. Single nodes and properties map to YAML perfectly. > It’s the complex way that we build the complete DTB that stress the YAML structures. Okay, good. >>> If we were to force YAML/DT file to be regular YAML files parseable by everything >>> we’d have to rethink a lot of conventions that DT files currently use and I’m afraid >>> a machine translation from DTS to YAML might not be feasible then. >>> >>> OTOH moving to that model might make it possible to use YAML constructs that are not >>> mapped at all to DTS. For instance >>> >>> - foo: true >>> bar: “string” >>> >>> - foo: false >>> bar: “another-string” >>> >>> is not possible to be mapped to a DT node/property structure right now. >> >> I’m not following. Are you saying DT has no way to encode a list of >> nodes? What use case are you imagining? >> > > This threw me off too at first. > > The problem is when you try to convert this into a live tree structure. > > The YAML encoding is > > +SEQ > +MAP > =VAL :foo > =VAL :true > =VAL :bar > =VAL "string > -MAP > +MAP > =VAL :foo > =VAL :false > =VAL :bar > =VAL "another-string > -MAP > -SEQ > > So it’s a sequence of MAPs. > > In a live tree DTS form would be > > noname-property = { foo=true; bar = “string”; }, { foo = false; bar=“another-string”; }; > > We don’t have the capability as right now to express something like this. > > Namely properties containing nodes, and the root of the live tree not being a node. Ah, so you're proposing property data that encodes key/value pairs, correct? Effectively adding another layer of namespace under a property, but that namespace is invisible to and generic DT parsing code in an OS (for how DT is defined today). I would straight out make that structure illegal in the YAML DT metaschema until such time as the semantics are defined. For now properties should only be allowed to contain scalars, or lists of scalars/lists which can be nested. [...] >>>> The encoding differentiates between nodes and properties implicitly >>>> base on whether the contents are a map, or a scalar/list. This does >>>> mean any parser needs to do a bit more work and it may impact what can >>>> be done with validation (I'm not going to talk about validation in >>>> this email though. We'll talk next week.) >>> >>> Hmm, I’m not sure I follow this. Is that related to the metadata problem. >> >> It has to do with iterating over properties or nodes. Nodes and props >> are mixed together, and any code processing them has to explicitly >> check the type to differentiate between the two. The alternative would >> be to collect all properties under a single key: >> >> parent: >> $props: >> foo: hello >> bar: world >> child1: {} >> child2: {} >> > > That’s no problem. It does look weird though. I am using the same parser > for parsing bindings and this might make the files a little bit weird. That it does. I don’t feel strongly about either approach, but I want to see how the different encodings affect doing the schema bits. > > Keep in mind that we have two YAML formats to parse; the hardware description > and the bindings describing them. Three! Hw description, schema & metaschema. :-) But metaschema should be a normal jsonschema file. > >>> >>> BTW, a blog post explaining the rationale behind yamldt is going to come >>> up soon at our website, I’ll post the link when it does. >> >> Cool. I see it got posted, but didn’t get a chance to read it before I >> got on this airplane. Will try to read this evening. >> >> (BTW, I’m arriving at about 11:00 tonight) >> > > Have a safe flight, you’ll get to beat me up tomorrow :) See you tomorrow. It will be good to catch up. g. ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-22 19:13 ` Grant Likely 0 siblings, 0 replies; 40+ messages in thread From: Grant Likely @ 2017-10-22 19:13 UTC (permalink / raw) To: Pantelis Antoniou Cc: David Gibson, Devicetree Compiler, Frank Rowand, Franklin S Cooper Jr, Geert Uytterhoeven, Linux Kernel Mailing List, Marek Vasut, Matt Porter, Phil Elwell, Rob Herring, Simon Glass, Tom Rini, devicetree-u79uwXL29TY76Z2rM5mHXA On Sun, Oct 22, 2017 at 7:23 PM, Pantelis Antoniou <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: > Hi Grant, > >> On Oct 22, 2017, at 19:54 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote: >> >> On Fri, Oct 20, 2017 at 8:16 PM, Pantelis Antoniou >> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: >>> Hi Grant, >>> >>>> On Oct 20, 2017, at 20:46 , Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote: >>>> >>>> On Thu, Sep 28, 2017 at 8:58 PM, Pantelis Antoniou >>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote: [...] >>> Yes. This is expected. I don’t think pure YAML form is a good match for all >>> the crazy things that are possible (and actually used in practice) with DTS. >> >> I don’t think it is as dire as that. The DTS structure is not complex >> and I think can easily be mapped into pure YAML. But, it does require >> treating the DTS structure separately from its semantic meaning. For >> example, the grammar of nodes, properties and labels easily maps to >> pure YAML, but phandles and overlay trees have semantic meaning that >> must be resolved by DT specific code. I’ll respond to you’re specific >> examples below… >> > > We are in complete agreement here. Single nodes and properties map to YAML perfectly. > It’s the complex way that we build the complete DTB that stress the YAML structures. Okay, good. >>> If we were to force YAML/DT file to be regular YAML files parseable by everything >>> we’d have to rethink a lot of conventions that DT files currently use and I’m afraid >>> a machine translation from DTS to YAML might not be feasible then. >>> >>> OTOH moving to that model might make it possible to use YAML constructs that are not >>> mapped at all to DTS. For instance >>> >>> - foo: true >>> bar: “string” >>> >>> - foo: false >>> bar: “another-string” >>> >>> is not possible to be mapped to a DT node/property structure right now. >> >> I’m not following. Are you saying DT has no way to encode a list of >> nodes? What use case are you imagining? >> > > This threw me off too at first. > > The problem is when you try to convert this into a live tree structure. > > The YAML encoding is > > +SEQ > +MAP > =VAL :foo > =VAL :true > =VAL :bar > =VAL "string > -MAP > +MAP > =VAL :foo > =VAL :false > =VAL :bar > =VAL "another-string > -MAP > -SEQ > > So it’s a sequence of MAPs. > > In a live tree DTS form would be > > noname-property = { foo=true; bar = “string”; }, { foo = false; bar=“another-string”; }; > > We don’t have the capability as right now to express something like this. > > Namely properties containing nodes, and the root of the live tree not being a node. Ah, so you're proposing property data that encodes key/value pairs, correct? Effectively adding another layer of namespace under a property, but that namespace is invisible to and generic DT parsing code in an OS (for how DT is defined today). I would straight out make that structure illegal in the YAML DT metaschema until such time as the semantics are defined. For now properties should only be allowed to contain scalars, or lists of scalars/lists which can be nested. [...] >>>> The encoding differentiates between nodes and properties implicitly >>>> base on whether the contents are a map, or a scalar/list. This does >>>> mean any parser needs to do a bit more work and it may impact what can >>>> be done with validation (I'm not going to talk about validation in >>>> this email though. We'll talk next week.) >>> >>> Hmm, I’m not sure I follow this. Is that related to the metadata problem. >> >> It has to do with iterating over properties or nodes. Nodes and props >> are mixed together, and any code processing them has to explicitly >> check the type to differentiate between the two. The alternative would >> be to collect all properties under a single key: >> >> parent: >> $props: >> foo: hello >> bar: world >> child1: {} >> child2: {} >> > > That’s no problem. It does look weird though. I am using the same parser > for parsing bindings and this might make the files a little bit weird. That it does. I don’t feel strongly about either approach, but I want to see how the different encodings affect doing the schema bits. > > Keep in mind that we have two YAML formats to parse; the hardware description > and the bindings describing them. Three! Hw description, schema & metaschema. :-) But metaschema should be a normal jsonschema file. > >>> >>> BTW, a blog post explaining the rationale behind yamldt is going to come >>> up soon at our website, I’ll post the link when it does. >> >> Cool. I see it got posted, but didn’t get a chance to read it before I >> got on this airplane. Will try to read this evening. >> >> (BTW, I’m arriving at about 11:00 tonight) >> > > Have a safe flight, you’ll get to beat me up tomorrow :) See you tomorrow. It will be good to catch up. g. -- To unsubscribe from this list: send the line "unsubscribe devicetree" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-23 10:08 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-23 10:08 UTC (permalink / raw) To: Grant Likely Cc: David Gibson, Devicetree Compiler, Frank Rowand, Franklin S Cooper Jr, Geert Uytterhoeven, Linux Kernel Mailing List, Marek Vasut, Matt Porter, Phil Elwell, Rob Herring, Simon Glass, Tom Rini, devicetree Hi all, I’ve just put up a blog-entry explaining the rationale behind yaml and device tree. https://www.konsulko.com/yaml-and-device-tree/ Please give it a quick glance and give feedback if you can. Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
* Re: [RFC] yamldt v0.5, now a DTS compiler too @ 2017-10-23 10:08 ` Pantelis Antoniou 0 siblings, 0 replies; 40+ messages in thread From: Pantelis Antoniou @ 2017-10-23 10:08 UTC (permalink / raw) To: Grant Likely Cc: David Gibson, Devicetree Compiler, Frank Rowand, Franklin S Cooper Jr, Geert Uytterhoeven, Linux Kernel Mailing List, Marek Vasut, Matt Porter, Phil Elwell, Rob Herring, Simon Glass, Tom Rini, devicetree-u79uwXL29TY76Z2rM5mHXA Hi all, I’ve just put up a blog-entry explaining the rationale behind yaml and device tree. https://www.konsulko.com/yaml-and-device-tree/ Please give it a quick glance and give feedback if you can. Regards — Pantelis ^ permalink raw reply [flat|nested] 40+ messages in thread
end of thread, other threads:[~2017-10-23 10:08 UTC | newest] Thread overview: 40+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-09-28 19:58 [RFC] yamldt v0.5, now a DTS compiler too Pantelis Antoniou 2017-09-28 19:58 ` Pantelis Antoniou 2017-10-01 22:00 ` Rob Herring 2017-10-01 22:00 ` Rob Herring 2017-10-02 7:36 ` Pantelis Antoniou 2017-10-02 19:46 ` Pantelis Antoniou 2017-10-03 7:17 ` Geert Uytterhoeven 2017-10-03 7:33 ` Pantelis Antoniou 2017-10-03 13:18 ` Rob Herring 2017-10-03 14:13 ` Pantelis Antoniou 2017-10-03 14:13 ` Pantelis Antoniou 2017-10-03 17:13 ` Rob Herring 2017-10-03 17:39 ` Pantelis Antoniou 2017-10-03 17:39 ` Pantelis Antoniou 2017-10-06 13:55 ` Rob Herring 2017-10-06 13:55 ` Rob Herring 2017-10-07 10:23 ` Pantelis Antoniou 2017-10-08 23:08 ` Frank Rowand 2017-10-08 23:08 ` Frank Rowand 2017-10-09 0:00 ` David Gibson 2017-10-09 0:00 ` David Gibson 2017-10-09 15:07 ` Pantelis Antoniou 2017-10-09 15:07 ` Pantelis Antoniou 2017-10-10 1:50 ` David Gibson 2017-10-10 15:19 ` Pantelis Antoniou 2017-10-10 15:19 ` Pantelis Antoniou 2017-10-11 3:49 ` David Gibson 2017-10-11 3:49 ` David Gibson 2017-10-09 14:59 ` Pantelis Antoniou 2017-10-20 17:46 ` Grant Likely 2017-10-20 17:46 ` Grant Likely 2017-10-20 19:16 ` Pantelis Antoniou 2017-10-22 17:54 ` Grant Likely 2017-10-22 17:54 ` Grant Likely 2017-10-22 18:23 ` Pantelis Antoniou 2017-10-22 18:23 ` Pantelis Antoniou 2017-10-22 19:13 ` Grant Likely 2017-10-22 19:13 ` Grant Likely 2017-10-23 10:08 ` Pantelis Antoniou 2017-10-23 10:08 ` Pantelis Antoniou
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.