Next steps for schema language

Next steps for schema language

[Grant Likely]

Hi Pantelis and Rob,

After the workshop next week, I'm trying to capture the direction
we're going for the schema format. Roughly I think we're aiming
towards:

- Schema files to be written in YAML
- DT files shall remain written in DTS for the foreseeable future.
YAML will be treated as an intermediary format
  - That said, we'll try to make design decisions that allow YAML to
be used as a source format.
- All schema files and yaml-encoded-dt files must be parsable by stock
YAML parsers
- Schema files to use the jsonschema vocabulary
  - (jsonschema assumes json files, but YAML is a superset so this will be okay)
  - Extended to add vocabulary for DT concepts (ignored by stock validators)
    - C-like expressions as used in Pantelis' yamldt could be added in this way
  - Need to write a jsonschema "metaschema" do define DT specific extensions
    - metaschema will be used to validate format of schema files
    - Existing tools can confirm is schema files are in the right format.
    - will make review a lot easier.

The yaml encoding produced by yamldt is a good start, but some changes
need to be made from the current code to be workable:
- The redefinition of the anchor/alias syntax needs to be dropped.
  - We need a labels/reference syntax instead.
  - I'm using a $labels property to contain a list of labels which is
working for me, but I'm open to other suggestions
  - A YAML style !phandle or !path type definition would work for
parsing references.
- At the top level, the yaml-encoded-dt needs to be structured as a
list, not a map.
  - Using a list will properly accounts for how multiple top level
trees are overlayed to create the final tree.
  - I'm using a $path property to encode the path of each top level
node. Again, I'm open to suggestions for a different approach

Pantelis, do you think you can rework yamldt to use that encoding?

I'm currently working on what is needed for a metaschema, and an
engine for validating device-specific schema files using a standard
jsonschema validation library. I'll post a git tree once I've got
something worth looking at.

Cheers,
g.

Re: Next steps for schema language

[Pantelis Antoniou]

Hi Grant,

Mostly agree, some notes below.

On Thu, 2017-11-02 at 16:44 +0000, Grant Likely wrote:
> Hi Pantelis and Rob,
> 
> After the workshop next week, I'm trying to capture the direction
> we're going for the schema format. Roughly I think we're aiming
> towards:
> 
> - Schema files to be written in YAML
> - DT files shall remain written in DTS for the foreseeable future.
> YAML will be treated as an intermediary format
>   - That said, we'll try to make design decisions that allow YAML to
> be used as a source format.
> - All schema files and yaml-encoded-dt files must be parsable by stock
> YAML parsers
> - Schema files to use the jsonschema vocabulary
>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
>     - C-like expressions as used in Pantelis' yamldt could be added in this way
>   - Need to write a jsonschema "metaschema" do define DT specific extensions
>     - metaschema will be used to validate format of schema files
>     - Existing tools can confirm is schema files are in the right format.
>     - will make review a lot easier.
> 
> The yaml encoding produced by yamldt is a good start, but some changes
> need to be made from the current code to be workable:
> - The redefinition of the anchor/alias syntax needs to be dropped.
>   - We need a labels/reference syntax instead.
>   - I'm using a $labels property to contain a list of labels which is
> working for me, but I'm open to other suggestions

I'm working on supporting just that. Should be ready for testing in a
couple of days. Do note that I think we should keep the old '*' usage
for supporting the binding files that are YAML.

>   - A YAML style !phandle or !path type definition would work for
> parsing references.

!phandle is a bad name IMO. It assumes that the implementation is going
to always be a cell integer containing a phandle. I think !ref is
better. Note that we must come with a way to encode types in JSON, since
they are not natively supported.

> - At the top level, the yaml-encoded-dt needs to be structured as a
> list, not a map.
>   - Using a list will properly accounts for how multiple top level
> trees are overlayed to create the final tree.


This is true only for non-resolved files. Resolved files are going to
be guaranteed a map. Does this cause problems for your validator?

>   - I'm using a $path property to encode the path of each top level
> node. Again, I'm open to suggestions for a different approach
> 

A bare scalar that starts with / can always be deduced to be a path.
Can you share some examples about your usage?

> Pantelis, do you think you can rework yamldt to use that encoding?
> 

Yes, it's no big problem.

> I'm currently working on what is needed for a metaschema, and an
> engine for validating device-specific schema files using a standard
> jsonschema validation library. I'll post a git tree once I've got
> something worth looking at.
> 

Please do.

> Cheers,
> g.

Regards

-- Pantelis

Re: Next steps for schema language

[Rob Herring]

On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> Hi Pantelis and Rob,
>
> After the workshop next week, I'm trying to capture the direction
> we're going for the schema format. Roughly I think we're aiming
> towards:
>
> - Schema files to be written in YAML
> - DT files shall remain written in DTS for the foreseeable future.
> YAML will be treated as an intermediary format
>   - That said, we'll try to make design decisions that allow YAML to
> be used as a source format.
> - All schema files and yaml-encoded-dt files must be parsable by stock
> YAML parsers
> - Schema files to use the jsonschema vocabulary
>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
>     - C-like expressions as used in Pantelis' yamldt could be added in this way
>   - Need to write a jsonschema "metaschema" do define DT specific extensions
>     - metaschema will be used to validate format of schema files
>     - Existing tools can confirm is schema files are in the right format.
>     - will make review a lot easier.

I want to start small here with defining top-level board/soc bindings.
This is essentially just defining the root node compatible strings.
Seems easy enough, right? However, I quickly run into the problem of
how to match for when to apply the schema. "compatible" is the obvious
choice, but that's also what I'm checking. We can't key off of what we
are validating. So we really need 2 schema. The first is for matching
on any valid compatible for board, then 2nd is checking for valid
combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
don't like that as we'd be listing compatibles twice. An alternative
would be we apply every board schema and exactly 1 must pass. Perhaps
we generate a schema that's a "oneOf" of all the boards? Then we just
need to tag board schemas in some way.

I'm thinking of this very specific case, but no doubt it is going to
apply in other places. This has been one of the harder problems with
writing dtc checks. What to key off of to trigger checks and how to
check errors in the key itself.

Rob

Re: Next steps for schema language

[Pantelis Antoniou]

Hi Rob,

> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> 
> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>> Hi Pantelis and Rob,
>> 
>> After the workshop next week, I'm trying to capture the direction
>> we're going for the schema format. Roughly I think we're aiming
>> towards:
>> 
>> - Schema files to be written in YAML
>> - DT files shall remain written in DTS for the foreseeable future.
>> YAML will be treated as an intermediary format
>>  - That said, we'll try to make design decisions that allow YAML to
>> be used as a source format.
>> - All schema files and yaml-encoded-dt files must be parsable by stock
>> YAML parsers
>> - Schema files to use the jsonschema vocabulary
>>  - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>  - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>    - C-like expressions as used in Pantelis' yamldt could be added in this way
>>  - Need to write a jsonschema "metaschema" do define DT specific extensions
>>    - metaschema will be used to validate format of schema files
>>    - Existing tools can confirm is schema files are in the right format.
>>    - will make review a lot easier.
> 
> I want to start small here with defining top-level board/soc bindings.
> This is essentially just defining the root node compatible strings.
> Seems easy enough, right? However, I quickly run into the problem of
> how to match for when to apply the schema. "compatible" is the obvious
> choice, but that's also what I'm checking. We can't key off of what we
> are validating. So we really need 2 schema. The first is for matching
> on any valid compatible for board, then 2nd is checking for valid
> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
> don't like that as we'd be listing compatibles twice. An alternative
> would be we apply every board schema and exactly 1 must pass. Perhaps
> we generate a schema that's a "oneOf" of all the boards? Then we just
> need to tag board schemas in some way.
> 

I’ve run into this as the first problem with validation using compatible properties.

The way I’ve solved it is by having a ‘selected’ property that is generating
a test for when to check a binding against a node.

For a concrete example using the spi-slave binding

—- spi-slave.yaml ---
...
inherings: *device-compatible
…

Where device compatible is:

%YAML 1.1
---
device-compatible: &device-compatible
  title: Contraint for devices with compatible properties
  # select node for checking when the compatible constraint and
  # the device status enable constraint are met.
  selected: [ "compatible", *device-status-enabled ]

  class: constraint
  virtual: true

The selected property here declares that any “compatible” property constraint
for the node under test must match and that the device-status-enabled
rule matches too.

The device-status-enabled is declared as follows:

%YAML 1.1
---
device-enabled:
  title: Contraint for enabled devices
 
  class: constraint
  virtual: true
  
  properties:
    status: &device-status-enabled
      category: optional
      type: str
      description: Marks device state as enabled
      constraint: |
        !exists || streq(v, "okay") || streq(v, "ok”)

This declares that the constraint matches when either the optional
status property is not present or if it is, it’s either “okay” or “ok”.

Taken together when the spi-slave binding is inherited (for instance
by the jedec,spi-nor binding:

— jedec,spi-nor.yaml —

…
inherits: *spi-slave

…
  compatible: &jedec-spi-nor-compatible
  category: required
  type: strseq
  constraint: | 
    anystreq(v,  "at25df321a") ||
     ….


> I'm thinking of this very specific case, but no doubt it is going to
> apply in other places. This has been one of the harder problems with
> writing dtc checks. What to key off of to trigger checks and how to
> check errors in the key itself.
> 
> Rob

Regards

— Pantelis

Re: Next steps for schema language

[Rob Herring]

On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hi Rob,
>
>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>
>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>> Hi Pantelis and Rob,
>>>
>>> After the workshop next week, I'm trying to capture the direction
>>> we're going for the schema format. Roughly I think we're aiming
>>> towards:
>>>
>>> - Schema files to be written in YAML
>>> - DT files shall remain written in DTS for the foreseeable future.
>>> YAML will be treated as an intermediary format
>>>  - That said, we'll try to make design decisions that allow YAML to
>>> be used as a source format.
>>> - All schema files and yaml-encoded-dt files must be parsable by stock
>>> YAML parsers
>>> - Schema files to use the jsonschema vocabulary
>>>  - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>>  - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>>    - C-like expressions as used in Pantelis' yamldt could be added in this way
>>>  - Need to write a jsonschema "metaschema" do define DT specific extensions
>>>    - metaschema will be used to validate format of schema files
>>>    - Existing tools can confirm is schema files are in the right format.
>>>    - will make review a lot easier.
>>
>> I want to start small here with defining top-level board/soc bindings.
>> This is essentially just defining the root node compatible strings.
>> Seems easy enough, right? However, I quickly run into the problem of
>> how to match for when to apply the schema. "compatible" is the obvious
>> choice, but that's also what I'm checking. We can't key off of what we
>> are validating. So we really need 2 schema. The first is for matching
>> on any valid compatible for board, then 2nd is checking for valid
>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>> don't like that as we'd be listing compatibles twice. An alternative
>> would be we apply every board schema and exactly 1 must pass. Perhaps
>> we generate a schema that's a "oneOf" of all the boards? Then we just
>> need to tag board schemas in some way.
>>
>
> I’ve run into this as the first problem with validation using compatible properties.
>
> The way I’ve solved it is by having a ‘selected’ property that is generating
> a test for when to check a binding against a node.

Okay, but what's the "jsonschema way" to do this is my question really.

Your example is a bit different too. It's a generic match (has
compatible), then check something else (status). I have a binding
specific match and then a binding specific check on the same thing. We
could have the same thing for a device. Match on a set of compatibles
for a device, then check combination of compatibles along with other
properties. If you have '"example,soc2-uart",
"example,fallback-uart"', we need to match on either string, but check
that both strings are present because either one alone is not valid.
Then to add to that, you may have soc3, soc4, etc.

Rob

Re: Next steps for schema language

[Pantelis Antoniou]

Hi Rob,

> On Nov 3, 2017, at 16:31 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> 
> On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> Hi Rob,
>> 
>>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>> 
>>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>>> Hi Pantelis and Rob,
>>>> 
>>>> After the workshop next week, I'm trying to capture the direction
>>>> we're going for the schema format. Roughly I think we're aiming
>>>> towards:
>>>> 
>>>> - Schema files to be written in YAML
>>>> - DT files shall remain written in DTS for the foreseeable future.
>>>> YAML will be treated as an intermediary format
>>>> - That said, we'll try to make design decisions that allow YAML to
>>>> be used as a source format.
>>>> - All schema files and yaml-encoded-dt files must be parsable by stock
>>>> YAML parsers
>>>> - Schema files to use the jsonschema vocabulary
>>>> - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>>> - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>>>   - C-like expressions as used in Pantelis' yamldt could be added in this way
>>>> - Need to write a jsonschema "metaschema" do define DT specific extensions
>>>>   - metaschema will be used to validate format of schema files
>>>>   - Existing tools can confirm is schema files are in the right format.
>>>>   - will make review a lot easier.
>>> 
>>> I want to start small here with defining top-level board/soc bindings.
>>> This is essentially just defining the root node compatible strings.
>>> Seems easy enough, right? However, I quickly run into the problem of
>>> how to match for when to apply the schema. "compatible" is the obvious
>>> choice, but that's also what I'm checking. We can't key off of what we
>>> are validating. So we really need 2 schema. The first is for matching
>>> on any valid compatible for board, then 2nd is checking for valid
>>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>>> don't like that as we'd be listing compatibles twice. An alternative
>>> would be we apply every board schema and exactly 1 must pass. Perhaps
>>> we generate a schema that's a "oneOf" of all the boards? Then we just
>>> need to tag board schemas in some way.
>>> 
>> 
>> I’ve run into this as the first problem with validation using compatible properties.
>> 
>> The way I’ve solved it is by having a ‘selected’ property that is generating
>> a test for when to check a binding against a node.
> 
> Okay, but what's the "jsonschema way" to do this is my question really.
> 

No idea :)

DT is weird enough that there might not be a way to describe this in
a regular jsonschema form. I would wait until Grant pitches in.

> Your example is a bit different too. It's a generic match (has
> compatible), then check something else (status). I have a binding
> specific match and then a binding specific check on the same thing. We
> could have the same thing for a device. Match on a set of compatibles
> for a device, then check combination of compatibles along with other
> properties. If you have '"example,soc2-uart",
> "example,fallback-uart"', we need to match on either string, but check
> that both strings are present because either one alone is not valid.
> Then to add to that, you may have soc3, soc4, etc.
> 

I’m working on something to address this. The current scheme that fires
validation tests on matching on every node is going to be reworked.

The intent is to match on root node for a compatible and then if a match
is found fire off compatible matches on subnodes.

The specific combination match is something that can be expressed with
a C expression so it doesn’t look that difficult to me.

> Rob

Regards

— Pantelis

Re: Next steps for schema language

[Frank Rowand]

+me

On 11/02/17 09:44, Grant Likely wrote:
> Hi Pantelis and Rob,
> 
> After the workshop next week, I'm trying to capture the direction
> we're going for the schema format. Roughly I think we're aiming
> towards:
> 
> - Schema files to be written in YAML
> - DT files shall remain written in DTS for the foreseeable future.
> YAML will be treated as an intermediary format
>   - That said, we'll try to make design decisions that allow YAML to
> be used as a source format.
> - All schema files and yaml-encoded-dt files must be parsable by stock
> YAML parsers
> - Schema files to use the jsonschema vocabulary
>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
>     - C-like expressions as used in Pantelis' yamldt could be added in this way
>   - Need to write a jsonschema "metaschema" do define DT specific extensions
>     - metaschema will be used to validate format of schema files
>     - Existing tools can confirm is schema files are in the right format.
>     - will make review a lot easier.
> 
> The yaml encoding produced by yamldt is a good start, but some changes
> need to be made from the current code to be workable:
> - The redefinition of the anchor/alias syntax needs to be dropped.
>   - We need a labels/reference syntax instead.
>   - I'm using a $labels property to contain a list of labels which is
> working for me, but I'm open to other suggestions
>   - A YAML style !phandle or !path type definition would work for
> parsing references.
> - At the top level, the yaml-encoded-dt needs to be structured as a
> list, not a map.
>   - Using a list will properly accounts for how multiple top level
> trees are overlayed to create the final tree.
>   - I'm using a $path property to encode the path of each top level
> node. Again, I'm open to suggestions for a different approach
> 
> Pantelis, do you think you can rework yamldt to use that encoding?
> 
> I'm currently working on what is needed for a metaschema, and an
> engine for validating device-specific schema files using a standard
> jsonschema validation library. I'll post a git tree once I've got
> something worth looking at.
> 
> Cheers,
> g.
> --
> To unsubscribe from this list: send the line "unsubscribe devicetree-spec" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Next steps for schema language

[Rob Herring]

On Fri, Nov 3, 2017 at 9:41 AM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hi Rob,
>
>> On Nov 3, 2017, at 16:31 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>
>> On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>> Hi Rob,
>>>
>>>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>>
>>>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>>>> Hi Pantelis and Rob,
>>>>>
>>>>> After the workshop next week, I'm trying to capture the direction
>>>>> we're going for the schema format. Roughly I think we're aiming
>>>>> towards:
>>>>>
>>>>> - Schema files to be written in YAML
>>>>> - DT files shall remain written in DTS for the foreseeable future.
>>>>> YAML will be treated as an intermediary format
>>>>> - That said, we'll try to make design decisions that allow YAML to
>>>>> be used as a source format.
>>>>> - All schema files and yaml-encoded-dt files must be parsable by stock
>>>>> YAML parsers
>>>>> - Schema files to use the jsonschema vocabulary
>>>>> - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>>>> - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>>>>   - C-like expressions as used in Pantelis' yamldt could be added in this way
>>>>> - Need to write a jsonschema "metaschema" do define DT specific extensions
>>>>>   - metaschema will be used to validate format of schema files
>>>>>   - Existing tools can confirm is schema files are in the right format.
>>>>>   - will make review a lot easier.
>>>>
>>>> I want to start small here with defining top-level board/soc bindings.
>>>> This is essentially just defining the root node compatible strings.
>>>> Seems easy enough, right? However, I quickly run into the problem of
>>>> how to match for when to apply the schema. "compatible" is the obvious
>>>> choice, but that's also what I'm checking. We can't key off of what we
>>>> are validating. So we really need 2 schema. The first is for matching
>>>> on any valid compatible for board, then 2nd is checking for valid
>>>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>>>> don't like that as we'd be listing compatibles twice. An alternative
>>>> would be we apply every board schema and exactly 1 must pass. Perhaps
>>>> we generate a schema that's a "oneOf" of all the boards? Then we just
>>>> need to tag board schemas in some way.
>>>>
>>>
>>> I’ve run into this as the first problem with validation using compatible properties.
>>>
>>> The way I’ve solved it is by having a ‘selected’ property that is generating
>>> a test for when to check a binding against a node.
>>
>> Okay, but what's the "jsonschema way" to do this is my question really.
>>
>
> No idea :)
>
> DT is weird enough that there might not be a way to describe this in
> a regular jsonschema form. I would wait until Grant pitches in.

I've played around with things a bit and the more I do the less happy
I am with jsonschema. Maybe this is not what Grant has in mind, but
here's the snippet of the compatible check I have:

properties:
  compatible:
    description: |
      Compatible strings for the board example.

    type: array
    items:
      type: string
      oneOf:
        - enum:
          - "example,board"
          - "example,board2"
        - enum:
          - "example,soc"


First, it is more verbose than I'd like and not a language immediately
intuitive to low-level C developers (at least for me). My first
mistake was that *Of values have to be schema objects when I really
want logic ops for values. Second, the constraints are not complete
and and I've not come up with how you would express them. Essentially,
we need to express at least one of each set is required and
"example,soc" must be last. I suppose we can come up with custom
expressions, but it seems to me that if we can't even express a simple
example like this with standard jsonschema then it is not a good
choice.

Don't take this as we should use eBPF either. Given the reasoning so
far for picking it, I'm not sold on it. Seems like a nice shiny hammer
looking for a problem.

Rob

Re: Next steps for schema language

[Grant Likely]

Hi folks,

I've also been playing with the schemas, and have created a bit of a
framework for actually validating that binding files are valid
schemas, and that a DT validates against all the available schemas.
However, I haven't had time to respond on this thread, or write up
much of what I've done. I'll do that tomorrow. For now, if you're
interested, you can find the code I've written here:

https://github.com/glikely/yaml-bindings

I forked the repo that Rob created for this test code. It will load
all the schema files that it can find in the schemas subtree. To run
it just pass the DT file into the script:

$ ./dt-validate.py test/juno.cpp.yaml

Cheers,
g.

On Mon, Nov 6, 2017 at 4:12 PM, Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Fri, Nov 3, 2017 at 9:41 AM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> Hi Rob,
>>
>>> On Nov 3, 2017, at 16:31 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>
>>> On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>> Hi Rob,
>>>>
>>>>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>>>
>>>>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
>>>>>> Hi Pantelis and Rob,
>>>>>>
>>>>>> After the workshop next week, I'm trying to capture the direction
>>>>>> we're going for the schema format. Roughly I think we're aiming
>>>>>> towards:
>>>>>>
>>>>>> - Schema files to be written in YAML
>>>>>> - DT files shall remain written in DTS for the foreseeable future.
>>>>>> YAML will be treated as an intermediary format
>>>>>> - That said, we'll try to make design decisions that allow YAML to
>>>>>> be used as a source format.
>>>>>> - All schema files and yaml-encoded-dt files must be parsable by stock
>>>>>> YAML parsers
>>>>>> - Schema files to use the jsonschema vocabulary
>>>>>> - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>>>>> - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>>>>>   - C-like expressions as used in Pantelis' yamldt could be added in this way
>>>>>> - Need to write a jsonschema "metaschema" do define DT specific extensions
>>>>>>   - metaschema will be used to validate format of schema files
>>>>>>   - Existing tools can confirm is schema files are in the right format.
>>>>>>   - will make review a lot easier.
>>>>>
>>>>> I want to start small here with defining top-level board/soc bindings.
>>>>> This is essentially just defining the root node compatible strings.
>>>>> Seems easy enough, right? However, I quickly run into the problem of
>>>>> how to match for when to apply the schema. "compatible" is the obvious
>>>>> choice, but that's also what I'm checking. We can't key off of what we
>>>>> are validating. So we really need 2 schema. The first is for matching
>>>>> on any valid compatible for board, then 2nd is checking for valid
>>>>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>>>>> don't like that as we'd be listing compatibles twice. An alternative
>>>>> would be we apply every board schema and exactly 1 must pass. Perhaps
>>>>> we generate a schema that's a "oneOf" of all the boards? Then we just
>>>>> need to tag board schemas in some way.
>>>>>
>>>>
>>>> I’ve run into this as the first problem with validation using compatible properties.
>>>>
>>>> The way I’ve solved it is by having a ‘selected’ property that is generating
>>>> a test for when to check a binding against a node.
>>>
>>> Okay, but what's the "jsonschema way" to do this is my question really.
>>>
>>
>> No idea :)
>>
>> DT is weird enough that there might not be a way to describe this in
>> a regular jsonschema form. I would wait until Grant pitches in.
>
> I've played around with things a bit and the more I do the less happy
> I am with jsonschema. Maybe this is not what Grant has in mind, but
> here's the snippet of the compatible check I have:
>
> properties:
>   compatible:
>     description: |
>       Compatible strings for the board example.
>
>     type: array
>     items:
>       type: string
>       oneOf:
>         - enum:
>           - "example,board"
>           - "example,board2"
>         - enum:
>           - "example,soc"
>
>
> First, it is more verbose than I'd like and not a language immediately
> intuitive to low-level C developers (at least for me). My first
> mistake was that *Of values have to be schema objects when I really
> want logic ops for values. Second, the constraints are not complete
> and and I've not come up with how you would express them. Essentially,
> we need to express at least one of each set is required and
> "example,soc" must be last. I suppose we can come up with custom
> expressions, but it seems to me that if we can't even express a simple
> example like this with standard jsonschema then it is not a good
> choice.
>
> Don't take this as we should use eBPF either. Given the reasoning so
> far for picking it, I'm not sold on it. Seems like a nice shiny hammer
> looking for a problem.
>
> Rob

Re: Next steps for schema language

[Grant Likely]

On Mon, Nov 6, 2017 at 4:12 PM, Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Fri, Nov 3, 2017 at 9:41 AM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> Hi Rob,
>>
>>> On Nov 3, 2017, at 16:31 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>
>>> On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>> Hi Rob,
>>>>
>>>>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>>>
>>>>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
>>>>>> Hi Pantelis and Rob,
>>>>>>
>>>>>> After the workshop next week, I'm trying to capture the direction
>>>>>> we're going for the schema format. Roughly I think we're aiming
>>>>>> towards:
>>>>>>
>>>>>> - Schema files to be written in YAML
>>>>>> - DT files shall remain written in DTS for the foreseeable future.
>>>>>> YAML will be treated as an intermediary format
>>>>>> - That said, we'll try to make design decisions that allow YAML to
>>>>>> be used as a source format.
>>>>>> - All schema files and yaml-encoded-dt files must be parsable by stock
>>>>>> YAML parsers
>>>>>> - Schema files to use the jsonschema vocabulary
>>>>>> - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>>>>> - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>>>>>   - C-like expressions as used in Pantelis' yamldt could be added in this way
>>>>>> - Need to write a jsonschema "metaschema" do define DT specific extensions
>>>>>>   - metaschema will be used to validate format of schema files
>>>>>>   - Existing tools can confirm is schema files are in the right format.
>>>>>>   - will make review a lot easier.
>>>>>
>>>>> I want to start small here with defining top-level board/soc bindings.
>>>>> This is essentially just defining the root node compatible strings.
>>>>> Seems easy enough, right? However, I quickly run into the problem of
>>>>> how to match for when to apply the schema. "compatible" is the obvious
>>>>> choice, but that's also what I'm checking. We can't key off of what we
>>>>> are validating. So we really need 2 schema. The first is for matching
>>>>> on any valid compatible for board, then 2nd is checking for valid
>>>>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>>>>> don't like that as we'd be listing compatibles twice. An alternative
>>>>> would be we apply every board schema and exactly 1 must pass. Perhaps
>>>>> we generate a schema that's a "oneOf" of all the boards? Then we just
>>>>> need to tag board schemas in some way.

Creating a big top level schema that includes every board as a "oneOf"
is a non-starter for me. It gets unwieldy in a hurry and doesn't
account for how to bring in device bindings.

I'm working with the model of loading all the schema files
individually and iterating over all the nodes in the DT. For each
node, check which schemas are applicable (sometimes more than one) and
use them to validate the node. All applicable schemas must pass.

An upshot of this model is that bindings don't need to define
absolutely everything, only what isn't covered by more generic
schemas. For instance, bindings don't need to define the format of
interrupts, #*-cells, reg, etc because the core schema already defines
those. Instead they only need to list the properties that are
required, and can add constraints on the values in standard
properties.

>>>> I’ve run into this as the first problem with validation using compatible properties.
>>>>
>>>> The way I’ve solved it is by having a ‘selected’ property that is generating
>>>> a test for when to check a binding against a node.
>>>
>>> Okay, but what's the "jsonschema way" to do this is my question really.

The most recent pre-release jsonschema draft defines if/then/else[1]
keywords for conditional validation, but I'm using a draft-4 validator
which doesn't implement that. Instead I did something similar to
Pantelis by adding a "select" property that contains a schema. If the
select schema matches, then the DT node must match the entire schema.

[1] http://json-schema.org/work-in-progress/WIP-jsonschema-validation.html#rfc.section.6.6

the "jsonschema way" would also be to compose a single schema that
validates the entire document, but that doesn't work in the DT context
simply because we are going to have a tonne of binding files. It will
be unmanageable to create a single overarching schema that explicitly
includes all of the individual device binding files into a single
validator instance.

Instead, I think the validator tool needs to load a directory of
binding files and be intelligent about when each one needs to be
applied to a node (such as keyed off compatible). That's what I'm
doing with the prototype code I pushed out yesterday. The validator
loads all the schema files it can find and then iterates over the
devicetree. When a node validates against the "select" schema, then it
checks the entire schema against the node. For example:

%YAML 1.1
---
id: "http://devicetree.org/schemas/soc/example.yaml#"
$schema: "http://json-schema.org/draft-04/schema#"
version: 1
title: ARM Juno boards
description: >
  A board binding example. Matches on a top-level compatible string and model.

# this binding is selected when the compatible property constraint matches
select:
  required: ["compatible", "$path"]
  properties:
    $path: { const: "/" }
    compatible:
      contains:
        enum: [ "arm,juno", "arm,juno-r1", "arm,juno-r2" ]

required:
- model
- psci
- cpus

properties:
  model:
    enum:
      - "ARM Juno development board (r1)"
      - "ARM Juno development board (r2)"

This is a board level binding file for the Juno board. There are three
important top level properties:
== select ==
Contains a schema. If the node is at the root ($path=='/') and
compatible is one of the juno boards, then this binding applys

== required ==
List of properties/nodes that must be present. In this case model,
psci, and cpus. compatible isn't listed because it is already
guaranteed to be present because it was in the select node. Also note
that the contents of the nodes/properties doesn't have to be
specified. The format of a lot of standard properties will already be
validated by the core DT schema.

For example, model must always be a simple string.

== properties ==
Schemas for specific properties can go here. In this case I've
constrained model to contain one of two strings, and in the test repo
this demonstrates a validation failure because the juno.cpp.dts
contains (r0) instead of (r1) or (r2).


>> No idea :)
>>
>> DT is weird enough that there might not be a way to describe this in
>> a regular jsonschema form. I would wait until Grant pitches in.
>
> I've played around with things a bit and the more I do the less happy
> I am with jsonschema. Maybe this is not what Grant has in mind, but
> here's the snippet of the compatible check I have:
>
> properties:
>   compatible:
>     description: |
>       Compatible strings for the board example.
>
>     type: array
>     items:
>       type: string
>       oneOf:
>         - enum:
>           - "example,board"
>           - "example,board2"
>         - enum:
>           - "example,soc"


I modified this one a bit to show how it would work with the select
property. In this case the binding matches against two possible
compatible strings, but the properties list also enforces
"example,soc" to appear in the compatible list.

# this binding is selected when the compatible property constraint matches
select:
  required: ["compatible"]
  properties:
    compatible:
      contains:
        enum: [ "example,board", "example,board2" ]

properties:
  # The "select" keyword above already ensures the board compatible is in the
  # list. This entry makes sure the soc compatible string is also there. It is
  # also a place to put the description for documentation purposes.
  compatible:
    contains:
        const: "example,soc"

> First, it is more verbose than I'd like and not a language immediately
> intuitive to low-level C developers (at least for me). My first
> mistake was that *Of values have to be schema objects when I really
> want logic ops for values.

Yes, more verbose that I would like too, but I have yet to come across
anything much better. I think defining an extensible schema language
is just hard and it brings with it a learning curve. Every schema
system I looked at has the same problem. No matter what we do we're
going to have the pain of it not being intuitive to people used to
programming in C.

For constant values, the const and enum properties seem to be most
concise way to specify a specific value using stock jsonschema. We can
however define new keywords for DT specific validation. A stock
validator will ignore them, but a DT aware validator can use them to
do more complete validation.

> Second, the constraints are not complete
> and and I've not come up with how you would express them. Essentially,
> we need to express at least one of each set is required and
> "example,soc" must be last. I suppose we can come up with custom
> expressions, but it seems to me that if we can't even express a simple
> example like this with standard jsonschema then it is not a good
> choice.

If the compatible string was a known size then ordering can be
enforced using the items property, but there isn't anything in the
spec or proposed for enforcing order in arbitrarily sized arrays. It
would need to be an extension.

I don't think that makes jsonschema as a whole a bad choice. It does a
lot of the things we need right away, and not matter what we choose
we're going to be poking at corner cases where the DT context doesn't
quite fit. At the very least, I think there needs to be more examples
converted over to see what it looks like in real world usage.

> Don't take this as we should use eBPF either. Given the reasoning so
> far for picking it, I'm not sold on it. Seems like a nice shiny hammer
> looking for a problem.
>
> Rob

Re: Next steps for schema language

[Grant Likely]

On Thu, Nov 2, 2017 at 6:13 PM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hi Grant,
>
> Mostly agree, some notes below.
>
> On Thu, 2017-11-02 at 16:44 +0000, Grant Likely wrote:
>> Hi Pantelis and Rob,
>>
>> After the workshop next week, I'm trying to capture the direction
>> we're going for the schema format. Roughly I think we're aiming
>> towards:
>>
>> - Schema files to be written in YAML
>> - DT files shall remain written in DTS for the foreseeable future.
>> YAML will be treated as an intermediary format
>>   - That said, we'll try to make design decisions that allow YAML to
>> be used as a source format.
>> - All schema files and yaml-encoded-dt files must be parsable by stock
>> YAML parsers
>> - Schema files to use the jsonschema vocabulary
>>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>     - C-like expressions as used in Pantelis' yamldt could be added in this way
>>   - Need to write a jsonschema "metaschema" do define DT specific extensions
>>     - metaschema will be used to validate format of schema files
>>     - Existing tools can confirm is schema files are in the right format.
>>     - will make review a lot easier.
>>
>> The yaml encoding produced by yamldt is a good start, but some changes
>> need to be made from the current code to be workable:
>> - The redefinition of the anchor/alias syntax needs to be dropped.
>>   - We need a labels/reference syntax instead.
>>   - I'm using a $labels property to contain a list of labels which is
>> working for me, but I'm open to other suggestions
>
> I'm working on supporting just that. Should be ready for testing in a
> couple of days. Do note that I think we should keep the old '*' usage
> for supporting the binding files that are YAML.

Not sure what you mean here. Do you mean allowing native YAML
anchors/aliases in binding files to reduce duplication? If so, I think
that should be discouraged in favour of jsonschema's native $ref
keyword for referencing other nodes. It's not quite as expressive as
anchors/alias, but it is portable for anyone who needs to transcode
into strict JSON.

>>   - A YAML style !phandle or !path type definition would work for
>> parsing references.
>
> !phandle is a bad name IMO. It assumes that the implementation is going
> to always be a cell integer containing a phandle. I think !ref is
> better.

Sure, I'm fine with that

> Note that we must come with a way to encode types in JSON, since
> they are not natively supported.

Agreed.

>
>> - At the top level, the yaml-encoded-dt needs to be structured as a
>> list, not a map.
>>   - Using a list will properly accounts for how multiple top level
>> trees are overlayed to create the final tree.
>
>
> This is true only for non-resolved files. Resolved files are going to
> be guaranteed a map. Does this cause problems for your validator?

Make the resolved file a list with only one entry at the top level,
then the problem goes away.

Regardless, I'm finding that the validator needs to walk the tree(s)
and be able to apply a binding at any level, so driver bindings won't
be affected by the structure at the top level. Board and SoC bindings
might.

>>   - I'm using a $path property to encode the path of each top level
>> node. Again, I'm open to suggestions for a different approach
>>
>
> A bare scalar that starts with / can always be deduced to be a path.

I don't understand what you mean. That scalar still needs to be
encoded in some way.

> Can you share some examples about your usage?

Consider an unresolved tree that has successive trees applied to
different levels:

/ { ... };
&etm0 { ... };
&etm1 { ... };
/ { ... };
/aliases { ... };

To transcode this into YAML the path/reference needs to be stored
somewhere. As already discussed, it cannot be a map because keys can
appear more than once and order of application matters, so it must be
a list. Some possible options for storing the path/reference in the
array structure are:

Store the path/tree pair as a tuple (an array of arrays)
- [ / , {...} ]
- [ &etm0, {...} ]
- [ &etm1 , {...} ]
- [ / , {...} ]
- [ /aliases , {...} ]

Or it could be stored as a special property in the node, something
that doesn't collide with child/property names. An array of maps:

- $path: "/"
  ...
- $path: "&etm0"
  ...
- $path: "&etm1"
  ...
- $path: "/"
  ...
- $path: "/aliases"
  ...

Personally, I prefer embedding the path right into the node because it
drops a level of nesting.

Thoughts?
g.

Re: Next steps for schema language

[Rob Herring]

On Tue, Nov 7, 2017 at 7:46 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> On Mon, Nov 6, 2017 at 4:12 PM, Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> On Fri, Nov 3, 2017 at 9:41 AM, Pantelis Antoniou
>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>> Hi Rob,
>>>
>>>> On Nov 3, 2017, at 16:31 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>>
>>>> On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>>> Hi Rob,
>>>>>
>>>>>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>>>>
>>>>>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>>>>>>> Hi Pantelis and Rob,
>>>>>>>
>>>>>>> After the workshop next week, I'm trying to capture the direction
>>>>>>> we're going for the schema format. Roughly I think we're aiming
>>>>>>> towards:
>>>>>>>
>>>>>>> - Schema files to be written in YAML
>>>>>>> - DT files shall remain written in DTS for the foreseeable future.
>>>>>>> YAML will be treated as an intermediary format
>>>>>>> - That said, we'll try to make design decisions that allow YAML to
>>>>>>> be used as a source format.
>>>>>>> - All schema files and yaml-encoded-dt files must be parsable by stock
>>>>>>> YAML parsers
>>>>>>> - Schema files to use the jsonschema vocabulary
>>>>>>> - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>>>>>> - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>>>>>>   - C-like expressions as used in Pantelis' yamldt could be added in this way
>>>>>>> - Need to write a jsonschema "metaschema" do define DT specific extensions
>>>>>>>   - metaschema will be used to validate format of schema files
>>>>>>>   - Existing tools can confirm is schema files are in the right format.
>>>>>>>   - will make review a lot easier.
>>>>>>
>>>>>> I want to start small here with defining top-level board/soc bindings.
>>>>>> This is essentially just defining the root node compatible strings.
>>>>>> Seems easy enough, right? However, I quickly run into the problem of
>>>>>> how to match for when to apply the schema. "compatible" is the obvious
>>>>>> choice, but that's also what I'm checking. We can't key off of what we
>>>>>> are validating. So we really need 2 schema. The first is for matching
>>>>>> on any valid compatible for board, then 2nd is checking for valid
>>>>>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>>>>>> don't like that as we'd be listing compatibles twice. An alternative
>>>>>> would be we apply every board schema and exactly 1 must pass. Perhaps
>>>>>> we generate a schema that's a "oneOf" of all the boards? Then we just
>>>>>> need to tag board schemas in some way.
>
> Creating a big top level schema that includes every board as a "oneOf"
> is a non-starter for me. It gets unwieldy in a hurry and doesn't
> account for how to bring in device bindings.
>
> I'm working with the model of loading all the schema files
> individually and iterating over all the nodes in the DT. For each
> node, check which schemas are applicable (sometimes more than one) and
> use them to validate the node. All applicable schemas must pass.
>
> An upshot of this model is that bindings don't need to define
> absolutely everything, only what isn't covered by more generic
> schemas. For instance, bindings don't need to define the format of
> interrupts, #*-cells, reg, etc because the core schema already defines
> those. Instead they only need to list the properties that are
> required, and can add constraints on the values in standard
> properties.
>
>>>>> I’ve run into this as the first problem with validation using compatible properties.
>>>>>
>>>>> The way I’ve solved it is by having a ‘selected’ property that is generating
>>>>> a test for when to check a binding against a node.
>>>>
>>>> Okay, but what's the "jsonschema way" to do this is my question really.
>
> The most recent pre-release jsonschema draft defines if/then/else[1]
> keywords for conditional validation, but I'm using a draft-4 validator
> which doesn't implement that. Instead I did something similar to
> Pantelis by adding a "select" property that contains a schema. If the
> select schema matches, then the DT node must match the entire schema.
>
> [1] http://json-schema.org/work-in-progress/WIP-jsonschema-validation.html#rfc.section.6.6
>
> the "jsonschema way" would also be to compose a single schema that
> validates the entire document, but that doesn't work in the DT context
> simply because we are going to have a tonne of binding files. It will
> be unmanageable to create a single overarching schema that explicitly
> includes all of the individual device binding files into a single
> validator instance.

And the jsonschema way would be with $ref, right? I can see both ways
being needed. For example, an I2C controller binding would want to
pull in an I2C bus schema. Otherwise, you'd need some tag to mark the
binding as an I2C bus and then apply the I2C bus based on that.

> Instead, I think the validator tool needs to load a directory of
> binding files and be intelligent about when each one needs to be
> applied to a node (such as keyed off compatible). That's what I'm
> doing with the prototype code I pushed out yesterday. The validator
> loads all the schema files it can find and then iterates over the
> devicetree. When a node validates against the "select" schema, then it
> checks the entire schema against the node. For example:
>
> %YAML 1.1
> ---
> id: "http://devicetree.org/schemas/soc/example.yaml#"
> $schema: "http://json-schema.org/draft-04/schema#"
> version: 1
> title: ARM Juno boards
> description: >
>   A board binding example. Matches on a top-level compatible string and model.
>
> # this binding is selected when the compatible property constraint matches
> select:
>   required: ["compatible", "$path"]
>   properties:
>     $path: { const: "/" }

See my pull request. This isn't actually working. :)

>     compatible:
>       contains:
>         enum: [ "arm,juno", "arm,juno-r1", "arm,juno-r2" ]
>
> required:
> - model
> - psci
> - cpus
>
> properties:
>   model:
>     enum:
>       - "ARM Juno development board (r1)"
>       - "ARM Juno development board (r2)"
>
> This is a board level binding file for the Juno board. There are three
> important top level properties:
> == select ==
> Contains a schema. If the node is at the root ($path=='/') and
> compatible is one of the juno boards, then this binding applys
>
> == required ==
> List of properties/nodes that must be present. In this case model,
> psci, and cpus. compatible isn't listed because it is already
> guaranteed to be present because it was in the select node. Also note
> that the contents of the nodes/properties doesn't have to be
> specified. The format of a lot of standard properties will already be
> validated by the core DT schema.
>
> For example, model must always be a simple string.
>
> == properties ==
> Schemas for specific properties can go here. In this case I've
> constrained model to contain one of two strings, and in the test repo
> this demonstrates a validation failure because the juno.cpp.dts
> contains (r0) instead of (r1) or (r2).
>
>
>>> No idea :)
>>>
>>> DT is weird enough that there might not be a way to describe this in
>>> a regular jsonschema form. I would wait until Grant pitches in.
>>
>> I've played around with things a bit and the more I do the less happy
>> I am with jsonschema. Maybe this is not what Grant has in mind, but
>> here's the snippet of the compatible check I have:
>>
>> properties:
>>   compatible:
>>     description: |
>>       Compatible strings for the board example.
>>
>>     type: array
>>     items:
>>       type: string
>>       oneOf:
>>         - enum:
>>           - "example,board"
>>           - "example,board2"
>>         - enum:
>>           - "example,soc"
>
>
> I modified this one a bit to show how it would work with the select
> property. In this case the binding matches against two possible
> compatible strings, but the properties list also enforces
> "example,soc" to appear in the compatible list.
>
> # this binding is selected when the compatible property constraint matches
> select:
>   required: ["compatible"]
>   properties:
>     compatible:
>       contains:
>         enum: [ "example,board", "example,board2" ]
>
> properties:
>   # The "select" keyword above already ensures the board compatible is in the
>   # list. This entry makes sure the soc compatible string is also there. It is
>   # also a place to put the description for documentation purposes.
>   compatible:
>     contains:
>         const: "example,soc"

I really dislike the split here with the duplication of the data. You
don't check the case where you only have the fallback (a common
error). We also need to check order.

  allOf:
    contains:
      const: "example,soc"
    contains:
      enum: [ "example,board", "example,board2" ]

BTW, this is also silently ignored because "contains" is a v6 keyword
and only v4 is supported. While silently ignoring in some contexts is
a feature, I think here that is a problem.

>> First, it is more verbose than I'd like and not a language immediately
>> intuitive to low-level C developers (at least for me). My first
>> mistake was that *Of values have to be schema objects when I really
>> want logic ops for values.
>
> Yes, more verbose that I would like too, but I have yet to come across
> anything much better. I think defining an extensible schema language
> is just hard and it brings with it a learning curve. Every schema
> system I looked at has the same problem. No matter what we do we're
> going to have the pain of it not being intuitive to people used to
> programming in C.

Did you consider fragments of python or some other language? Not
exactly sure what that would look like, but some subset of the
$language with a set of DT specific functions that can be called.
Perhaps a language that cares about whitespace like python is not a
good thing to embed in YAML/JSON.  I like the syntax of Pantelis' eBPF
constraints (being C like), it's really only the choice of backend
that I'm questioning there. Giving what everyone has said is a
non-goal (in kernel validation) as the primary reason for eBPF hasn't
helped the cause.

Have you looked at the $data keyword? Not sure if that would help
here. That too may be a v6 thing.

At least for this case, we simply have N lists of data and need to
reference the lists in 2 different ways. We need to match on any
element from the N lists and then validate we have only one element
from each list and in list order. The binding writer should only have
to supply those lists. IMO, having the data listed twice and having
boilerplate that's longer than the data are non-starters. It's one
thing for some vendor specific binding to have some verbose
jsonschema, but it's another thing for something that will be in every
single binding document.

Another case that comes to mind is how to implement generic checks for
I2C and SPI buses. Other than node name (which is often not the
generic name), we don't have any properties to match on (for many
common bindings we can use "#*-cells" properties). That leaves us with
needing a list of I2C controller compatible strings to match on.

> For constant values, the const and enum properties seem to be most
> concise way to specify a specific value using stock jsonschema. We can
> however define new keywords for DT specific validation. A stock
> validator will ignore them, but a DT aware validator can use them to
> do more complete validation.

I need to study how to do that. Have you found examples doing that?

>> Second, the constraints are not complete
>> and and I've not come up with how you would express them. Essentially,
>> we need to express at least one of each set is required and
>> "example,soc" must be last. I suppose we can come up with custom
>> expressions, but it seems to me that if we can't even express a simple
>> example like this with standard jsonschema then it is not a good
>> choice.
>
> If the compatible string was a known size then ordering can be
> enforced using the items property, but there isn't anything in the
> spec or proposed for enforcing order in arbitrarily sized arrays. It
> would need to be an extension.
>
> I don't think that makes jsonschema as a whole a bad choice. It does a
> lot of the things we need right away, and not matter what we choose
> we're going to be poking at corner cases where the DT context doesn't
> quite fit. At the very least, I think there needs to be more examples
> converted over to see what it looks like in real world usage.

I'm just highlighting the issues I see some of which may be due to my
what can be counted in hours of experience with jsonschema. I
certainly want to see how it works for specific examples. In
particular, I want to see things that can't be done easily with dtc.

Rob

Re: Next steps for schema language

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 6772 bytes --]

On Tue, Nov 07, 2017 at 02:14:06PM +0000, Grant Likely wrote:
> On Thu, Nov 2, 2017 at 6:13 PM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Hi Grant,
> >
> > Mostly agree, some notes below.
> >
> > On Thu, 2017-11-02 at 16:44 +0000, Grant Likely wrote:
> >> Hi Pantelis and Rob,
> >>
> >> After the workshop next week, I'm trying to capture the direction
> >> we're going for the schema format. Roughly I think we're aiming
> >> towards:
> >>
> >> - Schema files to be written in YAML
> >> - DT files shall remain written in DTS for the foreseeable future.
> >> YAML will be treated as an intermediary format
> >>   - That said, we'll try to make design decisions that allow YAML to
> >> be used as a source format.
> >> - All schema files and yaml-encoded-dt files must be parsable by stock
> >> YAML parsers
> >> - Schema files to use the jsonschema vocabulary
> >>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
> >>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
> >>     - C-like expressions as used in Pantelis' yamldt could be added in this way
> >>   - Need to write a jsonschema "metaschema" do define DT specific extensions
> >>     - metaschema will be used to validate format of schema files
> >>     - Existing tools can confirm is schema files are in the right format.
> >>     - will make review a lot easier.
> >>
> >> The yaml encoding produced by yamldt is a good start, but some changes
> >> need to be made from the current code to be workable:
> >> - The redefinition of the anchor/alias syntax needs to be dropped.
> >>   - We need a labels/reference syntax instead.
> >>   - I'm using a $labels property to contain a list of labels which is
> >> working for me, but I'm open to other suggestions
> >
> > I'm working on supporting just that. Should be ready for testing in a
> > couple of days. Do note that I think we should keep the old '*' usage
> > for supporting the binding files that are YAML.
> 
> Not sure what you mean here. Do you mean allowing native YAML
> anchors/aliases in binding files to reduce duplication? If so, I think
> that should be discouraged in favour of jsonschema's native $ref
> keyword for referencing other nodes. It's not quite as expressive as
> anchors/alias, but it is portable for anyone who needs to transcode
> into strict JSON.

Yeah, I'm not sure what's meant here either.  I think the anchor alias
syntax should be supported in the standard YAML way, with the alias
expanding to the full object referenced.  It doesn't help us for
phandle/path references, but it's useful in its own right.

Note that the problem with using alias syntax for references is purely
on the alias side.  I don't see a reason we can't use standard YAML
anchor syntax then something else to construct a reference to that
anchor.

The $labels magic property is pretty yucky, it'd be good to avoid it
if we can.

> >>   - A YAML style !phandle or !path type definition would work for
> >> parsing references.
> >
> > !phandle is a bad name IMO. It assumes that the implementation is going
> > to always be a cell integer containing a phandle. I think !ref is
> > better.

Uhh.. I don't understand your point here.  To output something the
kernel will understand an integer phandle is exactly what the
reference needs to expand to.

I'm dubious about changing that implementation, but even if we did
we'd still need a way to generate the current style for
compatibility.  So !phandle or !phandlerref (to match the !pathref I
think you already have) seems like a good name to me.

> Sure, I'm fine with that
> 
> > Note that we must come with a way to encode types in JSON, since
> > they are not natively supported.
> 
> Agreed.

I thought the YAML spec had some guidelines for flattening YAML to
JSON.

> >> - At the top level, the yaml-encoded-dt needs to be structured as a
> >> list, not a map.
> >>   - Using a list will properly accounts for how multiple top level
> >> trees are overlayed to create the final tree.
> >
> >
> > This is true only for non-resolved files. Resolved files are going to
> > be guaranteed a map. Does this cause problems for your validator?
> 
> Make the resolved file a list with only one entry at the top level,
> then the problem goes away.
> 
> Regardless, I'm finding that the validator needs to walk the tree(s)
> and be able to apply a binding at any level, so driver bindings won't
> be affected by the structure at the top level. Board and SoC bindings
> might.
> 
> >>   - I'm using a $path property to encode the path of each top level
> >> node. Again, I'm open to suggestions for a different approach
> >>
> >
> > A bare scalar that starts with / can always be deduced to be a path.
> 
> I don't understand what you mean. That scalar still needs to be
> encoded in some way.
> 
> > Can you share some examples about your usage?
> 
> Consider an unresolved tree that has successive trees applied to
> different levels:
> 
> / { ... };
> &etm0 { ... };
> &etm1 { ... };
> / { ... };
> /aliases { ... };
> 
> To transcode this into YAML the path/reference needs to be stored
> somewhere. As already discussed, it cannot be a map because keys can
> appear more than once and order of application matters, so it must be
> a list. Some possible options for storing the path/reference in the
> array structure are:

Seems to me you want a type-tagged map for each overlay component,
something like

!overlayfragment {
	target-label = "etm0";
	content = { /* the actual nodes */ };
}

> 
> Store the path/tree pair as a tuple (an array of arrays)
> - [ / , {...} ]
> - [ &etm0, {...} ]
> - [ &etm1 , {...} ]
> - [ / , {...} ]
> - [ /aliases , {...} ]

Works for now, but limited if we ever need more metadata for an
overlay/fragment.

> Or it could be stored as a special property in the node, something
> that doesn't collide with child/property names. An array of maps:
> 
> - $path: "/"
>   ...
> - $path: "&etm0"
>   ...
> - $path: "&etm1"
>   ...
> - $path: "/"
>   ...
> - $path: "/aliases"
>   ...
> 
> Personally, I prefer embedding the path right into the node because it
> drops a level of nesting.

I dislike this because it mixes "in band" and "out of band" info,
requiring more special handling in tools to separate them again.  I
really think a type-tagged map is the right (and YAML-ish) way to do
this.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Next steps for schema language

[Pantelis Antoniou]

Hi Grant,

Just pushed a few changes to yamldt which among other things enables
JSON support. Note that JSON is strictly an output format for now.

Notable features are:

- JSON output is now supported.

- '$' is supported as well as a reference indicator on bare scalars
similar to '*'

    property: *ref 
    property: $ref

are equivalent.

- Labels can be declared using the /label/ special property.
The use of /label/ was used since it's similar to the way /memreserve/
is a special keyword. Internally it's the same as declaring an
anchor.

  foo: &label
  foo:
    /label/: label

You can also use a sequence for multiple label declaration in one go.

  foo:
    /label/: [ label1, label2 ]

- !ref as a long form method for a reference:

  foo: *label
  foo: $label
  foo: !ref label

Are all equivalent.

- JSON does not support tags; so yaml tags are transformed to a tuple
sequence:

  foo: !int16 10

  foo: [ "\f!int16", 10 ]

The form feed escape and ! at the start of a string scalar allows
detection that this is a type declaration. This is arguably ugly but
then again, JSON is not meant to be used as a source format.

On Tue, 2017-11-07 at 14:14 +0000, Grant Likely wrote:
> On Thu, Nov 2, 2017 at 6:13 PM, Pantelis Antoniou
> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Hi Grant,
> >
> > Mostly agree, some notes below.
> >
> > On Thu, 2017-11-02 at 16:44 +0000, Grant Likely wrote:
> >> Hi Pantelis and Rob,
> >>
> >> After the workshop next week, I'm trying to capture the direction
> >> we're going for the schema format. Roughly I think we're aiming
> >> towards:
> >>
> >> - Schema files to be written in YAML
> >> - DT files shall remain written in DTS for the foreseeable future.
> >> YAML will be treated as an intermediary format
> >>   - That said, we'll try to make design decisions that allow YAML to
> >> be used as a source format.
> >> - All schema files and yaml-encoded-dt files must be parsable by stock
> >> YAML parsers
> >> - Schema files to use the jsonschema vocabulary
> >>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
> >>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
> >>     - C-like expressions as used in Pantelis' yamldt could be added in this way
> >>   - Need to write a jsonschema "metaschema" do define DT specific extensions
> >>     - metaschema will be used to validate format of schema files
> >>     - Existing tools can confirm is schema files are in the right format.
> >>     - will make review a lot easier.
> >>
> >> The yaml encoding produced by yamldt is a good start, but some changes
> >> need to be made from the current code to be workable:
> >> - The redefinition of the anchor/alias syntax needs to be dropped.
> >>   - We need a labels/reference syntax instead.
> >>   - I'm using a $labels property to contain a list of labels which is
> >> working for me, but I'm open to other suggestions
> >
> > I'm working on supporting just that. Should be ready for testing in a
> > couple of days. Do note that I think we should keep the old '*' usage
> > for supporting the binding files that are YAML.
> 
> Not sure what you mean here. Do you mean allowing native YAML
> anchors/aliases in binding files to reduce duplication? If so, I think
> that should be discouraged in favour of jsonschema's native $ref
> keyword for referencing other nodes. It's not quite as expressive as
> anchors/alias, but it is portable for anyone who needs to transcode
> into strict JSON.
> 

Sorry, didn't express my self clearly there. I meant using $ as a
reference prefix.

> >>   - A YAML style !phandle or !path type definition would work for
> >> parsing references.
> >
> > !phandle is a bad name IMO. It assumes that the implementation is going
> > to always be a cell integer containing a phandle. I think !ref is
> > better.
> 
> Sure, I'm fine with that
> 

Done.

> > Note that we must come with a way to encode types in JSON, since
> > they are not natively supported.
> 
> Agreed.
> 

So, I've come up with a way ^^^

Not very aesthetically pleasing though. I'm open to suggestions.

> >
> >> - At the top level, the yaml-encoded-dt needs to be structured as a
> >> list, not a map.
> >>   - Using a list will properly accounts for how multiple top level
> >> trees are overlayed to create the final tree.
> >
> >
> > This is true only for non-resolved files. Resolved files are going to
> > be guaranteed a map. Does this cause problems for your validator?
> 
> Make the resolved file a list with only one entry at the top level,
> then the problem goes away.
> 

Err, isn't that a bit of a hack? :)

> Regardless, I'm finding that the validator needs to walk the tree(s)
> and be able to apply a binding at any level, so driver bindings won't
> be affected by the structure at the top level. Board and SoC bindings
> might.
> 

Hmm, definitely for board and SoC bindings.

> >>   - I'm using a $path property to encode the path of each top level
> >> node. Again, I'm open to suggestions for a different approach
> >>
> >
> > A bare scalar that starts with / can always be deduced to be a path.
> 
> I don't understand what you mean. That scalar still needs to be
> encoded in some way.
> 

OK, let me explain.

In yaml strings need not be quoted if they are 'simple'

So:

  property: "foo"
  property: foo

Are equivalent, but the first one is a quoted string while the second is
a bare scalar that since it's not null, boolean, or a number, is a
string.

We can use this to deduce that a bare string that looks like a path is a
path.

  path-property: !path "/foo"
  path-property: /foo

We can safely deduce that these are equivalent.

If you're doing this in JSON, then you have no other way than what
you're using now.

> > Can you share some examples about your usage?
> 
> Consider an unresolved tree that has successive trees applied to
> different levels:
> 
> / { ... };
> &etm0 { ... };
> &etm1 { ... };
> / { ... };
> /aliases { ... };
> 
> To transcode this into YAML the path/reference needs to be stored
> somewhere. As already discussed, it cannot be a map because keys can
> appear more than once and order of application matters, so it must be
> a list. Some possible options for storing the path/reference in the
> array structure are:
> 
> Store the path/tree pair as a tuple (an array of arrays)
> - [ / , {...} ]
> - [ &etm0, {...} ]
> - [ &etm1 , {...} ]
> - [ / , {...} ]
> - [ /aliases , {...} ]
> 
> Or it could be stored as a special property in the node, something
> that doesn't collide with child/property names. An array of maps:
> 
> - $path: "/"
>   ...
> - $path: "&etm0"
>   ...
> - $path: "&etm1"
>   ...
> - $path: "/"
>   ...
> - $path: "/aliases"
>   ...
> 
> Personally, I prefer embedding the path right into the node because it
> drops a level of nesting.
> 

Expanding your example:

/ {
  one; 
  etm0: etm0 { etm0_one; };
  etm1: etm1 { etm1_one; };
  aliases { nada; }
};
&etm0 { two; };
&etm1 { three; };
/ { four; };
/aliases { five; }; 

For yamldt would encode it as:

one: true
etm0: &etm0
  etm0_one: true
etm1: &etm1
  etm1_one: true
aliases:
  nada: true
*etm0:
  two: true
*etm1:
  three: true
four: true
/aliases:
  five: true

The resulting yaml output would be:

one: true
four: true
etm0:
  etm0_one: true
  two: true
etm1:
  etm1_one: true
  three: true
aliases:
  nada: true
  five: true

Note that the output file is proper YAML, it's the source that we are
talking about.

I think a very simple way to encode it would be (using $ for ref and
labels with the new style):

- /:
    one: true
    etm0:
      /label/: etm0
      etm0_one: true
    etm1:
      /label/: etm1
      etm1_one: true
    aliases:
      nada: true
- $etm0:
    two: true
- $etm1:
    three: true
- /:
    four: true
- /aliases:
    five: true

You can even get rid of the sequence indicators and keep the first
one if you're sure about indentation

- /:
    one: true
    etm0:
      /label/: etm0
      etm0_one: true
    etm1:
      /label/: etm1
      etm1_one: true
    aliases:
      nada: true
  $etm0:
    two: true
  $etm1:
    three: true
  /:
    four: true
  /aliases:
    five: true

This is valid YAML AFAIKT.

yamldt does not parse these forms yet, but it's not a big deal to
support them since the bare sequences are only present in root.

> Thoughts?
> g.

Regards

-- Pantelis

Re: Next steps for schema language

[Pantelis Antoniou]

Hi David,

On Thu, 2017-11-09 at 17:48 +1100, David Gibson wrote:
> On Tue, Nov 07, 2017 at 02:14:06PM +0000, Grant Likely wrote:
> > On Thu, Nov 2, 2017 at 6:13 PM, Pantelis Antoniou
> > <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > > Hi Grant,
> > >
> > > Mostly agree, some notes below.
> > >
> > > On Thu, 2017-11-02 at 16:44 +0000, Grant Likely wrote:
> > >> Hi Pantelis and Rob,
> > >>
> > >> After the workshop next week, I'm trying to capture the direction
> > >> we're going for the schema format. Roughly I think we're aiming
> > >> towards:
> > >>
> > >> - Schema files to be written in YAML
> > >> - DT files shall remain written in DTS for the foreseeable future.
> > >> YAML will be treated as an intermediary format
> > >>   - That said, we'll try to make design decisions that allow YAML to
> > >> be used as a source format.
> > >> - All schema files and yaml-encoded-dt files must be parsable by stock
> > >> YAML parsers
> > >> - Schema files to use the jsonschema vocabulary
> > >>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
> > >>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
> > >>     - C-like expressions as used in Pantelis' yamldt could be added in this way
> > >>   - Need to write a jsonschema "metaschema" do define DT specific extensions
> > >>     - metaschema will be used to validate format of schema files
> > >>     - Existing tools can confirm is schema files are in the right format.
> > >>     - will make review a lot easier.
> > >>
> > >> The yaml encoding produced by yamldt is a good start, but some changes
> > >> need to be made from the current code to be workable:
> > >> - The redefinition of the anchor/alias syntax needs to be dropped.
> > >>   - We need a labels/reference syntax instead.
> > >>   - I'm using a $labels property to contain a list of labels which is
> > >> working for me, but I'm open to other suggestions
> > >
> > > I'm working on supporting just that. Should be ready for testing in a
> > > couple of days. Do note that I think we should keep the old '*' usage
> > > for supporting the binding files that are YAML.
> > 
> > Not sure what you mean here. Do you mean allowing native YAML
> > anchors/aliases in binding files to reduce duplication? If so, I think
> > that should be discouraged in favour of jsonschema's native $ref
> > keyword for referencing other nodes. It's not quite as expressive as
> > anchors/alias, but it is portable for anyone who needs to transcode
> > into strict JSON.
> 
> Yeah, I'm not sure what's meant here either.  I think the anchor alias
> syntax should be supported in the standard YAML way, with the alias
> expanding to the full object referenced.  It doesn't help us for
> phandle/path references, but it's useful in its own right.
>
> Note that the problem with using alias syntax for references is purely
> on the alias side.  I don't see a reason we can't use standard YAML
> anchor syntax then something else to construct a reference to that
> anchor.
> 


Yes, that make sense to keep both IMO.

The '*' as the expanding version, and the '$' as the non-expanding (like
DT uses).

> The $labels magic property is pretty yucky, it'd be good to avoid it
> if we can.
> 

I'm not a big fond of it either. However if you have more than one label
in a node it's cleaner than re-declaring the same name node as empty
with the additional label.

If one was to expand YAML, there's no significant problem with allowing
multiple label definitions instead of a single one.

> > >>   - A YAML style !phandle or !path type definition would work for
> > >> parsing references.
> > >
> > > !phandle is a bad name IMO. It assumes that the implementation is going
> > > to always be a cell integer containing a phandle. I think !ref is
> > > better.
> 
> Uhh.. I don't understand your point here.  To output something the
> kernel will understand an integer phandle is exactly what the
> reference needs to expand to.
> 
> I'm dubious about changing that implementation, but even if we did
> we'd still need a way to generate the current style for
> compatibility.  So !phandle or !phandlerref (to match the !pathref I
> think you already have) seems like a good name to me.
> 

My point is that this is a reference, that it's encoded as an integer
for the DTB output option is an implementation detail.

> > Sure, I'm fine with that
> > 
> > > Note that we must come with a way to encode types in JSON, since
> > > they are not natively supported.
> > 
> > Agreed.
> 
> I thought the YAML spec had some guidelines for flattening YAML to
> JSON.
> 

It does flatten the content. The type information is thrown away.
Anyway, I've come up with a relatively simple (yet a bit ugly) way.

foo: !int16 10

can be transcribed to JSON type preserving format as

foo: [ "\f!int16", 10 ]

A knowledgeable parser can deduce that the sequence corresponds to the
original source.

> > >> - At the top level, the yaml-encoded-dt needs to be structured as a
> > >> list, not a map.
> > >>   - Using a list will properly accounts for how multiple top level
> > >> trees are overlayed to create the final tree.
> > >
> > >
> > > This is true only for non-resolved files. Resolved files are going to
> > > be guaranteed a map. Does this cause problems for your validator?
> > 
> > Make the resolved file a list with only one entry at the top level,
> > then the problem goes away.
> > 
> > Regardless, I'm finding that the validator needs to walk the tree(s)
> > and be able to apply a binding at any level, so driver bindings won't
> > be affected by the structure at the top level. Board and SoC bindings
> > might.
> > 
> > >>   - I'm using a $path property to encode the path of each top level
> > >> node. Again, I'm open to suggestions for a different approach
> > >>
> > >
> > > A bare scalar that starts with / can always be deduced to be a path.
> > 
> > I don't understand what you mean. That scalar still needs to be
> > encoded in some way.
> > 
> > > Can you share some examples about your usage?
> > 
> > Consider an unresolved tree that has successive trees applied to
> > different levels:
> > 
> > / { ... };
> > &etm0 { ... };
> > &etm1 { ... };
> > / { ... };
> > /aliases { ... };
> > 
> > To transcode this into YAML the path/reference needs to be stored
> > somewhere. As already discussed, it cannot be a map because keys can
> > appear more than once and order of application matters, so it must be
> > a list. Some possible options for storing the path/reference in the
> > array structure are:
> 
> Seems to me you want a type-tagged map for each overlay component,
> something like
> 
> !overlayfragment {
> 	target-label = "etm0";
> 	content = { /* the actual nodes */ };
> }
> 
> > 
> > Store the path/tree pair as a tuple (an array of arrays)
> > - [ / , {...} ]
> > - [ &etm0, {...} ]
> > - [ &etm1 , {...} ]
> > - [ / , {...} ]
> > - [ /aliases , {...} ]
> 
> Works for now, but limited if we ever need more metadata for an
> overlay/fragment.
> 
> > Or it could be stored as a special property in the node, something
> > that doesn't collide with child/property names. An array of maps:
> > 
> > - $path: "/"
> >   ...
> > - $path: "&etm0"
> >   ...
> > - $path: "&etm1"
> >   ...
> > - $path: "/"
> >   ...
> > - $path: "/aliases"
> >   ...
> > 
> > Personally, I prefer embedding the path right into the node because it
> > drops a level of nesting.
> 
> I dislike this because it mixes "in band" and "out of band" info,
> requiring more special handling in tools to separate them again.  I
> really think a type-tagged map is the right (and YAML-ish) way to do
> this.
> 

The example I've posted earlier does work like this.

The form:

- /:
    one: true
  $etm0:
    two: true

Is functionally identical with:

- !ref "/":
    one: true
  !ref "etm0"
    two: true

Regards

-- Pantelis

Re: Next steps for schema language

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5530 bytes --]

On Fri, Nov 03, 2017 at 04:11:46PM +0200, Pantelis Antoniou wrote:
> Hi Rob,
> 
> > On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> > 
> > On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >> Hi Pantelis and Rob,
> >> 
> >> After the workshop next week, I'm trying to capture the direction
> >> we're going for the schema format. Roughly I think we're aiming
> >> towards:
> >> 
> >> - Schema files to be written in YAML
> >> - DT files shall remain written in DTS for the foreseeable future.
> >> YAML will be treated as an intermediary format
> >>  - That said, we'll try to make design decisions that allow YAML to
> >> be used as a source format.
> >> - All schema files and yaml-encoded-dt files must be parsable by stock
> >> YAML parsers
> >> - Schema files to use the jsonschema vocabulary
> >>  - (jsonschema assumes json files, but YAML is a superset so this will be okay)
> >>  - Extended to add vocabulary for DT concepts (ignored by stock validators)
> >>    - C-like expressions as used in Pantelis' yamldt could be added in this way
> >>  - Need to write a jsonschema "metaschema" do define DT specific extensions
> >>    - metaschema will be used to validate format of schema files
> >>    - Existing tools can confirm is schema files are in the right format.
> >>    - will make review a lot easier.
> > 
> > I want to start small here with defining top-level board/soc bindings.
> > This is essentially just defining the root node compatible strings.
> > Seems easy enough, right? However, I quickly run into the problem of
> > how to match for when to apply the schema. "compatible" is the obvious
> > choice, but that's also what I'm checking. We can't key off of what we
> > are validating. So we really need 2 schema. The first is for matching
> > on any valid compatible for board, then 2nd is checking for valid
> > combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
> > don't like that as we'd be listing compatibles twice. An alternative
> > would be we apply every board schema and exactly 1 must pass. Perhaps
> > we generate a schema that's a "oneOf" of all the boards? Then we just
> > need to tag board schemas in some way.
> > 
> 
> I’ve run into this as the first problem with validation using compatible properties.
> 
> The way I’ve solved it is by having a ‘selected’ property that is generating
> a test for when to check a binding against a node.
> 
> For a concrete example using the spi-slave binding
> 
> —- spi-slave.yaml ---
> ...
> inherings: *device-compatible
> …
> 
> Where device compatible is:
> 
> %YAML 1.1
> ---
> device-compatible: &device-compatible
>   title: Contraint for devices with compatible properties
>   # select node for checking when the compatible constraint and
>   # the device status enable constraint are met.
>   selected: [ "compatible", *device-status-enabled ]
> 
>   class: constraint
>   virtual: true
> 
> The selected property here declares that any “compatible” property constraint
> for the node under test must match and that the device-status-enabled
> rule matches too.

I think the basic idea of having a "selector" that says when the
binding applies is correct.  However, I think the details of this
verison (AFAICT) are a bit too ad hoc.

I think of it in terms of "patterns" which a node might or might not
match.  Those can be simple {compatible includes "blah"} or complex -
a full set of binding requirements.

Bindings will generally be of the form:
	if (PatternA) matches
		then (PatternB) must match

e.g.
	if (compatible includes "blah")
		then ("blah" binding details) must match

or
	if ("interrupts" exists)
		then (irq binding must apply)

But in order to keep things general, both the "selector" and the
"requirements" patterns should be specified in the same way.

I had the feeling that jsonschema worked that way from Grant's talk,
although I'm not 100% certain.

> 
> The device-status-enabled is declared as follows:
> 
> %YAML 1.1
> ---
> device-enabled:
>   title: Contraint for enabled devices
>  
>   class: constraint
>   virtual: true
>   
>   properties:
>     status: &device-status-enabled
>       category: optional
>       type: str
>       description: Marks device state as enabled
>       constraint: |
>         !exists || streq(v, "okay") || streq(v, "ok”)
> 
> This declares that the constraint matches when either the optional
> status property is not present or if it is, it’s either “okay” or “ok”.
> 
> Taken together when the spi-slave binding is inherited (for instance
> by the jedec,spi-nor binding:
> 
> — jedec,spi-nor.yaml —
> 
> …
> inherits: *spi-slave
> 
> …
>   compatible: &jedec-spi-nor-compatible
>   category: required
>   type: strseq
>   constraint: | 
>     anystreq(v,  "at25df321a") ||
>      ….
> 
> 
> > I'm thinking of this very specific case, but no doubt it is going to
> > apply in other places. This has been one of the harder problems with
> > writing dtc checks. What to key off of to trigger checks and how to
> > check errors in the key itself.
> > 
> > Rob
> 
> Regards
> 
> — Pantelis
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Next steps for schema language

[Grant Likely]

On Tue, Nov 14, 2017 at 12:11 AM, David Gibson
<david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> On Fri, Nov 03, 2017 at 04:11:46PM +0200, Pantelis Antoniou wrote:
>> Hi Rob,
>>
>> > On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>> >
>> > On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> >> Hi Pantelis and Rob,
>> >>
>> >> After the workshop next week, I'm trying to capture the direction
>> >> we're going for the schema format. Roughly I think we're aiming
>> >> towards:
>> >>
>> >> - Schema files to be written in YAML
>> >> - DT files shall remain written in DTS for the foreseeable future.
>> >> YAML will be treated as an intermediary format
>> >>  - That said, we'll try to make design decisions that allow YAML to
>> >> be used as a source format.
>> >> - All schema files and yaml-encoded-dt files must be parsable by stock
>> >> YAML parsers
>> >> - Schema files to use the jsonschema vocabulary
>> >>  - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>> >>  - Extended to add vocabulary for DT concepts (ignored by stock validators)
>> >>    - C-like expressions as used in Pantelis' yamldt could be added in this way
>> >>  - Need to write a jsonschema "metaschema" do define DT specific extensions
>> >>    - metaschema will be used to validate format of schema files
>> >>    - Existing tools can confirm is schema files are in the right format.
>> >>    - will make review a lot easier.
>> >
>> > I want to start small here with defining top-level board/soc bindings.
>> > This is essentially just defining the root node compatible strings.
>> > Seems easy enough, right? However, I quickly run into the problem of
>> > how to match for when to apply the schema. "compatible" is the obvious
>> > choice, but that's also what I'm checking. We can't key off of what we
>> > are validating. So we really need 2 schema. The first is for matching
>> > on any valid compatible for board, then 2nd is checking for valid
>> > combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>> > don't like that as we'd be listing compatibles twice. An alternative
>> > would be we apply every board schema and exactly 1 must pass. Perhaps
>> > we generate a schema that's a "oneOf" of all the boards? Then we just
>> > need to tag board schemas in some way.
>> >
>>
>> I’ve run into this as the first problem with validation using compatible properties.
>>
>> The way I’ve solved it is by having a ‘selected’ property that is generating
>> a test for when to check a binding against a node.
>>
>> For a concrete example using the spi-slave binding
>>
>> —- spi-slave.yaml ---
>> ...
>> inherings: *device-compatible
>> …
>>
>> Where device compatible is:
>>
>> %YAML 1.1
>> ---
>> device-compatible: &device-compatible
>>   title: Contraint for devices with compatible properties
>>   # select node for checking when the compatible constraint and
>>   # the device status enable constraint are met.
>>   selected: [ "compatible", *device-status-enabled ]
>>
>>   class: constraint
>>   virtual: true
>>
>> The selected property here declares that any “compatible” property constraint
>> for the node under test must match and that the device-status-enabled
>> rule matches too.
>
> I think the basic idea of having a "selector" that says when the
> binding applies is correct.  However, I think the details of this
> verison (AFAICT) are a bit too ad hoc.
>
> I think of it in terms of "patterns" which a node might or might not
> match.  Those can be simple {compatible includes "blah"} or complex -
> a full set of binding requirements.
>
> Bindings will generally be of the form:
>         if (PatternA) matches
>                 then (PatternB) must match
>
> e.g.
>         if (compatible includes "blah")
>                 then ("blah" binding details) must match
>
> or
>         if ("interrupts" exists)
>                 then (irq binding must apply)
>
> But in order to keep things general, both the "selector" and the
> "requirements" patterns should be specified in the same way.
>
> I had the feeling that jsonschema worked that way from Grant's talk,
> although I'm not 100% certain.

Yes, but it is only in the pre-release draft version at the moment.
jsonschema is proposing if/then/else keywords which behave pretty much
exactly as you describe. There is good discussion here:

https://github.com/json-schema-org/json-schema-spec/issues/180#issuecomment-266986089

g.

>
>>
>> The device-status-enabled is declared as follows:
>>
>> %YAML 1.1
>> ---
>> device-enabled:
>>   title: Contraint for enabled devices
>>
>>   class: constraint
>>   virtual: true
>>
>>   properties:
>>     status: &device-status-enabled
>>       category: optional
>>       type: str
>>       description: Marks device state as enabled
>>       constraint: |
>>         !exists || streq(v, "okay") || streq(v, "ok”)
>>
>> This declares that the constraint matches when either the optional
>> status property is not present or if it is, it’s either “okay” or “ok”.
>>
>> Taken together when the spi-slave binding is inherited (for instance
>> by the jedec,spi-nor binding:
>>
>> — jedec,spi-nor.yaml —
>>
>> …
>> inherits: *spi-slave
>>
>> …
>>   compatible: &jedec-spi-nor-compatible
>>   category: required
>>   type: strseq
>>   constraint: |
>>     anystreq(v,  "at25df321a") ||
>>      ….
>>
>>
>> > I'm thinking of this very specific case, but no doubt it is going to
>> > apply in other places. This has been one of the harder problems with
>> > writing dtc checks. What to key off of to trigger checks and how to
>> > check errors in the key itself.
>> >
>> > Rob
>>
>> Regards
>>
>> — Pantelis
>>
>
> --
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Re: Next steps for schema language

[Grant Likely]

On Tue, Nov 7, 2017 at 3:34 PM, Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> On Tue, Nov 7, 2017 at 7:46 AM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
>> On Mon, Nov 6, 2017 at 4:12 PM, Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>> On Fri, Nov 3, 2017 at 9:41 AM, Pantelis Antoniou
>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>> Hi Rob,
>>>>
>>>>> On Nov 3, 2017, at 16:31 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>>>
>>>>> On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
>>>>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>>>> Hi Rob,
>>>>>>
>>>>>>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
>>>>>>>
>>>>>>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
>>>>>>>> Hi Pantelis and Rob,
>>>>>>>>
>>>>>>>> After the workshop next week, I'm trying to capture the direction
>>>>>>>> we're going for the schema format. Roughly I think we're aiming
>>>>>>>> towards:
>>>>>>>>
>>>>>>>> - Schema files to be written in YAML
>>>>>>>> - DT files shall remain written in DTS for the foreseeable future.
>>>>>>>> YAML will be treated as an intermediary format
>>>>>>>> - That said, we'll try to make design decisions that allow YAML to
>>>>>>>> be used as a source format.
>>>>>>>> - All schema files and yaml-encoded-dt files must be parsable by stock
>>>>>>>> YAML parsers
>>>>>>>> - Schema files to use the jsonschema vocabulary
>>>>>>>> - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>>>>>>>> - Extended to add vocabulary for DT concepts (ignored by stock validators)
>>>>>>>>   - C-like expressions as used in Pantelis' yamldt could be added in this way
>>>>>>>> - Need to write a jsonschema "metaschema" do define DT specific extensions
>>>>>>>>   - metaschema will be used to validate format of schema files
>>>>>>>>   - Existing tools can confirm is schema files are in the right format.
>>>>>>>>   - will make review a lot easier.
>>>>>>>
>>>>>>> I want to start small here with defining top-level board/soc bindings.
>>>>>>> This is essentially just defining the root node compatible strings.
>>>>>>> Seems easy enough, right? However, I quickly run into the problem of
>>>>>>> how to match for when to apply the schema. "compatible" is the obvious
>>>>>>> choice, but that's also what I'm checking. We can't key off of what we
>>>>>>> are validating. So we really need 2 schema. The first is for matching
>>>>>>> on any valid compatible for board, then 2nd is checking for valid
>>>>>>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
>>>>>>> don't like that as we'd be listing compatibles twice. An alternative
>>>>>>> would be we apply every board schema and exactly 1 must pass. Perhaps
>>>>>>> we generate a schema that's a "oneOf" of all the boards? Then we just
>>>>>>> need to tag board schemas in some way.
>>
>> Creating a big top level schema that includes every board as a "oneOf"
>> is a non-starter for me. It gets unwieldy in a hurry and doesn't
>> account for how to bring in device bindings.
>>
>> I'm working with the model of loading all the schema files
>> individually and iterating over all the nodes in the DT. For each
>> node, check which schemas are applicable (sometimes more than one) and
>> use them to validate the node. All applicable schemas must pass.
>>
>> An upshot of this model is that bindings don't need to define
>> absolutely everything, only what isn't covered by more generic
>> schemas. For instance, bindings don't need to define the format of
>> interrupts, #*-cells, reg, etc because the core schema already defines
>> those. Instead they only need to list the properties that are
>> required, and can add constraints on the values in standard
>> properties.
>>
>>>>>> I’ve run into this as the first problem with validation using compatible properties.
>>>>>>
>>>>>> The way I’ve solved it is by having a ‘selected’ property that is generating
>>>>>> a test for when to check a binding against a node.
>>>>>
>>>>> Okay, but what's the "jsonschema way" to do this is my question really.
>>
>> The most recent pre-release jsonschema draft defines if/then/else[1]
>> keywords for conditional validation, but I'm using a draft-4 validator
>> which doesn't implement that. Instead I did something similar to
>> Pantelis by adding a "select" property that contains a schema. If the
>> select schema matches, then the DT node must match the entire schema.
>>
>> [1] http://json-schema.org/work-in-progress/WIP-jsonschema-validation.html#rfc.section.6.6
>>
>> the "jsonschema way" would also be to compose a single schema that
>> validates the entire document, but that doesn't work in the DT context
>> simply because we are going to have a tonne of binding files. It will
>> be unmanageable to create a single overarching schema that explicitly
>> includes all of the individual device binding files into a single
>> validator instance.
>
> And the jsonschema way would be with $ref, right? I can see both ways
> being needed. For example, an I2C controller binding would want to
> pull in an I2C bus schema. Otherwise, you'd need some tag to mark the
> binding as an I2C bus and then apply the I2C bus based on that.

Yes, one schema can pull in another using $ref, and the validator will
need to handle both cases:
1) iterating through the whole tree and applying the schemas
appropriate to each node
2) one schema including another, like in the case of an i2c controller
pulling in the i2c bus schema.

>> Instead, I think the validator tool needs to load a directory of
>> binding files and be intelligent about when each one needs to be
>> applied to a node (such as keyed off compatible). That's what I'm
>> doing with the prototype code I pushed out yesterday. The validator
>> loads all the schema files it can find and then iterates over the
>> devicetree. When a node validates against the "select" schema, then it
>> checks the entire schema against the node. For example:
>>
>> %YAML 1.1
>> ---
>> id: "http://devicetree.org/schemas/soc/example.yaml#"
>> $schema: "http://json-schema.org/draft-04/schema#"
>> version: 1
>> title: ARM Juno boards
>> description: >
>>   A board binding example. Matches on a top-level compatible string and model.
>>
>> # this binding is selected when the compatible property constraint matches
>> select:
>>   required: ["compatible", "$path"]
>>   properties:
>>     $path: { const: "/" }
>
> See my pull request. This isn't actually working. :)

Right, still got some things to fixup in terms of making sure the
schema is actually getting tested.

[...]
>> I modified this one a bit to show how it would work with the select
>> property. In this case the binding matches against two possible
>> compatible strings, but the properties list also enforces
>> "example,soc" to appear in the compatible list.
>>
>> # this binding is selected when the compatible property constraint matches
>> select:
>>   required: ["compatible"]
>>   properties:
>>     compatible:
>>       contains:
>>         enum: [ "example,board", "example,board2" ]
>>
>> properties:
>>   # The "select" keyword above already ensures the board compatible is in the
>>   # list. This entry makes sure the soc compatible string is also there. It is
>>   # also a place to put the description for documentation purposes.
>>   compatible:
>>     contains:
>>         const: "example,soc"
>
> I really dislike the split here with the duplication of the data. You
> don't check the case where you only have the fallback (a common
> error).

I've yet to come up with a way to merge them into a single statement
that doesn't descend into special cases. I agree that the split isn't
ideal, but I really like the model of the condition and the additional
requirements both being regular schema blocks. It means we've already
got all the language required for deciding when to apply a schema, and
it isn't limited to a small subset of scenarios.

> We also need to check order.
>
>   allOf:
>     contains:
>       const: "example,soc"
>     contains:
>       enum: [ "example,board", "example,board2" ]
>
> BTW, this is also silently ignored because "contains" is a v6 keyword
> and only v4 is supported. While silently ignoring in some contexts is
> a feature, I think here that is a problem.

Order is indeed a problem. We would need a custom keyword to check for that.

I don't mind silently ignoring if someone chooses to parse and
validate with a different validator that uses an older version of the
spec, but for the "official" tooling that we'll use for the kernel
there should be a minimum version required to make sure those keywords
are handled.

>
>>> First, it is more verbose than I'd like and not a language immediately
>>> intuitive to low-level C developers (at least for me). My first
>>> mistake was that *Of values have to be schema objects when I really
>>> want logic ops for values.
>>
>> Yes, more verbose that I would like too, but I have yet to come across
>> anything much better. I think defining an extensible schema language
>> is just hard and it brings with it a learning curve. Every schema
>> system I looked at has the same problem. No matter what we do we're
>> going to have the pain of it not being intuitive to people used to
>> programming in C.
>
> Did you consider fragments of python or some other language? Not
> exactly sure what that would look like, but some subset of the
> $language with a set of DT specific functions that can be called.
> Perhaps a language that cares about whitespace like python is not a
> good thing to embed in YAML/JSON.  I like the syntax of Pantelis' eBPF
> constraints (being C like), it's really only the choice of backend
> that I'm questioning there. Giving what everyone has said is a
> non-goal (in kernel validation) as the primary reason for eBPF hasn't
> helped the cause.

I hadn't before now, but yaml does support block text encoding, so
there is a way to do it while preserving correct whitespace. If I'm
reading things correctly, a block of python code would look something
like this:
  code-block: |
    def check_limits(value):
      return (value > 1) && (value < 5);

A quick test with the python interpreter seems to do the right thing
with the above snippet.

> Have you looked at the $data keyword? Not sure if that would help
> here. That too may be a v6 thing.

Hmmm, $data has potential. I need to dig into it further. However, it
looks like the $data proposal has been removed from draft-07 and is
deferred to draft-future[1].

[1] https://github.com/json-schema-org/json-schema-spec/issues/51

>
> At least for this case, we simply have N lists of data and need to
> reference the lists in 2 different ways. We need to match on any
> element from the N lists and then validate we have only one element
> from each list and in list order. The binding writer should only have
> to supply those lists. IMO, having the data listed twice and having
> boilerplate that's longer than the data are non-starters. It's one
> thing for some vendor specific binding to have some verbose
> jsonschema, but it's another thing for something that will be in every
> single binding document.

Right, including the list twice doesn't work. Let me have a think on
how to fix that.

> Another case that comes to mind is how to implement generic checks for
> I2C and SPI buses. Other than node name (which is often not the
> generic name), we don't have any properties to match on (for many
> common bindings we can use "#*-cells" properties). That leaves us with
> needing a list of I2C controller compatible strings to match on.

This would be a case where the device-specific binding for the i2c/spi
bus would need to invoke the generic binding. I think it would look
something like this:

allOf:
   { $ref: "#/reference/to/spi/binding"},
   { ... device specific binding in this block ... },

Any schema pulling in another schema could use the same form.

>
>> For constant values, the const and enum properties seem to be most
>> concise way to specify a specific value using stock jsonschema. We can
>> however define new keywords for DT specific validation. A stock
>> validator will ignore them, but a DT aware validator can use them to
>> do more complete validation.
>
> I need to study how to do that. Have you found examples doing that?

In python jsonschema, I believe the way to extend the validator is to
use jsonschema.validators.extend(). Haven't had a chance to play with
it though. :-(

http://python-jsonschema.readthedocs.io/en/latest/creating/

g.

Re: Next steps for schema language

[Grant Likely]

On Thu, Nov 9, 2017 at 6:48 AM, David Gibson
<david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> On Tue, Nov 07, 2017 at 02:14:06PM +0000, Grant Likely wrote:
>> On Thu, Nov 2, 2017 at 6:13 PM, Pantelis Antoniou
>> > I'm working on supporting just that. Should be ready for testing in a
>> > couple of days. Do note that I think we should keep the old '*' usage
>> > for supporting the binding files that are YAML.
>>
>> Not sure what you mean here. Do you mean allowing native YAML
>> anchors/aliases in binding files to reduce duplication? If so, I think
>> that should be discouraged in favour of jsonschema's native $ref
>> keyword for referencing other nodes. It's not quite as expressive as
>> anchors/alias, but it is portable for anyone who needs to transcode
>> into strict JSON.
>
> Yeah, I'm not sure what's meant here either.  I think the anchor alias
> syntax should be supported in the standard YAML way, with the alias
> expanding to the full object referenced.  It doesn't help us for
> phandle/path references, but it's useful in its own right.
>
> Note that the problem with using alias syntax for references is purely
> on the alias side.  I don't see a reason we can't use standard YAML
> anchor syntax then something else to construct a reference to that
> anchor.

The primary issue is that only one anchor can be applied to a node.
We'd need to drop support for multiple labels on the same node. I'm
not a fan of the trick of using a reference to a node in order to put
another anchor on it. That's pretty nasty too.

>
> The $labels magic property is pretty yucky, it'd be good to avoid it
> if we can.

I think it is a pretty clean way to add metadata to a node. It doesn't
conflict with the node/property name namespace, and it is extensible
for other kinds of metadata.

>
>> >>   - A YAML style !phandle or !path type definition would work for
>> >> parsing references.
>> >
>> > !phandle is a bad name IMO. It assumes that the implementation is going
>> > to always be a cell integer containing a phandle. I think !ref is
>> > better.
>
> Uhh.. I don't understand your point here.  To output something the
> kernel will understand an integer phandle is exactly what the
> reference needs to expand to.
>
> I'm dubious about changing that implementation, but even if we did
> we'd still need a way to generate the current style for
> compatibility.  So !phandle or !phandlerref (to match the !pathref I
> think you already have) seems like a good name to me.

I don't have any strong feelings about the name. As for format, I'm
using the following for path and phandle references. In both cases the
first example uses full path, the second a label, but in both cases
the property is tagged as a reference to another node.

path:
        s1: [!path "/memory@0"]
        s1: [!path "memlabel"]
phandle:
        ref: [[!phandle "/memory@0"]]
        ref: [[!phandle "memlabel"]]




>> Consider an unresolved tree that has successive trees applied to
>> different levels:
>>
>> / { ... };
>> &etm0 { ... };
>> &etm1 { ... };
>> / { ... };
>> /aliases { ... };
>>
>> To transcode this into YAML the path/reference needs to be stored
>> somewhere. As already discussed, it cannot be a map because keys can
>> appear more than once and order of application matters, so it must be
>> a list. Some possible options for storing the path/reference in the
>> array structure are:
>
> Seems to me you want a type-tagged map for each overlay component,
> something like
>
> !overlayfragment {
>         target-label = "etm0";
>         content = { /* the actual nodes */ };
> }
>
>>
>> Store the path/tree pair as a tuple (an array of arrays)
>> - [ / , {...} ]
>> - [ &etm0, {...} ]
>> - [ &etm1 , {...} ]
>> - [ / , {...} ]
>> - [ /aliases , {...} ]
>
> Works for now, but limited if we ever need more metadata for an
> overlay/fragment.
>
>> Or it could be stored as a special property in the node, something
>> that doesn't collide with child/property names. An array of maps:
>>
>> - $path: "/"
>>   ...
>> - $path: "&etm0"
>>   ...
>> - $path: "&etm1"
>>   ...
>> - $path: "/"
>>   ...
>> - $path: "/aliases"
>>   ...
>>
>> Personally, I prefer embedding the path right into the node because it
>> drops a level of nesting.
>
> I dislike this because it mixes "in band" and "out of band" info,
> requiring more special handling in tools to separate them again.  I
> really think a type-tagged map is the right (and YAML-ish) way to do
> this.

Using the same argument, my first take on node format is to separate
nodes and properties because theoretically they are in different
namespaces. Something like this:
node@0:
    $props:
        compatible: [ acme,toaster ]
        reg: [ [0] ]
    child1: { ... }
    child2: { ... }

But I really dislike having the extra level of indentation, and it
turns out it isn't difficult to parse & validate when only one level
is used. Nodes are always maps, properties are always arrays, and the
types do not overlap. (Besides, I think it is a good thing to stop
allowing a node and a property to share the same name). So for this
case, I think it is worth dropping $props to reduce the nesting level
for properties.

For the metadata case, I've got the same leaning. Since there is no
overlap between metadata properties ($...) and node/property
properties (ugh, namespace clash between yaml & dt), I think it is
better to have the metadata at the same level.

g.

Re: Next steps for schema language

[Grant Likely]

Hi Pantelis,

Thanks for the update. Comments below...

On Thu, Nov 9, 2017 at 10:16 AM, Pantelis Antoniou
<pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Hi Grant,
>
> Just pushed a few changes to yamldt which among other things enables
> JSON support. Note that JSON is strictly an output format for now.
>
> Notable features are:
>
> - JSON output is now supported.
>
> - '$' is supported as well as a reference indicator on bare scalars
> similar to '*'
>
>     property: *ref
>     property: $ref
>
> are equivalent.
>
> - Labels can be declared using the /label/ special property.
> The use of /label/ was used since it's similar to the way /memreserve/
> is a special keyword. Internally it's the same as declaring an
> anchor.

I'm not a fan of the /.../ style for the yaml encoding. It was part of
the tokenization in the .dts format, but is not necessary here. I
prefer a simpler form like '$labels'. I think prefixing metadata
property names with '$' because the YAML parser can handle $ without
special escaping, and $ does not conflict with the DT node/property
namespace.

>
>   foo: &label
>   foo:
>     /label/: label
>
> You can also use a sequence for multiple label declaration in one go.
>
>   foo:
>     /label/: [ label1, label2 ]

The $labels property should always be a sequence, regardless of if
there is one or more labels. Otherwise a parser needs to know about
both forms.

> - !ref as a long form method for a reference:
>
>   foo: *label
>   foo: $label
>   foo: !ref label
>
> Are all equivalent.

The *label and $label forms should be dropped entirely, and only the
tagged !ref form be supported. *label is ambiguous for a reader as to
whether it is a DT reference or an alias. $label doesn't differentiate
between a reference and a string that just happens to start with '$'.
The !ref form is unambiguous as to what it is for.

>
> - JSON does not support tags; so yaml tags are transformed to a tuple
> sequence:
>
>   foo: !int16 10
>
>   foo: [ "\f!int16", 10 ]
>
> The form feed escape and ! at the start of a string scalar allows
> detection that this is a type declaration. This is arguably ugly but
> then again, JSON is not meant to be used as a source format.

so... this is problematic because it makes the json data model
different from the yaml one. The offsets to datum in a property array
are different depending on if it is loaded from a json file or a yaml
file. In the json case, because it is a simpler format which doesn't
have the ability to specify type, I think integer data needs to be
emitted as plain bytes because that is the lowest common denominator.

If we're going through json, I don't think we need to try and preserve
all the datatype information. It would only be for communicating with
really-simple-software. (Yes, I know that's not the direction I was
going before, but JSON just is too limited a format to consider
anything other than a lossy output)

>
> On Tue, 2017-11-07 at 14:14 +0000, Grant Likely wrote:
>> On Thu, Nov 2, 2017 at 6:13 PM, Pantelis Antoniou
>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> > Hi Grant,
>> >
>> > Mostly agree, some notes below.
>> >
>> > On Thu, 2017-11-02 at 16:44 +0000, Grant Likely wrote:
>> >> Hi Pantelis and Rob,
>> >>
>> >> After the workshop next week, I'm trying to capture the direction
>> >> we're going for the schema format. Roughly I think we're aiming
>> >> towards:
>> >>
>> >> - Schema files to be written in YAML
>> >> - DT files shall remain written in DTS for the foreseeable future.
>> >> YAML will be treated as an intermediary format
>> >>   - That said, we'll try to make design decisions that allow YAML to
>> >> be used as a source format.
>> >> - All schema files and yaml-encoded-dt files must be parsable by stock
>> >> YAML parsers
>> >> - Schema files to use the jsonschema vocabulary
>> >>   - (jsonschema assumes json files, but YAML is a superset so this will be okay)
>> >>   - Extended to add vocabulary for DT concepts (ignored by stock validators)
>> >>     - C-like expressions as used in Pantelis' yamldt could be added in this way
>> >>   - Need to write a jsonschema "metaschema" do define DT specific extensions
>> >>     - metaschema will be used to validate format of schema files
>> >>     - Existing tools can confirm is schema files are in the right format.
>> >>     - will make review a lot easier.
>> >>
>> >> The yaml encoding produced by yamldt is a good start, but some changes
>> >> need to be made from the current code to be workable:
>> >> - The redefinition of the anchor/alias syntax needs to be dropped.
>> >>   - We need a labels/reference syntax instead.
>> >>   - I'm using a $labels property to contain a list of labels which is
>> >> working for me, but I'm open to other suggestions
>> >
>> > I'm working on supporting just that. Should be ready for testing in a
>> > couple of days. Do note that I think we should keep the old '*' usage
>> > for supporting the binding files that are YAML.
>>
>> Not sure what you mean here. Do you mean allowing native YAML
>> anchors/aliases in binding files to reduce duplication? If so, I think
>> that should be discouraged in favour of jsonschema's native $ref
>> keyword for referencing other nodes. It's not quite as expressive as
>> anchors/alias, but it is portable for anyone who needs to transcode
>> into strict JSON.
>>
>
> Sorry, didn't express my self clearly there. I meant using $ as a
> reference prefix.
>
>> >>   - A YAML style !phandle or !path type definition would work for
>> >> parsing references.
>> >
>> > !phandle is a bad name IMO. It assumes that the implementation is going
>> > to always be a cell integer containing a phandle. I think !ref is
>> > better.
>>
>> Sure, I'm fine with that
>>
>
> Done.
>
>> > Note that we must come with a way to encode types in JSON, since
>> > they are not natively supported.
>>
>> Agreed.
>>
>
> So, I've come up with a way ^^^
>
> Not very aesthetically pleasing though. I'm open to suggestions.
>
>> >
>> >> - At the top level, the yaml-encoded-dt needs to be structured as a
>> >> list, not a map.
>> >>   - Using a list will properly accounts for how multiple top level
>> >> trees are overlayed to create the final tree.
>> >
>> >
>> > This is true only for non-resolved files. Resolved files are going to
>> > be guaranteed a map. Does this cause problems for your validator?
>>
>> Make the resolved file a list with only one entry at the top level,
>> then the problem goes away.
>>
>
> Err, isn't that a bit of a hack? :)

Not really. Having the top level be a list of trees means the exact
same structure is used regardless of whether it is a resolved or an
unresolved tree. However, I have softened on this. The majority of the
validator will work on a resolved tree. It will get passed just the
tree portion, not the entire structure. Also, since yaml won't be the
source format for the foreseeable future, any decision made now can be
changed later.

>> Regardless, I'm finding that the validator needs to walk the tree(s)
>> and be able to apply a binding at any level, so driver bindings won't
>> be affected by the structure at the top level. Board and SoC bindings
>> might.
>>
>
> Hmm, definitely for board and SoC bindings.
>
>> >>   - I'm using a $path property to encode the path of each top level
>> >> node. Again, I'm open to suggestions for a different approach
>> >>
>> >
>> > A bare scalar that starts with / can always be deduced to be a path.
>>
>> I don't understand what you mean. That scalar still needs to be
>> encoded in some way.
>>
>
> OK, let me explain.
>
> In yaml strings need not be quoted if they are 'simple'
>
> So:
>
>   property: "foo"
>   property: foo
>
> Are equivalent, but the first one is a quoted string while the second is
> a bare scalar that since it's not null, boolean, or a number, is a
> string.
>
> We can use this to deduce that a bare string that looks like a path is a
> path.
>
>   path-property: !path "/foo"
>   path-property: /foo
>
> We can safely deduce that these are equivalent.

No, we cannot because we don't know what it is a path for! Is it a
Linux path or a devicetree path? Without knowing the binding for the
property, we cannot deduce if the path refers to another node in the
tree. It might just be data used by the driver that happens to start
with a '/'.

But regardless, that's not what I'm talking about with the $path
property. I've been using $path as a way to encode where a top level
node lives in the tree. This of course is only an issue for unresolved
trees. Resolved trees only have one top level node, and we know it
lives at "/". It is only when loading a .dts file with multiple top
level nodes does the location of each node need to be stored in $path.

>
> If you're doing this in JSON, then you have no other way than what
> you're using now.
>
>> > Can you share some examples about your usage?
>>
>> Consider an unresolved tree that has successive trees applied to
>> different levels:
>>
>> / { ... };
>> &etm0 { ... };
>> &etm1 { ... };
>> / { ... };
>> /aliases { ... };
>>
>> To transcode this into YAML the path/reference needs to be stored
>> somewhere. As already discussed, it cannot be a map because keys can
>> appear more than once and order of application matters, so it must be
>> a list. Some possible options for storing the path/reference in the
>> array structure are:
>>
>> Store the path/tree pair as a tuple (an array of arrays)
>> - [ / , {...} ]
>> - [ &etm0, {...} ]
>> - [ &etm1 , {...} ]
>> - [ / , {...} ]
>> - [ /aliases , {...} ]
>>
>> Or it could be stored as a special property in the node, something
>> that doesn't collide with child/property names. An array of maps:
>>
>> - $path: "/"
>>   ...
>> - $path: "&etm0"
>>   ...
>> - $path: "&etm1"
>>   ...
>> - $path: "/"
>>   ...
>> - $path: "/aliases"
>>   ...
>>
>> Personally, I prefer embedding the path right into the node because it
>> drops a level of nesting.
>>
>
> Expanding your example:
>
> / {
>   one;
>   etm0: etm0 { etm0_one; };
>   etm1: etm1 { etm1_one; };
>   aliases { nada; }
> };
> &etm0 { two; };
> &etm1 { three; };
> / { four; };
> /aliases { five; };
>
> For yamldt would encode it as:
>
> one: true
> etm0: &etm0
>   etm0_one: true
> etm1: &etm1
>   etm1_one: true
> aliases:
>   nada: true
> *etm0:
>   two: true
> *etm1:
>   three: true
> four: true
> /aliases:
>   five: true

> The resulting yaml output would be:
>
> one: true
> four: true
> etm0:
>   etm0_one: true
>   two: true
> etm1:
>   etm1_one: true
>   three: true
> aliases:
>   nada: true
>   five: true

Yes, I have no issue with this yaml encoding of the resolved tree.

> Note that the output file is proper YAML, it's the source that we are
> talking about.
>
> I think a very simple way to encode it would be (using $ for ref and
> labels with the new style):
>
> - /:
>     one: true
>     etm0:
>       /label/: etm0
>       etm0_one: true
>     etm1:
>       /label/: etm1
>       etm1_one: true
>     aliases:
>       nada: true
> - $etm0:
>     two: true
> - $etm1:
>     three: true
> - /:
>     four: true
> - /aliases:
>     five: true

I've been encoding the same structure like this:

unresolved_tree:
- $path: /
  one: true
  etm0:
    $labels: [ etm0 ]
    etm0_one: true
  etm1:
    $labels: [ etm1 ]
    etm1_one: true
 aliases:
    nada: true
- $path: !ref etm0
  two: true
- $path: !ref etm1
  three: true
- $path: /
  four: true
- $path: /aliases:
  five: true

The difference between our two approaches is:

Your structure:
  [ { (node ref/path): { prop1: val, prop2: val, ... } } ]
My structure:
  [ { $path: (node ref/path), prop1: val, prop2: val, ... } ]

The difference is the amount of nesting. You're using two maps, and
the outer one has only one key; the node ref. I'm using a single map
and a special property to encode the node ref. Visually the two maps
looks better, but in terms of accessing the data I far prefer the
single map approach.

>
> You can even get rid of the sequence indicators and keep the first
> one if you're sure about indentation
> - /:
>     one: true
>     etm0:
>       /label/: etm0
>       etm0_one: true
>     etm1:
>       /label/: etm1
>       etm1_one: true
>     aliases:
>       nada: true
>   $etm0:
>     two: true
>   $etm1:
>     three: true
>   /:
>     four: true
>   /aliases:
>     five: true

This is where it falls down again. The sequence indicators must be
there, otherwise it is just back to trying to use a map to encode
ordered top level trees. The ordering is lost because yaml doesn't
guarantee ordering in maps, which is needed in unresolved trees. While
it works for you in yamldt, a generic yaml parser will mess it up.

Cheers,
g.

Re: Next steps for schema language

[Grant Likely]

On Thu, Nov 2, 2017 at 4:44 PM, Grant Likely <grant.likely-s3s/WqlpOiPyB63q8FvJNQ@public.gmane.org> wrote:
> The yaml encoding produced by yamldt is a good start, but some changes
> need to be made from the current code to be workable:
> - The redefinition of the anchor/alias syntax needs to be dropped.
>   - We need a labels/reference syntax instead.
>   - I'm using a $labels property to contain a list of labels which is
> working for me, but I'm open to other suggestions
>   - A YAML style !phandle or !path type definition would work for
> parsing references.
> - At the top level, the yaml-encoded-dt needs to be structured as a
> list, not a map.
>   - Using a list will properly accounts for how multiple top level
> trees are overlayed to create the final tree.
>   - I'm using a $path property to encode the path of each top level
> node. Again, I'm open to suggestions for a different approach

I've come across another issue, this time with the property encoding.
Originally I thought to just encode properties in whatever form looks
best to us humans. So, for example a node might look like this:

uart@1000000:
    compatible: ["acme,uart", "ns16550"]  # list of strings
    model: "Fancy UART" .   # single string
    gpio-controller: true       # Boolean
    #gpio-cells: [ 2 ]            # Weird UART, it provides GPIOs.
Single integer value
    reg: [ 0xc0001000 0x1000 ];   # list of single values
    interrupt-parent: [ !phandle "intc1" ] .   # single phandle
    interrupts: [ [ 1, 0 ], [2, 0x2] [5, 0x6] ]  # 3 interrupt lines

However, I discovered it is tough to know how to go from .dts to yaml
in a consistent way because there is information missing about how it
should be encoded.

For example, should a property with a single string in it be encoded
as a single string (prop: strvalue) or as a list of strings (prop:
[strvalue]). The .dts syntax doesn't give us any clues about whether a
property is only ever going to be a single value (model,
interrupt-parent, #gpio-cells), or if it might be a list of
values/tuples (compatible, reg, interrupts). For validation, it
becomes a lot more complex if a value can be encoded in multiple ways.
For example, here are two ways to encode compatible:
        compatible: "acme,uart"
        compatible: [ "acme,uart" ]
The first is more concise, but it means the validator has to account
for both encodings.
        compatible: [ "acme,uart", "ns16550" ]

I hit a similar problem with groups of cells. While it looks better to
encode a single value without a sequence, the tool doesn't have any
information to know which properties are single values, and which are
sequences, but the validator still needs to deal with it. For example,
the following two lines are arguably equivalent, but which form should
the tooling emit?
        reg: 1
        reg: [1]

I also want to preserve grouping information as it appears in the dts.
Most of the time dts files already have things like reg and interrupt
tuples grouped for each entry. That information is very useful for
validation, and that grouping should be preserved. For example,
interrupts could be encoded as one of the following:

        (translating from dts:    interrupts = <1 0>, <2 2> <5 6>;)
        interrupts: [ 1 0 2 2 5 6 ]
        interrupts: [ [1,0], [2,2], [5,6]]

Both would be valid encodings, but the later carries information
useful for validation and helps to match what the writer intends to
the schema being used. I want to get that into the YAML output.

So, I propose the following:
- The yaml format should always encode properties as a sequences,
regardless of whether or not it contains only a single value. That
means that consumers don't need to handle both sequence and
non-sequence variants of a property. A single value will always
dereference as a sequence containing only one value. No ambiguity. So,
a list of strings would be encoded in the form:
        str-prop: [ "str1" ]
        str-prop: [ "str1", "str2" ]
        str-prop: [ "str1", "str2", "str3" ]
- Integer values (bytes, u16, cells, u64) will always be contained in
another sequence to represent the grouping from the .dts file. For
example:
        int-prop: [ [0] ]        # int-prop = <0>;
        int-prop: [ [0, 1] ]    # int-prop = <0 1>;
        int-prop: [ [0], [2] ]        # int-prop = <0>, <2>;
        int-prop: [ [0, 1], [2, 3] ]    # int-prop = <0 1>, <2 3>;
- I'm encoding other bit sizes with a tag at the group level to match
up with what is done in the .dts files:
        bytes: [ !u8 [ 0 1 2 3 4 5 ] ]     # bytes = [ 0 1 2 3 4 5 ]
        uint16: [ !u16 [ 6 7 8 9 ] ]        # uint16 = /bits/ 16 [ 6 7 8 9 ]
        uint32: [ [ 0 1 2 3 4 5 ] ]          # uint32 = [ 0 1 2 3 4 5 ]
        uint64: [ !u16 [ 6 7 8 9 ] ]        # uint64 = /bits/ 64 [ 6 7 8 9 ]
I could instead attach the tag to each number value, which is arguably
more flexibly for future refinement, but that would result in a lot of
tags in the case of a large byte property. For example:
        mac-address: [ [ !u8 0xA0, !u8 0xB0, !u8 0xC0, !u8 0xD0, !u8
0xE0, !u8 0xF0 ] ]

Finally, an example using a mixed property:
        mixed: [ "string1", !u8 [0xde, 0xad, 0xca, 0xfe], "string2",
[0x12345678, 0x9abcdef0], !u64 [0xA000B000C000D000] ]

Using this scheme, there will only ever be one way to encode a
property and the validation code doesn't need to account for all the
different variations.

Thoughts?

Cheers,
g.

Re: Next steps for schema language

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 12447 bytes --]

On Tue, Nov 07, 2017 at 01:46:38PM +0000, Grant Likely wrote:
> On Mon, Nov 6, 2017 at 4:12 PM, Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> > On Fri, Nov 3, 2017 at 9:41 AM, Pantelis Antoniou
> > <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >> Hi Rob,
> >>
> >>> On Nov 3, 2017, at 16:31 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >>>
> >>> On Fri, Nov 3, 2017 at 9:11 AM, Pantelis Antoniou
> >>> <pantelis.antoniou-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >>>> Hi Rob,
> >>>>
> >>>>> On Nov 3, 2017, at 15:59 , Rob Herring <robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> wrote:
> >>>>>
> >>>>> On Thu, Nov 2, 2017 at 11:44 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
> >>>>>> Hi Pantelis and Rob,
> >>>>>>
> >>>>>> After the workshop next week, I'm trying to capture the direction
> >>>>>> we're going for the schema format. Roughly I think we're aiming
> >>>>>> towards:
> >>>>>>
> >>>>>> - Schema files to be written in YAML
> >>>>>> - DT files shall remain written in DTS for the foreseeable future.
> >>>>>> YAML will be treated as an intermediary format
> >>>>>> - That said, we'll try to make design decisions that allow YAML to
> >>>>>> be used as a source format.
> >>>>>> - All schema files and yaml-encoded-dt files must be parsable by stock
> >>>>>> YAML parsers
> >>>>>> - Schema files to use the jsonschema vocabulary
> >>>>>> - (jsonschema assumes json files, but YAML is a superset so this will be okay)
> >>>>>> - Extended to add vocabulary for DT concepts (ignored by stock validators)
> >>>>>>   - C-like expressions as used in Pantelis' yamldt could be added in this way
> >>>>>> - Need to write a jsonschema "metaschema" do define DT specific extensions
> >>>>>>   - metaschema will be used to validate format of schema files
> >>>>>>   - Existing tools can confirm is schema files are in the right format.
> >>>>>>   - will make review a lot easier.
> >>>>>
> >>>>> I want to start small here with defining top-level board/soc bindings.
> >>>>> This is essentially just defining the root node compatible strings.
> >>>>> Seems easy enough, right? However, I quickly run into the problem of
> >>>>> how to match for when to apply the schema. "compatible" is the obvious
> >>>>> choice, but that's also what I'm checking. We can't key off of what we
> >>>>> are validating. So we really need 2 schema. The first is for matching
> >>>>> on any valid compatible for board, then 2nd is checking for valid
> >>>>> combinations (e.g. 1 board compatible followed by 1 SoC compatible). I
> >>>>> don't like that as we'd be listing compatibles twice. An alternative
> >>>>> would be we apply every board schema and exactly 1 must pass. Perhaps
> >>>>> we generate a schema that's a "oneOf" of all the boards? Then we just
> >>>>> need to tag board schemas in some way.
> 
> Creating a big top level schema that includes every board as a "oneOf"
> is a non-starter for me. It gets unwieldy in a hurry and doesn't
> account for how to bring in device bindings.

I agree.

> I'm working with the model of loading all the schema files
> individually and iterating over all the nodes in the DT. For each
> node, check which schemas are applicable (sometimes more than one) and
> use them to validate the node. All applicable schemas must pass.

Yes, I think this is the right approach.  I'd elaborate to say there
are basically two ways a schema becomes applicable:

  1) The schema "self selects".  This will usually be a schema saying
     it should apply because of the value of compatible, but could
     also include things like the generic interrupts binding saying it
     applies to any nodes with an "interrupts" property.

  2) One schema which applies specifies that another schema should
     also apply.  The common case is where one schema "inherits" from
     another - the more specific schema can say when it applies, then
     force the base schema to also apply.  This is useful in cases
     where there's a general pattern for things used across a bunch of
     devices, but not a clear common comaptible value or similar to
     know when it applies.

     The more complex case of this is a schema applying to one node
     saying that another schema applies to another node.  e.g. a PCI
     bridge saying that the PCI device schema must apply to all its
     subnodes.  Or, in a more complex example still, the schema for an
     interrupt controller saying that a schema defining the specific
     format of its interrupt specifiers must apply to nodes which have
     this intc as their interrupt parent.
> 
> An upshot of this model is that bindings don't need to define
> absolutely everything, only what isn't covered by more generic
> schemas. For instance, bindings don't need to define the format of
> interrupts, #*-cells, reg, etc because the core schema already defines
> those. Instead they only need to list the properties that are
> required, and can add constraints on the values in standard
> properties.
> 
> >>>> I’ve run into this as the first problem with validation using compatible properties.
> >>>>
> >>>> The way I’ve solved it is by having a ‘selected’ property that is generating
> >>>> a test for when to check a binding against a node.
> >>>
> >>> Okay, but what's the "jsonschema way" to do this is my question really.
> 
> The most recent pre-release jsonschema draft defines if/then/else[1]
> keywords for conditional validation, but I'm using a draft-4 validator
> which doesn't implement that. Instead I did something similar to
> Pantelis by adding a "select" property that contains a schema. If the
> select schema matches, then the DT node must match the entire schema.
> 
> [1] http://json-schema.org/work-in-progress/WIP-jsonschema-validation.html#rfc.section.6.6
> 
> the "jsonschema way" would also be to compose a single schema that
> validates the entire document, but that doesn't work in the DT context
> simply because we are going to have a tonne of binding files. It will
> be unmanageable to create a single overarching schema that explicitly
> includes all of the individual device binding files into a single
> validator instance.
> 
> Instead, I think the validator tool needs to load a directory of
> binding files and be intelligent about when each one needs to be
> applied to a node (such as keyed off compatible). That's what I'm
> doing with the prototype code I pushed out yesterday. The validator
> loads all the schema files it can find and then iterates over the
> devicetree. When a node validates against the "select" schema, then it
> checks the entire schema against the node. For example:
> 
> %YAML 1.1
> ---
> id: "http://devicetree.org/schemas/soc/example.yaml#"
> $schema: "http://json-schema.org/draft-04/schema#"
> version: 1
> title: ARM Juno boards
> description: >
>   A board binding example. Matches on a top-level compatible string and model.
> 
> # this binding is selected when the compatible property constraint matches
> select:
>   required: ["compatible", "$path"]
>   properties:
>     $path: { const: "/" }
>     compatible:
>       contains:
>         enum: [ "arm,juno", "arm,juno-r1", "arm,juno-r2" ]
> 
> required:
> - model
> - psci
> - cpus
> 
> properties:
>   model:
>     enum:
>       - "ARM Juno development board (r1)"
>       - "ARM Juno development board (r2)"
> 
> This is a board level binding file for the Juno board. There are three
> important top level properties:
> == select ==
> Contains a schema. If the node is at the root ($path=='/') and
> compatible is one of the juno boards, then this binding applys
> 
> == required ==
> List of properties/nodes that must be present. In this case model,
> psci, and cpus. compatible isn't listed because it is already
> guaranteed to be present because it was in the select node. Also note
> that the contents of the nodes/properties doesn't have to be
> specified. The format of a lot of standard properties will already be
> validated by the core DT schema.
> 
> For example, model must always be a simple string.
> 
> == properties ==
> Schemas for specific properties can go here. In this case I've
> constrained model to contain one of two strings, and in the test repo
> this demonstrates a validation failure because the juno.cpp.dts
> contains (r0) instead of (r1) or (r2).
> 
> 
> >> No idea :)
> >>
> >> DT is weird enough that there might not be a way to describe this in
> >> a regular jsonschema form. I would wait until Grant pitches in.
> >
> > I've played around with things a bit and the more I do the less happy
> > I am with jsonschema. Maybe this is not what Grant has in mind, but
> > here's the snippet of the compatible check I have:
> >
> > properties:
> >   compatible:
> >     description: |
> >       Compatible strings for the board example.
> >
> >     type: array
> >     items:
> >       type: string
> >       oneOf:
> >         - enum:
> >           - "example,board"
> >           - "example,board2"
> >         - enum:
> >           - "example,soc"
> 
> 
> I modified this one a bit to show how it would work with the select
> property. In this case the binding matches against two possible
> compatible strings, but the properties list also enforces
> "example,soc" to appear in the compatible list.
> 
> # this binding is selected when the compatible property constraint matches
> select:
>   required: ["compatible"]
>   properties:
>     compatible:
>       contains:
>         enum: [ "example,board", "example,board2" ]
> 
> properties:
>   # The "select" keyword above already ensures the board compatible is in the
>   # list. This entry makes sure the soc compatible string is also there. It is
>   # also a place to put the description for documentation purposes.
>   compatible:
>     contains:
>         const: "example,soc"
> 
> > First, it is more verbose than I'd like and not a language immediately
> > intuitive to low-level C developers (at least for me). My first
> > mistake was that *Of values have to be schema objects when I really
> > want logic ops for values.
> 
> Yes, more verbose that I would like too, but I have yet to come across
> anything much better. I think defining an extensible schema language
> is just hard and it brings with it a learning curve. Every schema
> system I looked at has the same problem. No matter what we do we're
> going to have the pain of it not being intuitive to people used to
> programming in C.
> 
> For constant values, the const and enum properties seem to be most
> concise way to specify a specific value using stock jsonschema. We can
> however define new keywords for DT specific validation. A stock
> validator will ignore them, but a DT aware validator can use them to
> do more complete validation.
> 
> > Second, the constraints are not complete
> > and and I've not come up with how you would express them. Essentially,
> > we need to express at least one of each set is required and
> > "example,soc" must be last. I suppose we can come up with custom
> > expressions, but it seems to me that if we can't even express a simple
> > example like this with standard jsonschema then it is not a good
> > choice.
> 
> If the compatible string was a known size then ordering can be
> enforced using the items property, but there isn't anything in the
> spec or proposed for enforcing order in arbitrarily sized arrays. It
> would need to be an extension.
> 
> I don't think that makes jsonschema as a whole a bad choice. It does a
> lot of the things we need right away, and not matter what we choose
> we're going to be poking at corner cases where the DT context doesn't
> quite fit. At the very least, I think there needs to be more examples
> converted over to see what it looks like in real world usage.
> 
> > Don't take this as we should use eBPF either. Given the reasoning so
> > far for picking it, I'm not sold on it. Seems like a nice shiny hammer
> > looking for a problem.
> >
> > Rob

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]