[RFC PATCH 0/5] DT binding documents using text markup

[RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

During the Device Tree microconference at Linux Plumbers 2015, we had
a short discussion about how to improve DT Binding Documentation. A
number of issues were raised (again, as these things have been
discussed in the past) including:

	* Inconsistency between binding documents due to prose text
	  format.
	* Inability to reliably machine read bindings for mass update
          or search.
        * Bit rot of bindings as new conventions are agreed upon but
	  only new bindings are changed.

Grant Likely probably summed up the issue best with "...as long as
bindings are human readable, we'll have issues...". The context
of that comment was, of course, regarding our current documents
written in very inconsistent prose style. When the topic of needing
the bindings in a rigid format was raised, there was general head
nodding that this was needed. It was noted that this has been
discussed many times before and nothing has been done.

My proposed solution to the problem is to convert all DT bindings
a rigid text markup format. In choosing a text markup language my
requirements were:

	1) Human readable
	2) Well documented
	3) Easy to translate to other data formats
	4) Well supported by tools and libraries

After looking at a number of markup options, YAML stood out as the
one that meets all of these requirements. The YAML syntax is adopted
in many projects specifically because of the high level of readability.
A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
There's a number of tools to convert between YAML and other popular
data formats such as JSON and XML. XML was cited by Behan Webster
during the microconference as an important data format as the type
of developers that may produce comprehensive DTS Binding validation
tools will want to use XML. Every major scripting language has a
high level binding to the low level libyaml C library to facilitate
handling of YAML data files.

One caveat with YAML is it does not tolerate tabs. Yes, I said it.
No tabs! This can be managed with proper editor modes and also with
helper scripts to strip tabs to aid in people passing planned
checkpatch.pl checks that would run YAML DT Binding specific tag
validators for new bindings.

The scope of the initial YAML DT Binding format was specifically
limited to supporting *only* the content we have in bindings today.
The idea here is to propose and agree on something that will take
us just a few steps in the right direction. If we move *all* current
binding content to a machine parseable format, additional features
can be added with more automation and scripting. As it stands today,
because of the inconsistency of the wording of the files, we can't
add a lot of new features to the content until we convert what we
have today into a standard format.

With that said, it should be noted that some new features such as
"type" tags to indicate cell types could be added to support
additional DTS validation beyond what the current content supports.
Another possibility is adding "range" type information to validate
the legal values for a cell.

This series is broken up into three major parts:

1) The documentation defining the YAML DT binding format
2) A skeleton device binding example illustrating use of this format
3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
   ti-phy.txt

As a proof of concept of what can be done with a proper machine
readable DT binding source file, there's a simple markdown document
generator at https://github.com/konsulko/dtgendoc. Also, to see
actual output from the generator, the generated markdown from those
bindings is viewable at https://github.com/konsulko/dtgendoc/wiki

There's a lot of other possibilities for validation tools using
only the data we have today in the bindings. In addition, Frank
Rowand covered some DT debug techniques that would benefit from
the binding documentation being 100% reliably searchable.

I found it useful to see a side-by-side view of a converted doc
versus the original content, so here's a screenshot of eeprom.txt
vs. eeprom.yaml:
https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml

When we decide on a text markup format that is acceptable, then the
next step is to convert all the bindings. That process would start
with the complete set of generic bindings as they will be referenced
by the actual device bindings.

If the RFC wasn't explicit enough...comments welcome.

-Matt

Matt Porter (5):
  Documentation: dt-bindings: add documentation on new DT binding format
  Documentation: dt-bindings: add example DT binding document
  Documentation: dt-bindings: add YAML eeprom binding
  Documentation: dt-bindings: phy: add YAML generic PHY binding
  Documentation: dt-bindings: phy: add YAML TI PHY binding

 .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++
 Documentation/devicetree/bindings/eeprom.yaml      |  44 ++++++
 .../devicetree/bindings/phy/phy-bindings.yaml      |  89 +++++++++++
 Documentation/devicetree/bindings/phy/ti-phy.yaml  | 166 +++++++++++++++++++++
 Documentation/devicetree/bindings/skeleton.yaml    |  98 ++++++++++++
 5 files changed, 503 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
 create mode 100644 Documentation/devicetree/bindings/eeprom.yaml
 create mode 100644 Documentation/devicetree/bindings/phy/phy-bindings.yaml
 create mode 100644 Documentation/devicetree/bindings/phy/ti-phy.yaml
 create mode 100644 Documentation/devicetree/bindings/skeleton.yaml

-- 
2.1.4

[RFC PATCH 1/5] Documentation: dt-bindings: add documentation on new DT binding format

[Matt Porter]

Documentation explaining the syntax and format of the YAML-based DT binding
documentation.

Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
---
 .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
 1 file changed, 106 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt

diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
new file mode 100644
index 0000000..f9acc22
--- /dev/null
+++ b/Documentation/devicetree/bindings/dt-binding-format.txt
@@ -0,0 +1,106 @@
+--------------------------
+Device Tree Binding Format
+--------------------------
+
+Background
+----------
+
+DT bindings historically were written as text in prose format which
+led to issues in usability of that source documentation. Some of
+these issues include the need to programmatically process binding
+source documentation to do DTS validation, perform mass updates to
+format/style, and to generate publishable documentation in HTML or
+PDF form.
+
+Overview
+--------
+
+The DT binding format is based on the YAML text markup language.
+Although there are many text markup options available, YAML
+fulfills all requirements considered for a DT binding source format
+which include:
+
+1) Must be human readable
+2) Must be easily translated to other data formats (XML, JSON, etc).
+3) Must have sufficient tools and libraries to enable developers to
+   build new tools for DT binding processing
+4) Must have a complete spec to refer to syntax
+
+YAML is documentated in the specification found at
+http://www.yaml.org/spec/1.2/spec.html
+
+The required YAML DT binding tag format and syntax are defined in
+the following sections.
+
+YAML DT Binding Syntax
+----------------------
+
+* Lines starting with "#" are comments and not part of the binding itself
+* "%YAML 1.2" starts a file, indicating the version of YAML in use
+* "---" starts a binding document
+* "..." ends a binding document
+* Multiple binding documents may exist in a single file
+* Tabs are not permitted
+* Scope is denoted by indentation of two spaces
+* Key value pairs are denoted by "key: value"
+* Sequences are denoted by "- "
+* Scalar values may convert newlines to spaces and preserve blank
+  lines for long description formatting using ">"
+* Scalar values may escape all reserved characters and preserve
+  newlines by using "|" to denote literal style
+
+For additional information on YAML syntax, refer to the specification
+at http://www.yaml.org/spec/1.2/spec.html
+
+YAML DT Binding Format
+----------------------
+
+The following YAML types are supported in the DT binding format:
+
+* [R] id: unique identifier in property form (e.g. skel-device)
+
+* [R] title: title of the binding
+
+* [O] maintainer: sequence of maintainers
+      [R] name: name and email of maintainer or mailing list in RFC822
+                form.
+
+* [O] description: full description of the binding
+
+* [O] compatible: sequence of valid compatible descriptors
+      [R] name: the compatible string surrounded in double quotes
+      [O] deprecated: a deprecated compatible string surrounded in
+                      double quotes
+      [O] description: description of the compatible string
+
+* [O] required: sequence of required properties:
+      [R] name: name of the property surrounded in double quotes
+      [R] description: description of the property
+      [O] reference: optional reference to a binding id
+
+* [O] optional: sequence of optional properties:
+      [R] name: name of the property surrounded in double quotes
+      [R] description: description of the property
+      [O] reference: optional reference to a binding id
+
+* [O] deprecated: sequence of deprecated properties:
+      [R] name: name of the property surrounded in double quotes
+      [R] description: description of the property
+      [O] reference: optional reference to a binding id
+
+* [R] example: sequence of examples:
+      [R] dts: DT source of example usage. The example text must use
+               literal style ("|") so that it retains indentation and
+               newlines.
+      [O] description: description of the example
+
+Note: [R] and [O] denote required and optional fields, respectively.
+
+Skeleton Binding
+----------------
+
+The skeleton.yaml binding found in the top of the DT binding tree
+is the canonical example of syntax and format to use when writing
+a DT binding document. It is maintained with the latest formatting
+conventions, making it the best starting point when writing a new DT
+binding.
-- 
2.1.4

[RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Matt Porter]

Add a skeleton DT binding document that serves as the canonical
example for implementing YAML-based DT bindings documentation.
The skeleton binding illustrates use of all fields and variations
described in the dt-binding-format.txt documentation.

Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
---
 Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
 1 file changed, 98 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/skeleton.yaml

diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
new file mode 100644
index 0000000..175965f
--- /dev/null
+++ b/Documentation/devicetree/bindings/skeleton.yaml
@@ -0,0 +1,98 @@
+%YAML 1.2
+---
+id: skel-device
+
+title: Skeleton Device
+
+maintainer:
+  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
+
+description: >
+  The Skeleton Device binding represents the SK11 device produced by
+  the Skeleton Corporation. The binding can also support compatible
+  clones made by second source vendors.
+
+compatible:
+  - name: "skel,sk11"
+  - name: "faux,fx11"
+    description: A clone of the original sk11 device
+
+required:
+  - name: "reg"
+    description: chip select address of skeleton device
+    reference: spi-slave
+  - name: "spi-max-frequency"
+    description: >
+      Maximum SPI clocking speed of skeleton device in Hz, must be
+      1000000
+    reference: spi-slave
+
+optional:
+  - name: "spi-cs-high"
+    description: >
+      Set if skeleton device configuration straps are set for chip
+      select polarity high
+    reference: spi-slave
+
+deprecated:
+  - name: "skel,deprecated1"
+    description: >
+      First of two deprecated properties.
+  - name: "skel,deprecated2"
+    description: >
+      Second of two deprecated properties.
+
+example:
+  - dts: |
+      sk11@0 {
+              compatible = "skel,sk11";
+              reg = <0>;
+              spi-max-frequency = <1000000>;
+              spi-cs-high;
+      };
+...
+
+---
+id: skel-mini
+
+title: Skeleton Mini Device
+
+maintainer:
+  - name: Rogue Developer <rogue-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
+
+description: >
+  The Skeleton Mini Device binding represents the SK47x series devices
+  produced by the Skeleton Corporation.
+
+compatible:
+  - name: "skel,sk472"
+  - name: "skel,sk473"
+  - name: "skel,sk474"
+  - deprecated: "skel,sk47x"
+
+required:
+  - name: "reg"
+    description: offset and length of the registers
+
+optional:
+  - name: "skel,sync-mode"
+    description: Enable synchronous transfer mode
+
+example:
+  - dts: |
+      sk472@beef0000 {
+              compatible = "skel,sk472";
+              reg = <0xbeef0000 0x100>;
+      };
+    description: >
+      Demonstrates an SK472 in normal mode.
+
+  - dts: |
+      sk474@dead0000 {
+              compatible = "skel,sk474";
+              reg = <0xdead0000 0x100>;
+              skel,sync-mode;
+      };
+    description: >
+      Demonstrates an SK474 in synchronous mode.
+...
-- 
2.1.4

[RFC PATCH 3/5] Documentation: dt-bindings: add YAML eeprom binding

[Matt Porter]

Convert the eeprom.txt binding to standard YAML DT
binding format.

Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
---
 Documentation/devicetree/bindings/eeprom.yaml | 44 +++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/eeprom.yaml

diff --git a/Documentation/devicetree/bindings/eeprom.yaml b/Documentation/devicetree/bindings/eeprom.yaml
new file mode 100644
index 0000000..bf1244b
--- /dev/null
+++ b/Documentation/devicetree/bindings/eeprom.yaml
@@ -0,0 +1,44 @@
+%YAML 1.2
+---
+id: eeprom-i2c
+
+title: EEPROMs (I2C)
+
+compatible:
+  - name: "<manufacturer>,24c00"
+  - name: "<manufacturer>,24c01"
+  - name: "<manufacturer>,24c02"
+  - name: "<manufacturer>,24c04"
+  - name: "<manufacturer>,24c08"
+  - name: "<manufacturer>,24c16"
+  - name: "<manufacturer>,24c32"
+  - name: "<manufacturer>,24c64"
+  - name: "<manufacturer>,24c128"
+  - name: "<manufacturer>,24c256"
+  - name: "<manufacturer>,24c512"
+  - name: "<manufacturer>,24c1024"
+  - name: "<manufacturer>,spd"
+
+required:
+  - name: "reg"
+    description: the I2c address of the EEPROM
+
+optional:
+  - name: "pagesize"
+    description: >
+      the length of the pagesize for writing. Please consult the
+      manual of your device, that value varies a lot. A wrong value
+      may result in data loss! If not specified, a safety value of
+      '1' is used which will be very slow.
+  - name: "read-only"
+    description: >
+      this parameterless property disables writes to the eeprom
+
+example:
+  - dts: |
+      eeprom@52 {
+              compatible = "atmel,24c32";
+              reg = <0x52>;
+              pagesize = <32>;
+       };
+...
-- 
2.1.4

[RFC PATCH 4/5] Documentation: dt-bindings: phy: add YAML generic PHY binding

[Matt Porter]

Convert the generic phy-bindings.txt binding to standard YAML
DT binding format.

Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
---
 .../devicetree/bindings/phy/phy-bindings.yaml      | 89 ++++++++++++++++++++++
 1 file changed, 89 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/phy/phy-bindings.yaml

diff --git a/Documentation/devicetree/bindings/phy/phy-bindings.yaml b/Documentation/devicetree/bindings/phy/phy-bindings.yaml
new file mode 100644
index 0000000..9b164d1
--- /dev/null
+++ b/Documentation/devicetree/bindings/phy/phy-bindings.yaml
@@ -0,0 +1,89 @@
+%YAML 1.2
+---
+id: phy-device
+
+title: PHY device
+
+description: >
+  This document explains only the device tree data binding.  For general
+  information about PHY subsystem refer to Documentation/phy.txt
+
+required:
+  - name: "#phy-cells"
+    description: >
+      number of cells in a PHY specifier. The meaning of all those
+      cells is defined by the binding for the phy node. The PHY
+      provider can use the values in cells to find the appropriate
+      PHY.
+
+optional:
+  - name: "phy-supply"
+    description: >
+      phandle to a regulator that provides power to the PHY. This
+      regulator will be managed during the PHY power on/off sequence.
+
+example:
+  - dts: |
+      phys: phy {
+          compatible = "xxx";
+          reg = <...>;
+          .
+          .
+          #phy-cells = <1>;
+          .
+          .
+      };
+    description: >
+      Describes an IP block (PHY provider) that implements 2 different
+      PHYs. In order to differentiate between these 2 PHYs, an
+      additional specifier should be given while trying to get a
+      reference to it.
+...
+
+---
+id: phy-user
+
+title: PHY user
+
+required:
+  - name: "phys"
+    description: >
+      the phandle for the PHY device (used by the PHY subsystem)
+  - name: "phy-names"
+    description: >
+      the names of the PHY corresponding to the PHYs present in the
+      *phys* phandle
+
+example:
+  - dts: |
+      usb1: usb_otg_ss@xxx {
+          compatible = "xxx";
+          reg = <xxx>;
+          .
+          .
+          phys = <&usb2_phy>, <&usb3_phy>;
+          phy-names = "usb2phy", "usb3phy";
+          .
+          .
+      };
+    description: >
+       Represents a controller that uses two PHYs, one for usb2 and one
+       for usb3.
+
+  - dts: |
+      usb2: usb_otg_ss@xxx {
+          compatible = "xxx";
+          reg = <xxx>;
+          .
+          .
+          phys = <&phys 1>;
+          phy-names = "usbphy";
+          .
+          .
+      };
+    description: >
+      Represents a controller that uses one of the PHYs of the PHY
+      provider device defined previously. Note that the phy handle
+      has an additional specifier "1" to differentiate between the
+      two PHYs.
+...
-- 
2.1.4

[RFC PATCH 5/5] Documentation: dt-bindings: phy: add YAML TI PHY binding

[Matt Porter]

Convert the ti-phy.txt binding to standard YAML DT binding
format. This binding references the YAML generic PHY binding.

Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
---
 Documentation/devicetree/bindings/phy/ti-phy.yaml | 166 ++++++++++++++++++++++
 1 file changed, 166 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/phy/ti-phy.yaml

diff --git a/Documentation/devicetree/bindings/phy/ti-phy.yaml b/Documentation/devicetree/bindings/phy/ti-phy.yaml
new file mode 100644
index 0000000..ec93501
--- /dev/null
+++ b/Documentation/devicetree/bindings/phy/ti-phy.yaml
@@ -0,0 +1,166 @@
+%YAML 1.2
+---
+id: omap-control-phy
+
+title: OMAP Control PHY
+
+compatible:
+  - name: "ti,control-phy-otghs"
+    description: if it has otghs_control mailbox register as on OMAP4.
+  - name: "ti,control-phy-usb2"
+    description: if it has Power down bit in control_dev_conf register
+                 e.g. USB2_PHY on OMAP5.
+  - name: "ti,control-phy-pipe3"
+    description: if it has DPLL and individual Rx & Tx power control
+                 e.g. USB3 PHY and SATA PHY on OMAP5.
+  - name: "ti,control-phy-pcie"
+    description: for pcie to support external clock for pcie and to
+                 set PCS delay value. e.g. PCIE PHY in DRA7x
+  - name: "ti,control-phy-usb2-dra7"
+    description: if it has power down register like USB2 PHY on DRA7
+                 DRA7 platform.
+  - name: "ti,control-phy-usb2-am437"
+    description: if it has power down register like USB2 PHY on AM437
+                 platform.
+required:
+  - name: "reg"
+    description: register ranges as listed in the reg-names property
+  - name: "reg-names"
+    description: >
+                 "otghs_control" for control-phy-otghs "power",
+                 "pcie_pcs" and "control_sma" for control-phy-pcie
+                 "power" for all other types
+
+example:
+  - dts: |
+      omap_control_usb: omap-control-usb@4a002300 {
+              compatible = "ti,control-phy-otghs";
+              reg = <0x4a00233c 0x4>;
+              reg-names = "otghs_control";
+      };
+...
+
+---
+id: omap-usb2-phy
+
+title: OMAP USB2 PHY
+
+compatible:
+  - name: "ti,omap-usb2"
+
+required:
+  - name: "reg"
+    description: >
+                 Address and length of the register set for the device.
+  - name: "#phy-cells"
+    description: >
+                 determine the number of cells that should be given in
+                 the phandle while referencing this phy.
+    reference: phy-device
+  - name: "clocks"
+    description: >
+                 a list of phandles and clock-specifier pairs, one for
+                 each entry in clock-names.
+    reference: clock-consumer
+  - name: "clock-names"
+    description: >
+                 should include:
+                 * "wkupclk" - wakeup clock.
+                 * "refclk" - reference clock (optional).
+    reference: clock-consumer
+
+optional:
+  - name: "ctrl-module"
+    description: >
+                 phandle of the control module used by PHY driver to
+                 power on the PHY.
+
+example:
+  - dts: |
+      usb2phy@4a0ad080 {
+                compatible = "ti,omap-usb2";
+                reg = <0x4a0ad080 0x58>;
+                ctrl-module = <&omap_control_usb>;
+                #phy-cells = <0>;
+                clocks = <&usb_phy_cm_clk32k>, <&usb_otg_ss_refclk960m>;
+                clock-names = "wkupclk", "refclk";
+      };
+    description: >
+                 This is usually a subnode of ocp2scp to which it is
+                 connected.
+...
+
+---
+id: ti-pipe3-phy
+
+title: TI Pipe3 PHY
+
+compatible:
+  - name: "ti,phy-usb3"
+  - name: "ti,phy-pipe3-sata"
+  - name: "ti,phy-pipe3-pcie"
+  - deprecated: "ti,omap-usb3"
+
+required:
+  - name: "reg"
+    description: >
+                 Address and length of the register set for the device.
+  - name: "reg-names"
+    description: >
+                 The names of the register addresses corresponding to
+                 the registers filled in "reg".
+  - name: "#phy-cells"
+    description: >
+                 determine the number of cells that should be given in
+                 the phandle while referencing this phy.
+    reference: phy-device
+  - name: "clocks"
+    description: >
+                 a list of phandles and clock-specifier pairs, one for
+                 each entry in clock-names.
+    reference: clock-consumer
+  - name: "clock-names"
+    description: >
+                 should include:
+                 * "wkupclk" - wakeup clock.
+                 * "sysclk" - system clock.
+                 * "refclk" - reference clock.
+                 * "dpll_ref" - external dpll ref clk
+                 * "dpll_ref_m2" - external dpll ref clk
+                 * "phy-div" - divider for apll
+                 * "div-clk" - apll clock
+    reference: clock-consumer
+
+optional:
+  - name: "ctrl-module"
+    description: >
+                 phandle of the control module used by PHY driver to
+                 power on the PHY.
+  - name: "id"
+    description: >
+                 If there are multiple instance of the same type, in
+                 order to differentiate between each instance "id"
+                 can be used (e.g., multi-lane PCIe PHY). If "id" is
+                 not provided, it is set to default value of '1'.
+
+example:
+  - dts: |
+      usb3phy@4a084400 {
+              compatible = "ti,phy-usb3";
+              reg = <0x4a084400 0x80>,
+                    <0x4a084800 0x64>,
+                    <0x4a084c00 0x40>;
+              reg-names = "phy_rx", "phy_tx", "pll_ctrl";
+              ctrl-module = <&omap_control_usb>;
+              #phy-cells = <0>;
+              clocks = <&usb_phy_cm_clk32k>,
+                       <&sys_clkin>,
+                       <&usb_otg_ss_refclk960m>;
+              clock-names = "wkupclk",
+                            "sysclk",
+                            "refclk";
+      };
+    description: >
+                 This is usually a subnode of ocp2scp to which it is
+                 connected.
+...
-- 
2.1.4

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Rob Herring]

On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> During the Device Tree microconference at Linux Plumbers 2015, we had
> a short discussion about how to improve DT Binding Documentation. A
> number of issues were raised (again, as these things have been
> discussed in the past) including:
>
>         * Inconsistency between binding documents due to prose text
>           format.
>         * Inability to reliably machine read bindings for mass update
>           or search.
>         * Bit rot of bindings as new conventions are agreed upon but
>           only new bindings are changed.

Thanks for pulling this together.

> Grant Likely probably summed up the issue best with "...as long as
> bindings are human readable, we'll have issues...". The context
> of that comment was, of course, regarding our current documents
> written in very inconsistent prose style. When the topic of needing
> the bindings in a rigid format was raised, there was general head
> nodding that this was needed. It was noted that this has been
> discussed many times before and nothing has been done.
>
> My proposed solution to the problem is to convert all DT bindings
> a rigid text markup format. In choosing a text markup language my
> requirements were:
>
>         1) Human readable
>         2) Well documented
>         3) Easy to translate to other data formats
>         4) Well supported by tools and libraries
>
> After looking at a number of markup options, YAML stood out as the
> one that meets all of these requirements. The YAML syntax is adopted
> in many projects specifically because of the high level of readability.
> A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
> There's a number of tools to convert between YAML and other popular
> data formats such as JSON and XML. XML was cited by Behan Webster
> during the microconference as an important data format as the type
> of developers that may produce comprehensive DTS Binding validation
> tools will want to use XML. Every major scripting language has a
> high level binding to the low level libyaml C library to facilitate
> handling of YAML data files.

Being markup language novice, this looks good to me.

> One caveat with YAML is it does not tolerate tabs. Yes, I said it.
> No tabs! This can be managed with proper editor modes and also with
> helper scripts to strip tabs to aid in people passing planned
> checkpatch.pl checks that would run YAML DT Binding specific tag
> validators for new bindings.

What do parsers do with tabs? Throw an error?

Beyond tabs, how do we check files can be parsed both generically and
for any binding specific requirements. We now need a schema for
checking the schema. We need some equivalent to compile testing.

An example such as checking that compatible strings are documented as
checkpatch.pl does would be nice. Roughly, that would be just list all
compatible values.

> The scope of the initial YAML DT Binding format was specifically
> limited to supporting *only* the content we have in bindings today.
> The idea here is to propose and agree on something that will take
> us just a few steps in the right direction. If we move *all* current
> binding content to a machine parseable format, additional features
> can be added with more automation and scripting. As it stands today,
> because of the inconsistency of the wording of the files, we can't
> add a lot of new features to the content until we convert what we
> have today into a standard format.
>
> With that said, it should be noted that some new features such as
> "type" tags to indicate cell types could be added to support
> additional DTS validation beyond what the current content supports.
> Another possibility is adding "range" type information to validate
> the legal values for a cell.
>
> This series is broken up into three major parts:
>
> 1) The documentation defining the YAML DT binding format
> 2) A skeleton device binding example illustrating use of this format
> 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
>    ti-phy.txt
>
> As a proof of concept of what can be done with a proper machine
> readable DT binding source file, there's a simple markdown document
> generator at https://github.com/konsulko/dtgendoc. Also, to see
> actual output from the generator, the generated markdown from those
> bindings is viewable at https://github.com/konsulko/dtgendoc/wiki

Nice.

> There's a lot of other possibilities for validation tools using
> only the data we have today in the bindings. In addition, Frank
> Rowand covered some DT debug techniques that would benefit from
> the binding documentation being 100% reliably searchable.
>
> I found it useful to see a side-by-side view of a converted doc
> versus the original content, so here's a screenshot of eeprom.txt
> vs. eeprom.yaml:
> https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml
>
> When we decide on a text markup format that is acceptable, then the
> next step is to convert all the bindings. That process would start
> with the complete set of generic bindings as they will be referenced
> by the actual device bindings.

You are going to do that for everyone, right? ;)

I've got some comments on the specific format as well.

Rob

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Rob Herring]

On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Add a skeleton DT binding document that serves as the canonical
> example for implementing YAML-based DT bindings documentation.
> The skeleton binding illustrates use of all fields and variations
> described in the dt-binding-format.txt documentation.
>
> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> ---
>  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
>  1 file changed, 98 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
>
> diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
> new file mode 100644
> index 0000000..175965f
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/skeleton.yaml
> @@ -0,0 +1,98 @@
> +%YAML 1.2
> +---
> +id: skel-device
> +
> +title: Skeleton Device
> +
> +maintainer:
> +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>

We'd want to tie this into get_maintainers.pl obviously.

> +
> +description: >
> +  The Skeleton Device binding represents the SK11 device produced by
> +  the Skeleton Corporation. The binding can also support compatible
> +  clones made by second source vendors.
> +
> +compatible:
> +  - name: "skel,sk11"
> +  - name: "faux,fx11"

Is this an OR or AND? We need both.

The complicated case is "one of {specific names} followed by {generic name}."

> +    description: A clone of the original sk11 device
> +
> +required:
> +  - name: "reg"

We definitely need type info from the start.

> +    description: chip select address of skeleton device
> +    reference: spi-slave

I would like to not have to list properties if the inherited binding
lists it. The problem is we need to say how many cells and the order
(not a problem here, but for mmio devices).

Perhaps we can list the reference at the top level for the node
instead of for every property.

> +  - name: "spi-max-frequency"
> +    description: >
> +      Maximum SPI clocking speed of skeleton device in Hz, must be
> +      1000000
> +    reference: spi-slave

Rather than listing the property and having constraint in description,
perhaps we could add constraints like this:

- spi-max-frequency-range: 1000000 1000000

Or groups of constraints:

- spi-max-frequency-constraints:
  range: 1000000 1000000
  some-other-constraint: <value>

> +
> +optional:
> +  - name: "spi-cs-high"
> +    description: >
> +      Set if skeleton device configuration straps are set for chip
> +      select polarity high
> +    reference: spi-slave
> +
> +deprecated:
> +  - name: "skel,deprecated1"
> +    description: >
> +      First of two deprecated properties.
> +  - name: "skel,deprecated2"
> +    description: >
> +      Second of two deprecated properties.
> +
> +example:
> +  - dts: |
> +      sk11@0 {
> +              compatible = "skel,sk11";
> +              reg = <0>;
> +              spi-max-frequency = <1000000>;
> +              spi-cs-high;
> +      };

At least in this example, we could generate it. Examples are nice, but
we have dts files full of examples already. I get a fair number of
"fix the example" patches, so maybe we should eliminate the simple
ones.

Rob

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Pantelis Antoniou]

Hi Rob,

> On Aug 28, 2015, at 17:53 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> Add a skeleton DT binding document that serves as the canonical
>> example for implementing YAML-based DT bindings documentation.
>> The skeleton binding illustrates use of all fields and variations
>> described in the dt-binding-format.txt documentation.
>> 
>> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
>> ---
>> Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
>> 1 file changed, 98 insertions(+)
>> create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
>> 
>> diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
>> new file mode 100644
>> index 0000000..175965f
>> --- /dev/null
>> +++ b/Documentation/devicetree/bindings/skeleton.yaml
>> @@ -0,0 +1,98 @@
>> +%YAML 1.2
>> +---
>> +id: skel-device
>> +
>> +title: Skeleton Device
>> +
>> +maintainer:
>> +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> 
> We'd want to tie this into get_maintainers.pl obviously.
> 
>> +
>> +description: >
>> +  The Skeleton Device binding represents the SK11 device produced by
>> +  the Skeleton Corporation. The binding can also support compatible
>> +  clones made by second source vendors.
>> +
>> +compatible:
>> +  - name: "skel,sk11"
>> +  - name: "faux,fx11"
> 
> Is this an OR or AND? We need both.
> 
> The complicated case is "one of {specific names} followed by {generic name}."
> 
>> +    description: A clone of the original sk11 device
>> +
>> +required:
>> +  - name: "reg"
> 
> We definitely need type info from the start.
> 
>> +    description: chip select address of skeleton device
>> +    reference: spi-slave
> 
> I would like to not have to list properties if the inherited binding
> lists it. The problem is we need to say how many cells and the order
> (not a problem here, but for mmio devices).
> 
> Perhaps we can list the reference at the top level for the node
> instead of for every property.
> 
>> +  - name: "spi-max-frequency"
>> +    description: >
>> +      Maximum SPI clocking speed of skeleton device in Hz, must be
>> +      1000000
>> +    reference: spi-slave
> 
> Rather than listing the property and having constraint in description,
> perhaps we could add constraints like this:
> 
> - spi-max-frequency-range: 1000000 1000000
> 
> Or groups of constraints:
> 
> - spi-max-frequency-constraints:
>  range: 1000000 1000000
>  some-other-constraint: <value>
> 

I would like something that looks like a C expression.

For instance
- constraint: spi-max-frequency >= 1000000 && spi-max-frequency < 10000000

>> +
>> +optional:
>> +  - name: "spi-cs-high"
>> +    description: >
>> +      Set if skeleton device configuration straps are set for chip
>> +      select polarity high
>> +    reference: spi-slave
>> +
>> +deprecated:
>> +  - name: "skel,deprecated1"
>> +    description: >
>> +      First of two deprecated properties.
>> +  - name: "skel,deprecated2"
>> +    description: >
>> +      Second of two deprecated properties.
>> +
>> +example:
>> +  - dts: |
>> +      sk11@0 {
>> +              compatible = "skel,sk11";
>> +              reg = <0>;
>> +              spi-max-frequency = <1000000>;
>> +              spi-cs-high;
>> +      };
> 
> At least in this example, we could generate it. Examples are nice, but
> we have dts files full of examples already. I get a fair number of
> "fix the example" patches, so maybe we should eliminate the simple
> ones.
> 

How about we make sure we parse the example in the binding always, so
that we verify that it does much what the binding states?

> Rob

Regards

— Pantelis

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Pantelis Antoniou]

Hi Rob,

> On Aug 28, 2015, at 17:53 , Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> 
> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> Add a skeleton DT binding document that serves as the canonical
>> example for implementing YAML-based DT bindings documentation.
>> The skeleton binding illustrates use of all fields and variations
>> described in the dt-binding-format.txt documentation.
>> 
>> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
>> ---
>> Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
>> 1 file changed, 98 insertions(+)
>> create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
>> 
>> diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
>> new file mode 100644
>> index 0000000..175965f
>> --- /dev/null
>> +++ b/Documentation/devicetree/bindings/skeleton.yaml
>> @@ -0,0 +1,98 @@
>> +%YAML 1.2
>> +---
>> +id: skel-device
>> +
>> +title: Skeleton Device
>> +
>> +maintainer:
>> +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> 
> We'd want to tie this into get_maintainers.pl obviously.
> 
>> +
>> +description: >
>> +  The Skeleton Device binding represents the SK11 device produced by
>> +  the Skeleton Corporation. The binding can also support compatible
>> +  clones made by second source vendors.
>> +
>> +compatible:
>> +  - name: "skel,sk11"
>> +  - name: "faux,fx11"
> 
> Is this an OR or AND? We need both.
> 
> The complicated case is "one of {specific names} followed by {generic name}."
> 
>> +    description: A clone of the original sk11 device
>> +
>> +required:
>> +  - name: "reg"
> 
> We definitely need type info from the start.
> 
>> +    description: chip select address of skeleton device
>> +    reference: spi-slave
> 
> I would like to not have to list properties if the inherited binding
> lists it. The problem is we need to say how many cells and the order
> (not a problem here, but for mmio devices).
> 
> Perhaps we can list the reference at the top level for the node
> instead of for every property.
> 
>> +  - name: "spi-max-frequency"
>> +    description: >
>> +      Maximum SPI clocking speed of skeleton device in Hz, must be
>> +      1000000
>> +    reference: spi-slave
> 
> Rather than listing the property and having constraint in description,
> perhaps we could add constraints like this:
> 
> - spi-max-frequency-range: 1000000 1000000
> 
> Or groups of constraints:
> 
> - spi-max-frequency-constraints:
>  range: 1000000 1000000
>  some-other-constraint: <value>
> 

I would like something that looks like a C expression.

For instance
- constraint: spi-max-frequency >= 1000000 && spi-max-frequency < 10000000

>> +
>> +optional:
>> +  - name: "spi-cs-high"
>> +    description: >
>> +      Set if skeleton device configuration straps are set for chip
>> +      select polarity high
>> +    reference: spi-slave
>> +
>> +deprecated:
>> +  - name: "skel,deprecated1"
>> +    description: >
>> +      First of two deprecated properties.
>> +  - name: "skel,deprecated2"
>> +    description: >
>> +      Second of two deprecated properties.
>> +
>> +example:
>> +  - dts: |
>> +      sk11@0 {
>> +              compatible = "skel,sk11";
>> +              reg = <0>;
>> +              spi-max-frequency = <1000000>;
>> +              spi-cs-high;
>> +      };
> 
> At least in this example, we could generate it. Examples are nice, but
> we have dts files full of examples already. I get a fair number of
> "fix the example" patches, so maybe we should eliminate the simple
> ones.
> 

How about we make sure we parse the example in the binding always, so
that we verify that it does much what the binding states?

> Rob

Regards

— Pantelis

--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Fri, Aug 28, 2015 at 09:26:17AM -0500, Rob Herring wrote:
> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > During the Device Tree microconference at Linux Plumbers 2015, we had
> > a short discussion about how to improve DT Binding Documentation. A
> > number of issues were raised (again, as these things have been
> > discussed in the past) including:
> >
> >         * Inconsistency between binding documents due to prose text
> >           format.
> >         * Inability to reliably machine read bindings for mass update
> >           or search.
> >         * Bit rot of bindings as new conventions are agreed upon but
> >           only new bindings are changed.
> 
> Thanks for pulling this together.
> 
> > Grant Likely probably summed up the issue best with "...as long as
> > bindings are human readable, we'll have issues...". The context
> > of that comment was, of course, regarding our current documents
> > written in very inconsistent prose style. When the topic of needing
> > the bindings in a rigid format was raised, there was general head
> > nodding that this was needed. It was noted that this has been
> > discussed many times before and nothing has been done.
> >
> > My proposed solution to the problem is to convert all DT bindings
> > a rigid text markup format. In choosing a text markup language my
> > requirements were:
> >
> >         1) Human readable
> >         2) Well documented
> >         3) Easy to translate to other data formats
> >         4) Well supported by tools and libraries
> >
> > After looking at a number of markup options, YAML stood out as the
> > one that meets all of these requirements. The YAML syntax is adopted
> > in many projects specifically because of the high level of readability.
> > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
> > There's a number of tools to convert between YAML and other popular
> > data formats such as JSON and XML. XML was cited by Behan Webster
> > during the microconference as an important data format as the type
> > of developers that may produce comprehensive DTS Binding validation
> > tools will want to use XML. Every major scripting language has a
> > high level binding to the low level libyaml C library to facilitate
> > handling of YAML data files.
> 
> Being markup language novice, this looks good to me.
> 
> > One caveat with YAML is it does not tolerate tabs. Yes, I said it.
> > No tabs! This can be managed with proper editor modes and also with
> > helper scripts to strip tabs to aid in people passing planned
> > checkpatch.pl checks that would run YAML DT Binding specific tag
> > validators for new bindings.
> 
> What do parsers do with tabs? Throw an error?

Yes, they throw an error. Keep in mind that most of what I used to start
are general purpose conversion tools on top of a particular scripting
languages's high level binding to libyaml. The error output leaves a
bit to be desired for our use case. In any case, when I was developing
the skeleton.yaml I used the yaml script from
https://github.com/ryo/yamltools to catch all these syntax errors I
was inserting..like tabs. the PyYaml binding being used in my PoC
dtgendoc does the same thing but I don't gracefully handle those
errors like we could.

> Beyond tabs, how do we check files can be parsed both generically and
> for any binding specific requirements. We now need a schema for
> checking the schema. We need some equivalent to compile testing.

Right. So, I think what you are touching on is something I should
have expanded on in the TODO list. Basically, we need a scripted
tool that we run from checkpatch.pl that 1) reads the .yaml and
validates the YAML itself (that comes for free in the high level
parsers) reporting errors in a sensical manner 2) validates our
DT binding specific tags

Now, I would caution about trying to do too much on Day 1 or we
could end up back at the "never doing anything" stage. It would
be an improvement to simply check that the basic tags exist as
shown in the [R] or [O] fields in the documentation. One thing
I should point out is that I carefully avoided marking some tags
as [R] where existing bindings don't have them...even if logically,
a description should be required on every binding. The idea here
is to avoid updating content at the same time that we are updating
the format. Rather, I think it would be better to get the base
format updated, then come back with a janitorial team and add
descriptions (since now we can generate a worklist of those
bindings missing a top-level description) and systematically
fix those and review with the appropriate maintainers.

> An example such as checking that compatible strings are documented as
> checkpatch.pl does would be nice. Roughly, that would be just list all
> compatible values.

Ok, so my comments above were strictly about a validator for the
binding doc submission itself. I can add an example based on your
checkpatch.pl to adapt it to the .yaml compatible tags.

> > The scope of the initial YAML DT Binding format was specifically
> > limited to supporting *only* the content we have in bindings today.
> > The idea here is to propose and agree on something that will take
> > us just a few steps in the right direction. If we move *all* current
> > binding content to a machine parseable format, additional features
> > can be added with more automation and scripting. As it stands today,
> > because of the inconsistency of the wording of the files, we can't
> > add a lot of new features to the content until we convert what we
> > have today into a standard format.
> >
> > With that said, it should be noted that some new features such as
> > "type" tags to indicate cell types could be added to support
> > additional DTS validation beyond what the current content supports.
> > Another possibility is adding "range" type information to validate
> > the legal values for a cell.
> >
> > This series is broken up into three major parts:
> >
> > 1) The documentation defining the YAML DT binding format
> > 2) A skeleton device binding example illustrating use of this format
> > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
> >    ti-phy.txt
> >
> > As a proof of concept of what can be done with a proper machine
> > readable DT binding source file, there's a simple markdown document
> > generator at https://github.com/konsulko/dtgendoc. Also, to see
> > actual output from the generator, the generated markdown from those
> > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki
> 
> Nice.
> 
> > There's a lot of other possibilities for validation tools using
> > only the data we have today in the bindings. In addition, Frank
> > Rowand covered some DT debug techniques that would benefit from
> > the binding documentation being 100% reliably searchable.
> >
> > I found it useful to see a side-by-side view of a converted doc
> > versus the original content, so here's a screenshot of eeprom.txt
> > vs. eeprom.yaml:
> > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml
> >
> > When we decide on a text markup format that is acceptable, then the
> > next step is to convert all the bindings. That process would start
> > with the complete set of generic bindings as they will be referenced
> > by the actual device bindings.
> 
> You are going to do that for everyone, right? ;)

Let's just say that I'm banking on others helping here once we have
a format agreed upon. If we can hold the binding doc schema definition
initially to just define tags for content that already exists in our
textual binding docs, the effort for conversion is tolerable. To give
an example, that phy-bindings.txt, it took 15 minutes to convert and
and pass through the yaml parser and dtgendoc. The reason is that it's
pure reformatting work. It doesn't take any special knowledge of the
hardware and it doesn't involve reviewing dts files to extra
additional information. Some of the annoyances can be streamlined
like tab stripping and handling the two space indentation to make
this process faster. One of my next things is to get a simple tool
going that reports problems with conversions, essentially what I
said was needed to integrate with checkpatch, so this process of
conversion is even faster. Trivial peripheral bindings like eeprom.txt
can be done in 5 minutes or so right now.

If we decide we must have tags like "type:" in the initial binding
doc schema definition *and* we must add that content in each
conversion, then this becomes more time consuming to validate that
information against working dts files. IMHO, we'd be better off
to get the base format straight, addressing missing pieces like
all the compatible permutations, and convert them all with
just that content. After that, we come back and add new content
features like type: tagging. I'm trying to find a reasonable
place to do this incrementally since the volume of bindings to
convert is enormous.

But to answer your question, if we get a format I'll do
conversions and hope I'm not alone.

> I've got some comments on the specific format as well.

Great, thanks.

-Matt

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Matt Porter]

On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Add a skeleton DT binding document that serves as the canonical
> > example for implementing YAML-based DT bindings documentation.
> > The skeleton binding illustrates use of all fields and variations
> > described in the dt-binding-format.txt documentation.
> >
> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> > ---
> >  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
> >  1 file changed, 98 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
> > new file mode 100644
> > index 0000000..175965f
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/skeleton.yaml
> > @@ -0,0 +1,98 @@
> > +%YAML 1.2
> > +---
> > +id: skel-device
> > +
> > +title: Skeleton Device
> > +
> > +maintainer:
> > +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
>
> We'd want to tie this into get_maintainers.pl obviously.

Right, I broke my rule of "no new tags for content we don't already
have" by adding this. It stems out of the discussion at LPC where
Mark suggested we could avoid the core bindings being moved by adding
maintainer info into the binding itself.

This is an area where more docs are needed. Ideally, on the first pass
conversion we would not attempt to populate these. I had considered
that we *could* only add maintainer: tags on core bindings to and
adjust get_maintainers.pl to use that info to override the standard
directory-based info if it exists. I don't think it's necessary to add
a specific maintainer for all of the peripheral bindings if the default
is the subsystem maintainer and the dt list. We could at least start
directing core binding discussion to the -spec list which is where I
think you'd like it.

> > +
> > +description: >
> > +  The Skeleton Device binding represents the SK11 device produced by
> > +  the Skeleton Corporation. The binding can also support compatible
> > +  clones made by second source vendors.
> > +
> > +compatible:
> > +  - name: "skel,sk11"
> > +  - name: "faux,fx11"
> 
> Is this an OR or AND? We need both.

True, this only covers the OR case atm.

> The complicated case is "one of {specific names} followed by {generic name}."

I need to rethink these. I do have deprecated: tag for that case and
possibly "name:" gets split to "generic:" and "specific:" and we can
then do the right thing.

For the above I would have:

compatible:
  - specific: "skel,sk11"
  - specific: "faux,fx11"

and something like the Allwinner simple framebuffer would be:

compatible:
  - generic: "simple-framebuffer"
  - specific: "allwinner,simple-framebuffer"

where our validator would insist on seeing one of the specific: tags
along with the generic: tag. A lot of bindings, given current doc
patterns would just have two tags like above.

> > +    description: A clone of the original sk11 device
> > +
> > +required:
> > +  - name: "reg"
> 
> We definitely need type info from the start.

Are you sure? Ugh.

My big contention here is that we don't carry that content in the
docs today so we shouldn't try to add it in the initial conversion.

I think you are right in that we should have it documented in the
schema but I'm concerned that we make the starting conversion
effort too large by adding type info to all converted docs.

I think we gain a lot even without at the start, but I understand
why we need it and how it will help reduce the review firehose
with the associated validation tools.

> 
> > +    description: chip select address of skeleton device
> > +    reference: spi-slave
> 
> I would like to not have to list properties if the inherited binding
> lists it. The problem is we need to say how many cells and the order
> (not a problem here, but for mmio devices).

Yeah, make sense.

> Perhaps we can list the reference at the top level for the node
> instead of for every property.

That's a good point. I was wondering if per prop references would
get unwieldy once we actually add them into all the converted docs.
There's a huge number of existing docs without proper references.

Ok, I'll take a look at collecting references per group of properties.

> 
> > +  - name: "spi-max-frequency"
> > +    description: >
> > +      Maximum SPI clocking speed of skeleton device in Hz, must be
> > +      1000000
> > +    reference: spi-slave
> 
> Rather than listing the property and having constraint in description,
> perhaps we could add constraints like this:
> 
> - spi-max-frequency-range: 1000000 1000000
> 
> Or groups of constraints:
> 
> - spi-max-frequency-constraints:
>   range: 1000000 1000000
>   some-other-constraint: <value>

I was hoping to avoid this to start due to my argument for keeping it
simple for the first pass at conversion. However, the latter looks
flexible enough. We have cases with enumerated values as well to handle
the require some thought.

> > +
> > +optional:
> > +  - name: "spi-cs-high"
> > +    description: >
> > +      Set if skeleton device configuration straps are set for chip
> > +      select polarity high
> > +    reference: spi-slave
> > +
> > +deprecated:
> > +  - name: "skel,deprecated1"
> > +    description: >
> > +      First of two deprecated properties.
> > +  - name: "skel,deprecated2"
> > +    description: >
> > +      Second of two deprecated properties.
> > +
> > +example:
> > +  - dts: |
> > +      sk11@0 {
> > +              compatible = "skel,sk11";
> > +              reg = <0>;
> > +              spi-max-frequency = <1000000>;
> > +              spi-cs-high;
> > +      };
> 
> At least in this example, we could generate it. Examples are nice, but
> we have dts files full of examples already. I get a fair number of
> "fix the example" patches, so maybe we should eliminate the simple
> ones.

Yeah, I would say that a huge majority of the examples we have now
become useless with this format. I like that idea of just generating
it for those users trying to read and understand what it looks like in
dts form. Less room for error. We can retain the examples field for
those complex examples that need human intervention to craft and
describe them properly.

-Matt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Rob Herring]

On Fri, Aug 28, 2015 at 12:49 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
>> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> > Add a skeleton DT binding document that serves as the canonical
>> > example for implementing YAML-based DT bindings documentation.
>> > The skeleton binding illustrates use of all fields and variations
>> > described in the dt-binding-format.txt documentation.
>> >
>> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
>> > ---
>> >  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
>> >  1 file changed, 98 insertions(+)
>> >  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
>> >
>> > diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
>> > new file mode 100644
>> > index 0000000..175965f
>> > --- /dev/null
>> > +++ b/Documentation/devicetree/bindings/skeleton.yaml
>> > @@ -0,0 +1,98 @@
>> > +%YAML 1.2
>> > +---
>> > +id: skel-device
>> > +
>> > +title: Skeleton Device
>> > +
>> > +maintainer:
>> > +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
>>
>> We'd want to tie this into get_maintainers.pl obviously.
>
> Right, I broke my rule of "no new tags for content we don't already
> have" by adding this. It stems out of the discussion at LPC where
> Mark suggested we could avoid the core bindings being moved by adding
> maintainer info into the binding itself.

If you stated that rule, I missed it...

> This is an area where more docs are needed. Ideally, on the first pass
> conversion we would not attempt to populate these. I had considered
> that we *could* only add maintainer: tags on core bindings to and
> adjust get_maintainers.pl to use that info to override the standard
> directory-based info if it exists. I don't think it's necessary to add
> a specific maintainer for all of the peripheral bindings if the default
> is the subsystem maintainer and the dt list. We could at least start
> directing core binding discussion to the -spec list which is where I
> think you'd like it.

Yes, certainly this can solve my problem around handling core bindings.

Some subsystem maintainers require a driver have a maintainer. We
could certainly enforce that too. We could probably automatically
populate this with the original author as a starting point. OTOH,
bindings should not change frequently.

>
>> > +
>> > +description: >
>> > +  The Skeleton Device binding represents the SK11 device produced by
>> > +  the Skeleton Corporation. The binding can also support compatible
>> > +  clones made by second source vendors.
>> > +
>> > +compatible:
>> > +  - name: "skel,sk11"
>> > +  - name: "faux,fx11"
>>
>> Is this an OR or AND? We need both.
>
> True, this only covers the OR case atm.
>
>> The complicated case is "one of {specific names} followed by {generic name}."
>
> I need to rethink these. I do have deprecated: tag for that case and
> possibly "name:" gets split to "generic:" and "specific:" and we can
> then do the right thing.
>
> For the above I would have:
>
> compatible:
>   - specific: "skel,sk11"
>   - specific: "faux,fx11"

This can still be an AND or OR relationship.

>
> and something like the Allwinner simple framebuffer would be:
>
> compatible:
>   - generic: "simple-framebuffer"
>   - specific: "allwinner,simple-framebuffer"
>
> where our validator would insist on seeing one of the specific: tags
> along with the generic: tag. A lot of bindings, given current doc
> patterns would just have two tags like above.

We could follow Pantelis' suggestion and do C style logic expressions:

name: ("chip1-ipblock" || "chip2-ipblock") && "generic-ipblock"


>
>> > +    description: A clone of the original sk11 device
>> > +
>> > +required:
>> > +  - name: "reg"
>>
>> We definitely need type info from the start.
>
> Are you sure? Ugh.
>
> My big contention here is that we don't carry that content in the
> docs today so we shouldn't try to add it in the initial conversion.

What? We have that all over the place. Boolean properties, phandles, etc.

> I think you are right in that we should have it documented in the
> schema but I'm concerned that we make the starting conversion
> effort too large by adding type info to all converted docs.

How we do the conversion certainly needs to be worked out. However,
first we need some sort of agreement that yes this looks promising
before we spend too much time on it. We need to wait for all the YAML
haters to come out of hiding. :) Once the direction is settled, then
we can iron out the details of the structure. Finally, then we can
determine how to do the conversion. Just because we define how type
information looks doesn't mean it has to be there day 1.

> I think we gain a lot even without at the start, but I understand
> why we need it and how it will help reduce the review firehose
> with the associated validation tools.
>
>>
>> > +    description: chip select address of skeleton device
>> > +    reference: spi-slave
>>
>> I would like to not have to list properties if the inherited binding
>> lists it. The problem is we need to say how many cells and the order
>> (not a problem here, but for mmio devices).
>
> Yeah, make sense.
>
>> Perhaps we can list the reference at the top level for the node
>> instead of for every property.
>
> That's a good point. I was wondering if per prop references would
> get unwieldy once we actually add them into all the converted docs.
> There's a huge number of existing docs without proper references.

Yes, in many cases they are implied. It should be simple enough to
generate the list though whether they point to the doc file or just
list the properties.

>
> Ok, I'll take a look at collecting references per group of properties.
>
>>
>> > +  - name: "spi-max-frequency"
>> > +    description: >
>> > +      Maximum SPI clocking speed of skeleton device in Hz, must be
>> > +      1000000
>> > +    reference: spi-slave
>>
>> Rather than listing the property and having constraint in description,
>> perhaps we could add constraints like this:
>>
>> - spi-max-frequency-range: 1000000 1000000
>>
>> Or groups of constraints:
>>
>> - spi-max-frequency-constraints:
>>   range: 1000000 1000000
>>   some-other-constraint: <value>
>
> I was hoping to avoid this to start due to my argument for keeping it
> simple for the first pass at conversion. However, the latter looks
> flexible enough. We have cases with enumerated values as well to handle
> the require some thought.

We can always add constraints as a second step, but I'd like to define
the structure at least.

Rob

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Frank Rowand]

On 8/27/2015 10:23 PM, Matt Porter wrote:
> During the Device Tree microconference at Linux Plumbers 2015, we had
> a short discussion about how to improve DT Binding Documentation. A
> number of issues were raised (again, as these things have been
> discussed in the past) including:
> 
> 	* Inconsistency between binding documents due to prose text
> 	  format.
> 	* Inability to reliably machine read bindings for mass update
>           or search.
>         * Bit rot of bindings as new conventions are agreed upon but
> 	  only new bindings are changed.
> 
> Grant Likely probably summed up the issue best with "...as long as
> bindings are human readable, we'll have issues...". The context
> of that comment was, of course, regarding our current documents
> written in very inconsistent prose style. When the topic of needing
> the bindings in a rigid format was raised, there was general head
> nodding that this was needed. It was noted that this has been
> discussed many times before and nothing has been done.
> 
> My proposed solution to the problem is to convert all DT bindings
> a rigid text markup format. In choosing a text markup language my
> requirements were:
> 
> 	1) Human readable
> 	2) Well documented
> 	3) Easy to translate to other data formats
> 	4) Well supported by tools and libraries

Thanks for taking this on!  The proposal looks like a great start.
The requirements look good.


> After looking at a number of markup options, YAML stood out as the
> one that meets all of these requirements. The YAML syntax is adopted
> in many projects specifically because of the high level of readability.
> A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
> There's a number of tools to convert between YAML and other popular
> data formats such as JSON and XML. XML was cited by Behan Webster
> during the microconference as an important data format as the type
> of developers that may produce comprehensive DTS Binding validation
> tools will want to use XML. Every major scripting language has a
> high level binding to the low level libyaml C library to facilitate
> handling of YAML data files.
> 
> One caveat with YAML is it does not tolerate tabs. Yes, I said it.
> No tabs! This can be managed with proper editor modes and also with
> helper scripts to strip tabs to aid in people passing planned
> checkpatch.pl checks that would run YAML DT Binding specific tag
> validators for new bindings.

I'm just a casual user of some markup languages, nowhere near knowledgeable
or an expert, so will be listening to reasoned opinions on the choice of
YAML.  But a quick look at the spec you referenced and the examples you
provided gives me some confidence that YAML will work.


> The scope of the initial YAML DT Binding format was specifically
> limited to supporting *only* the content we have in bindings today.
> The idea here is to propose and agree on something that will take
> us just a few steps in the right direction. If we move *all* current

Yes, this is a great approach so we can start making reasonable progress.
(Caveat below.)

> binding content to a machine parseable format, additional features
> can be added with more automation and scripting. As it stands today,
> because of the inconsistency of the wording of the files, we can't
> add a lot of new features to the content until we convert what we
> have today into a standard format.
> 
> With that said, it should be noted that some new features such as
> "type" tags to indicate cell types could be added to support
> additional DTS validation beyond what the current content supports.
> Another possibility is adding "range" type information to validate
> the legal values for a cell.

The caveat: I think we should look at some obvious features (such as
"type" that you mention) to make sure that YAML and the chosen
structuring will easily and effectively support the features.

Thinking about how some features might be implemented led to comments
that I will make in replies to the other patches.

I agree that we should not attempt to add the additional information
enabled by the new features if the information is not in the present
bindings.  Adding the new information in a subsequent pass seems to be
a good strategy.

> 
> This series is broken up into three major parts:
> 
> 1) The documentation defining the YAML DT binding format
> 2) A skeleton device binding example illustrating use of this format
> 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
>    ti-phy.txt
> 
> As a proof of concept of what can be done with a proper machine
> readable DT binding source file, there's a simple markdown document
> generator at https://github.com/konsulko/dtgendoc. Also, to see
> actual output from the generator, the generated markdown from those
> bindings is viewable at https://github.com/konsulko/dtgendoc/wiki
> 
> There's a lot of other possibilities for validation tools using
> only the data we have today in the bindings. In addition, Frank
> Rowand covered some DT debug techniques that would benefit from
> the binding documentation being 100% reliably searchable.
> 
> I found it useful to see a side-by-side view of a converted doc
> versus the original content, so here's a screenshot of eeprom.txt
> vs. eeprom.yaml:
> https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml

This binding is your patch 3, but the side by side example you provide
makes it much easier to follow along, so I'll comment here.

Part of the information about compatible got lost in the conversion.
The part that is lost is:

   If there is no specific driver for <manufacturer>, a generic
   driver based on <type> is selected.

I will discuss this a little more in response to patch 1, where I will
ask whether compatible should be treated differently than other
properties.  So maybe any further discussion of this should be in
reply to that email.


> When we decide on a text markup format that is acceptable, then the
> next step is to convert all the bindings. That process would start
> with the complete set of generic bindings as they will be referenced
> by the actual device bindings.
> 
> If the RFC wasn't explicit enough...comments welcome.
> 
> -Matt
> 
> Matt Porter (5):
>   Documentation: dt-bindings: add documentation on new DT binding format
>   Documentation: dt-bindings: add example DT binding document
>   Documentation: dt-bindings: add YAML eeprom binding
>   Documentation: dt-bindings: phy: add YAML generic PHY binding
>   Documentation: dt-bindings: phy: add YAML TI PHY binding
> 
>  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++
>  Documentation/devicetree/bindings/eeprom.yaml      |  44 ++++++
>  .../devicetree/bindings/phy/phy-bindings.yaml      |  89 +++++++++++
>  Documentation/devicetree/bindings/phy/ti-phy.yaml  | 166 +++++++++++++++++++++
>  Documentation/devicetree/bindings/skeleton.yaml    |  98 ++++++++++++
>  5 files changed, 503 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
>  create mode 100644 Documentation/devicetree/bindings/eeprom.yaml
>  create mode 100644 Documentation/devicetree/bindings/phy/phy-bindings.yaml
>  create mode 100644 Documentation/devicetree/bindings/phy/ti-phy.yaml
>  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> 

Re: [RFC PATCH 1/5] Documentation: dt-bindings: add documentation on new DT binding format

[Frank Rowand]

On 8/27/2015 10:23 PM, Matt Porter wrote:
> Documentation explaining the syntax and format of the YAML-based DT binding
> documentation.
> 
> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> ---
>  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
>  1 file changed, 106 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> 
> diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> new file mode 100644
> index 0000000..f9acc22
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> @@ -0,0 +1,106 @@
> +--------------------------
> +Device Tree Binding Format
> +--------------------------
> +
> +Background
> +----------
> +
> +DT bindings historically were written as text in prose format which
> +led to issues in usability of that source documentation. Some of
> +these issues include the need to programmatically process binding
> +source documentation to do DTS validation, perform mass updates to
> +format/style, and to generate publishable documentation in HTML or
> +PDF form.
> +
> +Overview
> +--------
> +
> +The DT binding format is based on the YAML text markup language.
> +Although there are many text markup options available, YAML
> +fulfills all requirements considered for a DT binding source format
> +which include:
> +
> +1) Must be human readable
> +2) Must be easily translated to other data formats (XML, JSON, etc).
> +3) Must have sufficient tools and libraries to enable developers to
> +   build new tools for DT binding processing
> +4) Must have a complete spec to refer to syntax
> +
> +YAML is documentated in the specification found at
> +http://www.yaml.org/spec/1.2/spec.html
> +
> +The required YAML DT binding tag format and syntax are defined in
> +the following sections.
> +
> +YAML DT Binding Syntax
> +----------------------
> +
> +* Lines starting with "#" are comments and not part of the binding itself
> +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> +* "---" starts a binding document
> +* "..." ends a binding document
> +* Multiple binding documents may exist in a single file
> +* Tabs are not permitted
> +* Scope is denoted by indentation of two spaces
> +* Key value pairs are denoted by "key: value"
> +* Sequences are denoted by "- "
> +* Scalar values may convert newlines to spaces and preserve blank
> +  lines for long description formatting using ">"
> +* Scalar values may escape all reserved characters and preserve
> +  newlines by using "|" to denote literal style
> +
> +For additional information on YAML syntax, refer to the specification
> +at http://www.yaml.org/spec/1.2/spec.html
> +
> +YAML DT Binding Format
> +----------------------
> +
> +The following YAML types are supported in the DT binding format:
> +
> +* [R] id: unique identifier in property form (e.g. skel-device)
> +
> +* [R] title: title of the binding
> +
> +* [O] maintainer: sequence of maintainers
> +      [R] name: name and email of maintainer or mailing list in RFC822
> +                form.
> +
> +* [O] description: full description of the binding
> +
> +* [O] compatible: sequence of valid compatible descriptors
> +      [R] name: the compatible string surrounded in double quotes
> +      [O] deprecated: a deprecated compatible string surrounded in
> +                      double quotes
> +      [O] description: description of the compatible string
> +
> +* [O] required: sequence of required properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] optional: sequence of optional properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] deprecated: sequence of deprecated properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id

I commented in reply to patch 0 that we should think through how
this structure will support adding new features as the next step
after converting existing bindings.

The specific case I started thinking about was the distinction between
required, optional, required-if, and optional-if.  A property might
be required in all cases, optional in all cases, required in some
specified cases, and/or optional in some specified cases.  So a
property could be both "required-if" and "optional-if".  Or it
might just be one of the two.

Here is where my YAML ignorance jumps out.  I really do not know the
best way to approach it.  And I do not know if the syntax I am using
is even remotely correct - I am trying to extrapolate from the examples
in the spec, without having read past the examples yet.  My naive knee
jerk alternatives where (by example, because I can explain it more
concisely that way instead of the way it should be described in this file):

---- (1)

required:
  - &P1
    name: "prop1"
    required-if: < some condition >

optional:
  - *P1
    optional-if: < some other condition>
  - name: "prop2"
    required: true
  - name: "prop3"
    optional: true

---- (2)

property:
  - name: "prop1"
    required-if: < some condition >
    optional-if: < some other condition >
  - name: "prop2"
    required: true
  - name: "prop3"
    optional: true


In a related question, I commented in my reply to patch 0 that some of
the information about the compatible got lost in the conversion from
the existing binding to the YAML format.  That led me to wonder why
the compatible property should be at the YAML top level (same as
"optional" and "required").  My (again naive, YAML amateur) thought
is that the form should be something like:

[O] property:
    [R] optional: | required: | required-if: | optional-if: | required-if: optional-if:
    [O] deprecated: a deprecated compatible string surrounded in
                    double quotes | ???
    [R] name: name of the property surrounded in double quotes
    [R] description: description of the property
    [O] reference: optional reference to a binding id

Apologies for bizarre format for "optional:", but for the purposes of this
comment, hopefully my intent will be clear.

Then "compatible" becomes just another "property:", just as it is in the
.dts, but any processing or analysis tool knows that the name "compatible"
is used in a very specific way and has magical meaning for .dts.

Then if compatible is just another "property:" then the information that
lost was in the conversion process would not be lost, it would appear
in "description:".


> +
> +* [R] example: sequence of examples:
> +      [R] dts: DT source of example usage. The example text must use
> +               literal style ("|") so that it retains indentation and
> +               newlines.
> +      [O] description: description of the example
> +
> +Note: [R] and [O] denote required and optional fields, respectively.
> +
> +Skeleton Binding
> +----------------
> +
> +The skeleton.yaml binding found in the top of the DT binding tree
> +is the canonical example of syntax and format to use when writing
> +a DT binding document. It is maintained with the latest formatting
> +conventions, making it the best starting point when writing a new DT
> +binding.

It would be good to have a comment of YAML style.  Looking at the spec,
there appear to be multiple ways to format content that is functionally
equivalent.  For example:

   # a flow sequence
   - [name        , hr, avg  ]
   - [Mark McGwire, 65, 0.278]
   - [Sammy Sosa  , 63, 0.288]

vs.

   # a flow mapping
   Mark McGwire: {hr: 65, avg: 0.278}
   Sammy Sosa: {
       hr: 63,
       avg: 0.288
     }

This is my first quick reading of the YAML spec, so maybe a flow sequence and
a flow mapping are not equivalent, but that was my naive impression.

I think that in the interest of human readability (requirement 1), that a
specific formatting style should be specified.  That will make life easier
for people who do not know YAML, but are (1) merely copying from existing binding
docs to create new ones, or (2) trying to understand how different bindings
differ.

-Frank

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Frank Rowand]

On 8/28/2015 11:55 AM, Rob Herring wrote:
> On Fri, Aug 28, 2015 at 12:49 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
>>> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>>> Add a skeleton DT binding document that serves as the canonical
>>>> example for implementing YAML-based DT bindings documentation.
>>>> The skeleton binding illustrates use of all fields and variations
>>>> described in the dt-binding-format.txt documentation.
>>>>
>>>> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
>>>> ---
>>>>  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
>>>>  1 file changed, 98 insertions(+)
>>>>  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
>>>>
>>>> diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
>>>> new file mode 100644
>>>> index 0000000..175965f
>>>> --- /dev/null
>>>> +++ b/Documentation/devicetree/bindings/skeleton.yaml
>>>> @@ -0,0 +1,98 @@
>>>> +%YAML 1.2
>>>> +---
>>>> +id: skel-device
>>>> +
>>>> +title: Skeleton Device
>>>> +
>>>> +maintainer:
>>>> +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
>>>
>>> We'd want to tie this into get_maintainers.pl obviously.
>>
>> Right, I broke my rule of "no new tags for content we don't already
>> have" by adding this. It stems out of the discussion at LPC where
>> Mark suggested we could avoid the core bindings being moved by adding
>> maintainer info into the binding itself.
> 
> If you stated that rule, I missed it...
> 
>> This is an area where more docs are needed. Ideally, on the first pass
>> conversion we would not attempt to populate these. I had considered
>> that we *could* only add maintainer: tags on core bindings to and
>> adjust get_maintainers.pl to use that info to override the standard
>> directory-based info if it exists. I don't think it's necessary to add
>> a specific maintainer for all of the peripheral bindings if the default
>> is the subsystem maintainer and the dt list. We could at least start
>> directing core binding discussion to the -spec list which is where I
>> think you'd like it.

I think that adding the maintainer: tags for core bindings in the conversion
process (and optionally add the tag for other bindings in the person
converting has the driver and subsystem knowledge to intelligently
do so -- not just a blind list from get_maintainer) would be a valid
exception to your desire to not add new information on the conversion
step.  As much as other uses of information from bindings is near and
dear to me, I think that throttling the binding review fire hose is
the highest priority use of additional information.


> Yes, certainly this can solve my problem around handling core bindings.
> 
> Some subsystem maintainers require a driver have a maintainer. We
> could certainly enforce that too. We could probably automatically
> populate this with the original author as a starting point. OTOH,
> bindings should not change frequently.
> 
>>
>>>> +
>>>> +description: >
>>>> +  The Skeleton Device binding represents the SK11 device produced by
>>>> +  the Skeleton Corporation. The binding can also support compatible
>>>> +  clones made by second source vendors.
>>>> +
>>>> +compatible:
>>>> +  - name: "skel,sk11"
>>>> +  - name: "faux,fx11"
>>>
>>> Is this an OR or AND? We need both.
>>
>> True, this only covers the OR case atm.
>>
>>> The complicated case is "one of {specific names} followed by {generic name}."
>>
>> I need to rethink these. I do have deprecated: tag for that case and
>> possibly "name:" gets split to "generic:" and "specific:" and we can
>> then do the right thing.
>>
>> For the above I would have:
>>
>> compatible:
>>   - specific: "skel,sk11"
>>   - specific: "faux,fx11"
> 
> This can still be an AND or OR relationship.
> 
>>
>> and something like the Allwinner simple framebuffer would be:
>>
>> compatible:
>>   - generic: "simple-framebuffer"
>>   - specific: "allwinner,simple-framebuffer"
>>
>> where our validator would insist on seeing one of the specific: tags
>> along with the generic: tag. A lot of bindings, given current doc
>> patterns would just have two tags like above.
> 
> We could follow Pantelis' suggestion and do C style logic expressions:
> 
> name: ("chip1-ipblock" || "chip2-ipblock") && "generic-ipblock"
> 
> 
>>
>>>> +    description: A clone of the original sk11 device
>>>> +
>>>> +required:
>>>> +  - name: "reg"
>>>
>>> We definitely need type info from the start.
>>
>> Are you sure? Ugh.
>>
>> My big contention here is that we don't carry that content in the
>> docs today so we shouldn't try to add it in the initial conversion.
> 
> What? We have that all over the place. Boolean properties, phandles, etc.

Piling on...  If the type information is obvious in the current binding
doc, then it should be captured in the conversion and not lost.  If the
information is not currently present then it should not be required in
the conversion step.

Which means that we need to define the how to specify the type information.


>> I think you are right in that we should have it documented in the
>> schema but I'm concerned that we make the starting conversion
>> effort too large by adding type info to all converted docs.
> 
> How we do the conversion certainly needs to be worked out. However,
> first we need some sort of agreement that yes this looks promising
> before we spend too much time on it. We need to wait for all the YAML
> haters to come out of hiding. :) Once the direction is settled, then
> we can iron out the details of the structure. Finally, then we can
> determine how to do the conversion. Just because we define how type
> information looks doesn't mean it has to be there day 1.
> 
>> I think we gain a lot even without at the start, but I understand
>> why we need it and how it will help reduce the review firehose
>> with the associated validation tools.
>>
>>>
>>>> +    description: chip select address of skeleton device
>>>> +    reference: spi-slave
>>>
>>> I would like to not have to list properties if the inherited binding
>>> lists it. The problem is we need to say how many cells and the order
>>> (not a problem here, but for mmio devices).
>>
>> Yeah, make sense.
>>
>>> Perhaps we can list the reference at the top level for the node
>>> instead of for every property.
>>
>> That's a good point. I was wondering if per prop references would
>> get unwieldy once we actually add them into all the converted docs.
>> There's a huge number of existing docs without proper references.
> 
> Yes, in many cases they are implied. It should be simple enough to
> generate the list though whether they point to the doc file or just
> list the properties.
> 
>>
>> Ok, I'll take a look at collecting references per group of properties.
>>
>>>
>>>> +  - name: "spi-max-frequency"
>>>> +    description: >
>>>> +      Maximum SPI clocking speed of skeleton device in Hz, must be
>>>> +      1000000
>>>> +    reference: spi-slave
>>>
>>> Rather than listing the property and having constraint in description,
>>> perhaps we could add constraints like this:
>>>
>>> - spi-max-frequency-range: 1000000 1000000
>>>
>>> Or groups of constraints:
>>>
>>> - spi-max-frequency-constraints:
>>>   range: 1000000 1000000
>>>   some-other-constraint: <value>
>>
>> I was hoping to avoid this to start due to my argument for keeping it
>> simple for the first pass at conversion. However, the latter looks
>> flexible enough. We have cases with enumerated values as well to handle
>> the require some thought.
> 
> We can always add constraints as a second step, but I'd like to define
> the structure at least.

Agreed, this is something that I would like to see defined, just to make
sure that we can do it in a reasonable way with the proposed tool.


> Rob
> 


-Frank

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Rob Herring]

On Fri, Aug 28, 2015 at 12:13 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> On Fri, Aug 28, 2015 at 09:26:17AM -0500, Rob Herring wrote:
>> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> > During the Device Tree microconference at Linux Plumbers 2015, we had
>> > a short discussion about how to improve DT Binding Documentation. A

[...]

>> > One caveat with YAML is it does not tolerate tabs. Yes, I said it.
>> > No tabs! This can be managed with proper editor modes and also with
>> > helper scripts to strip tabs to aid in people passing planned
>> > checkpatch.pl checks that would run YAML DT Binding specific tag
>> > validators for new bindings.
>>
>> What do parsers do with tabs? Throw an error?
>
> Yes, they throw an error. Keep in mind that most of what I used to start
> are general purpose conversion tools on top of a particular scripting
> languages's high level binding to libyaml. The error output leaves a
> bit to be desired for our use case. In any case, when I was developing
> the skeleton.yaml I used the yaml script from
> https://github.com/ryo/yamltools to catch all these syntax errors I
> was inserting..like tabs. the PyYaml binding being used in my PoC
> dtgendoc does the same thing but I don't gracefully handle those
> errors like we could.
>
>> Beyond tabs, how do we check files can be parsed both generically and
>> for any binding specific requirements. We now need a schema for
>> checking the schema. We need some equivalent to compile testing.
>
> Right. So, I think what you are touching on is something I should
> have expanded on in the TODO list. Basically, we need a scripted
> tool that we run from checkpatch.pl that 1) reads the .yaml and
> validates the YAML itself (that comes for free in the high level
> parsers) reporting errors in a sensical manner 2) validates our
> DT binding specific tags

We all know that no one runs checkpatch.pl. ;) I really want the basic
checking of the doc files to run from make (and run by 0-day). The
tool dependency could be an issue though. However, DocBook builds from
make and I don't think many people check that regularly. Then there is
using the binding docs to check dts files. That should probably be
part of the dtb building.

> Now, I would caution about trying to do too much on Day 1 or we
> could end up back at the "never doing anything" stage.

Certainly, but I would like to have a plan for what Day 2 and 3 look like.

> It would
> be an improvement to simply check that the basic tags exist as
> shown in the [R] or [O] fields in the documentation. One thing
> I should point out is that I carefully avoided marking some tags
> as [R] where existing bindings don't have them...even if logically,
> a description should be required on every binding. The idea here
> is to avoid updating content at the same time that we are updating
> the format. Rather, I think it would be better to get the base
> format updated, then come back with a janitorial team and add
> descriptions (since now we can generate a worklist of those
> bindings missing a top-level description) and systematically
> fix those and review with the appropriate maintainers.

Yes. Any checking would be a great improvement.

[...]

>> > When we decide on a text markup format that is acceptable, then the
>> > next step is to convert all the bindings. That process would start
>> > with the complete set of generic bindings as they will be referenced
>> > by the actual device bindings.
>>
>> You are going to do that for everyone, right? ;)
>
> Let's just say that I'm banking on others helping here once we have
> a format agreed upon. If we can hold the binding doc schema definition
> initially to just define tags for content that already exists in our
> textual binding docs, the effort for conversion is tolerable. To give
> an example, that phy-bindings.txt, it took 15 minutes to convert and
> and pass through the yaml parser and dtgendoc. The reason is that it's
> pure reformatting work. It doesn't take any special knowledge of the
> hardware and it doesn't involve reviewing dts files to extra
> additional information. Some of the annoyances can be streamlined
> like tab stripping and handling the two space indentation to make
> this process faster. One of my next things is to get a simple tool
> going that reports problems with conversions, essentially what I
> said was needed to integrate with checkpatch, so this process of
> conversion is even faster. Trivial peripheral bindings like eeprom.txt
> can be done in 5 minutes or so right now.

What if instead of using the docs as a starting point, we use dts
files as a starting point? They give us something parse-able that we
can do some automation with.

Additionaly, perhaps we can do a mass conversion of all the doc files
from txt to yaml where all the current text is converted to comments
and we fill in boilerplate and whatever we can convert with some
automation? The downside here would be it will be hard to tell which
conversions are complete.

Putting both together, we could then fairly easily for example extract
compatible strings from dts files, lookup which doc file they are in
and fill in the compatible string info.

> If we decide we must have tags like "type:" in the initial binding
> doc schema definition *and* we must add that content in each
> conversion, then this becomes more time consuming to validate that
> information against working dts files. IMHO, we'd be better off
> to get the base format straight, addressing missing pieces like
> all the compatible permutations, and convert them all with
> just that content. After that, we come back and add new content
> features like type: tagging. I'm trying to find a reasonable
> place to do this incrementally since the volume of bindings to
> convert is enormous.

I wouldn't say we have to add it, but we need to maintain type info.

Are you proposing we can actually validate dts files on day 1? That I
would expect to come later.

Rob

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Tim Bird]

On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> During the Device Tree microconference at Linux Plumbers 2015, we had
> a short discussion about how to improve DT Binding Documentation. A
> number of issues were raised (again, as these things have been
> discussed in the past) including:
>
>         * Inconsistency between binding documents due to prose text
>           format.
>         * Inability to reliably machine read bindings for mass update
>           or search.
>         * Bit rot of bindings as new conventions are agreed upon but
>           only new bindings are changed.
>
> Grant Likely probably summed up the issue best with "...as long as
> bindings are human readable, we'll have issues...".
I'd rephrase that to "... as long as bindings are not machine readable,
we'll have issues...".  Human readable and machine readable are not
mutually exclusive.  XML is not, IMHO, human readable, but YAML looks
pretty good, so I think it's a good choice.

> The context
> of that comment was, of course, regarding our current documents
> written in very inconsistent prose style. When the topic of needing
> the bindings in a rigid format was raised, there was general head
> nodding that this was needed. It was noted that this has been
> discussed many times before and nothing has been done.
>
> My proposed solution to the problem is to convert all DT bindings
> a rigid text markup format. In choosing a text markup language my
> requirements were:
>
>         1) Human readable
>         2) Well documented
>         3) Easy to translate to other data formats
>         4) Well supported by tools and libraries
>
> After looking at a number of markup options, YAML stood out as the
> one that meets all of these requirements. The YAML syntax is adopted
> in many projects specifically because of the high level of readability.
> A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
> There's a number of tools to convert between YAML and other popular
> data formats such as JSON and XML. XML was cited by Behan Webster
> during the microconference as an important data format as the type
> of developers that may produce comprehensive DTS Binding validation
> tools will want to use XML. Every major scripting language has a
> high level binding to the low level libyaml C library to facilitate
> handling of YAML data files.
>
> One caveat with YAML is it does not tolerate tabs. Yes, I said it.
> No tabs! This can be managed with proper editor modes and also with
> helper scripts to strip tabs to aid in people passing planned
> checkpatch.pl checks that would run YAML DT Binding specific tag
> validators for new bindings.

This will be an endless source of pain.  Based on your examples,
it looks like we only need a very small subset of YAML.  I'd be willing
to write a parser for the language, but we might lose the ability to
do conversions to other formats.  It depends on how comprehensive
and automatic the helper scripts to strip tabs would be.

>
> The scope of the initial YAML DT Binding format was specifically
> limited to supporting *only* the content we have in bindings today.
> The idea here is to propose and agree on something that will take
> us just a few steps in the right direction. If we move *all* current
> binding content to a machine parseable format, additional features
> can be added with more automation and scripting. As it stands today,
> because of the inconsistency of the wording of the files, we can't
> add a lot of new features to the content until we convert what we
> have today into a standard format.
>
> With that said, it should be noted that some new features such as
> "type" tags to indicate cell types could be added to support
> additional DTS validation beyond what the current content supports.
> Another possibility is adding "range" type information to validate
> the legal values for a cell.

I would recommend we add type and something for value constraints.
Even if we don't add populate those immediately, I think thinking
through some of the possible uses will be helpful to avoid doing
something dumb.
'choices' or 'enums' seem frequently used.  I see lots of "must be one
of..." in the binding docs I read.

Another important thing to document would be a 'superclass' or
'inherit-from' attribute, for things that are of a specific type (i2c,
regulator, clock, etc.)

> This series is broken up into three major parts:
>
> 1) The documentation defining the YAML DT binding format
> 2) A skeleton device binding example illustrating use of this format
> 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
>    ti-phy.txt
>
> As a proof of concept of what can be done with a proper machine
> readable DT binding source file, there's a simple markdown document
> generator at https://github.com/konsulko/dtgendoc. Also, to see
> actual output from the generator, the generated markdown from those
> bindings is viewable at https://github.com/konsulko/dtgendoc/wiki
>
> There's a lot of other possibilities for validation tools using
> only the data we have today in the bindings. In addition, Frank
> Rowand covered some DT debug techniques that would benefit from
> the binding documentation being 100% reliably searchable.
>
> I found it useful to see a side-by-side view of a converted doc
> versus the original content, so here's a screenshot of eeprom.txt
> vs. eeprom.yaml:
> https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml
>
> When we decide on a text markup format that is acceptable, then the
> next step is to convert all the bindings. That process would start
> with the complete set of generic bindings as they will be referenced
> by the actual device bindings.

Although there are a lot of files to convert, I think the job is manageable,
and can be done with conversion sprints at conferences and the like.
You don't have to be a kernel 'C' coder to do a conversion, so I think
a lot of non-traditional contributers can help out.  I'm already
thinking about what could be done at an Embedded Linux Conference.

One thing that would help would be a conversion guide.  It would have
things like: "if you see this type of language, put that in the
description, some examples are ..."

I'll have more comments on individual patches.
 -- Tim

Re: [RFC PATCH 1/5] Documentation: dt-bindings: add documentation on new DT binding format

[Tim Bird]

On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Documentation explaining the syntax and format of the YAML-based DT binding
> documentation.
>
> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> ---
>  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
>  1 file changed, 106 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
>
> diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> new file mode 100644
> index 0000000..f9acc22
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> @@ -0,0 +1,106 @@
> +--------------------------
> +Device Tree Binding Format
> +--------------------------
> +
> +Background
> +----------
> +
> +DT bindings historically were written as text in prose format which
> +led to issues in usability of that source documentation. Some of
> +these issues include the need to programmatically process binding
> +source documentation to do DTS validation, perform mass updates to
> +format/style, and to generate publishable documentation in HTML or
> +PDF form.
> +
> +Overview
> +--------
> +
> +The DT binding format is based on the YAML text markup language.
> +Although there are many text markup options available, YAML
> +fulfills all requirements considered for a DT binding source format
> +which include:
> +
> +1) Must be human readable
> +2) Must be easily translated to other data formats (XML, JSON, etc).
> +3) Must have sufficient tools and libraries to enable developers to
> +   build new tools for DT binding processing
> +4) Must have a complete spec to refer to syntax
> +
> +YAML is documentated in the specification found at
> +http://www.yaml.org/spec/1.2/spec.html
> +
> +The required YAML DT binding tag format and syntax are defined in
> +the following sections.
> +
> +YAML DT Binding Syntax
> +----------------------
> +
> +* Lines starting with "#" are comments and not part of the binding itself
> +* "%YAML 1.2" starts a file, indicating the version of YAML in use

I think it would be good to specify a DT Binding Format version as well.
I would expect we may expand the syntax, or we may alter it, so it
would be good to have a number for our specific syntax.  This would
help in some circumstances, if we make multiple passes over the
documents to achieve our conversion.  Maybe this should be added
below.

> +* "---" starts a binding document
> +* "..." ends a binding document
> +* Multiple binding documents may exist in a single file
> +* Tabs are not permitted
> +* Scope is denoted by indentation of two spaces
> +* Key value pairs are denoted by "key: value"
> +* Sequences are denoted by "- "
> +* Scalar values may convert newlines to spaces and preserve blank
> +  lines for long description formatting using ">"
> +* Scalar values may escape all reserved characters and preserve
> +  newlines by using "|" to denote literal style
> +
> +For additional information on YAML syntax, refer to the specification
> +at http://www.yaml.org/spec/1.2/spec.html
> +
> +YAML DT Binding Format
> +----------------------
> +
> +The following YAML types are supported in the DT binding format:
> +

Based on my comment above, I recommend:
[R] binding-format: 1

> +* [R] id: unique identifier in property form (e.g. skel-device)
> +
> +* [R] title: title of the binding
> +
> +* [O] maintainer: sequence of maintainers
> +      [R] name: name and email of maintainer or mailing list in RFC822
> +                form.
> +
> +* [O] description: full description of the binding
> +
> +* [O] compatible: sequence of valid compatible descriptors
> +      [R] name: the compatible string surrounded in double quotes
> +      [O] deprecated: a deprecated compatible string surrounded in
> +                      double quotes
> +      [O] description: description of the compatible string

compatible is a property like all the others, and I think it's confusing
to have it in a separate section.  For example, when listing other
properties, the "name:" field specifies the property name, but for
the compatible section the "name:" field specifies the possible values
for the compatible property.  This is (at least to me) quite confusing.

Can we make this a regular property, with an option 'deprecated' attribute.
I guess we may need to distinguish between deprecated properties and
deprecated vales of properties.
The above 'deprecated' specifies deprecated values for the compatible
property, while
the section below specifies deprecated properties.  I don't see a way
to indicate a deprecated
value for a still-existing property, but that may be too much detail for now.


> +* [O] required: sequence of required properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id

I would recommend we add types and value constraints to the
binding schema:

[O] type: type of value for this property, and if specified must be
one of the following (without quotes):
"string", "number", "array", "phandle"
[O] value constraint: specifies a limit on the value for a property.
This can be specified as English text, or be a range or choice constraint.
The latter two should be expressed with the following exact english
phrases (without quotes):
"must be in the range x to y, inclusive"
where x and y are numbers
"must be one of <choice 1>, <choice 2>, ..."
where <choice x> is a string, which should be quoted
if it includes a comma.

Any other constraint is expressed in free-form English text

I would recommend adding these to this binding schema doc, but
not requiring them on the first conversion pass of the docs.

Basically my idea is that you want the each pass over the binding
documents to be as painless as possible.  Every time a user encounters
something they're not sure about, it will slow them down.  There will
be lots of ambiguous situations, and we want the process to be as
lossless as possible, so I would recommend a strategy of doing as
much as people can, and if in doubt just put any outstanding text
in the descriptions.  The descriptions will always be free-form text,
and we can come along later and convert these to types, constraints,
as we see fit.

> +* [O] optional: sequence of optional properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] deprecated: sequence of deprecated properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [R] example: sequence of examples:
> +      [R] dts: DT source of example usage. The example text must use
> +               literal style ("|") so that it retains indentation and
> +               newlines.
> +      [O] description: description of the example
> +

I'd recommend having a top-level notes section:
* [O] note: any other notes about this node that are not covered by
preceding property descriptions.

> +Note: [R] and [O] denote required and optional fields, respectively.

I'd put this note before the format description.

 -- Tim

> +
> +Skeleton Binding
> +----------------
> +
> +The skeleton.yaml binding found in the top of the DT binding tree
> +is the canonical example of syntax and format to use when writing
> +a DT binding document. It is maintained with the latest formatting
> +conventions, making it the best starting point when writing a new DT
> +binding.
> --

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Tim Bird]

On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:

> Now, I would caution about trying to do too much on Day 1 or we
> could end up back at the "never doing anything" stage.

I strongly agree with that (despite my having just sent a bunch of comments
on the binding format document).

> It would
> be an improvement to simply check that the basic tags exist as
> shown in the [R] or [O] fields in the documentation. One thing
> I should point out is that I carefully avoided marking some tags
> as [R] where existing bindings don't have them...even if logically,
> a description should be required on every binding. The idea here
> is to avoid updating content at the same time that we are updating
> the format.

I strongly agree with this as well.  I think that the goal for the first
pass should be "no lost text".  That is, if it doesn't convert into something
in the binding format, it should still be preserved somewhere, for evaluation
and formatting on a subsequent pass.

> Rather, I think it would be better to get the base
> format updated, then come back with a janitorial team and add
> descriptions (since now we can generate a worklist of those
> bindings missing a top-level description) and systematically
> fix those and review with the appropriate maintainers.

Agreed.  It will be very useful, once we get our first pass done, to
be able to find: 1) documents not converted yet, 2) property definitions
that are missing certain fields, 3) things that could still be formatted,
from text that is remaining in the description strings or other free-form text.
This should be easy to script.

>> An example such as checking that compatible strings are documented as
>> checkpatch.pl does would be nice. Roughly, that would be just list all
>> compatible values.
>
> Ok, so my comments above were strictly about a validator for the
> binding doc submission itself. I can add an example based on your
> checkpatch.pl to adapt it to the .yaml compatible tags.
>
>> > The scope of the initial YAML DT Binding format was specifically
>> > limited to supporting *only* the content we have in bindings today.
>> > The idea here is to propose and agree on something that will take
>> > us just a few steps in the right direction. If we move *all* current
>> > binding content to a machine parseable format, additional features
>> > can be added with more automation and scripting. As it stands today,
>> > because of the inconsistency of the wording of the files, we can't
>> > add a lot of new features to the content until we convert what we
>> > have today into a standard format.
>> >
>> > With that said, it should be noted that some new features such as
>> > "type" tags to indicate cell types could be added to support
>> > additional DTS validation beyond what the current content supports.
>> > Another possibility is adding "range" type information to validate
>> > the legal values for a cell.
>> >
>> > This series is broken up into three major parts:
>> >
>> > 1) The documentation defining the YAML DT binding format
>> > 2) A skeleton device binding example illustrating use of this format
>> > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
>> >    ti-phy.txt
>> >
>> > As a proof of concept of what can be done with a proper machine
>> > readable DT binding source file, there's a simple markdown document
>> > generator at https://github.com/konsulko/dtgendoc. Also, to see
>> > actual output from the generator, the generated markdown from those
>> > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki
>>
>> Nice.
>>
>> > There's a lot of other possibilities for validation tools using
>> > only the data we have today in the bindings. In addition, Frank
>> > Rowand covered some DT debug techniques that would benefit from
>> > the binding documentation being 100% reliably searchable.
>> >
>> > I found it useful to see a side-by-side view of a converted doc
>> > versus the original content, so here's a screenshot of eeprom.txt
>> > vs. eeprom.yaml:
>> > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml
>> >
>> > When we decide on a text markup format that is acceptable, then the
>> > next step is to convert all the bindings. That process would start
>> > with the complete set of generic bindings as they will be referenced
>> > by the actual device bindings.
>>
>> You are going to do that for everyone, right? ;)
>
> Let's just say that I'm banking on others helping here once we have
> a format agreed upon. If we can hold the binding doc schema definition
> initially to just define tags for content that already exists in our
> textual binding docs, the effort for conversion is tolerable. To give
> an example, that phy-bindings.txt, it took 15 minutes to convert and
> and pass through the yaml parser and dtgendoc. The reason is that it's
> pure reformatting work. It doesn't take any special knowledge of the
> hardware and it doesn't involve reviewing dts files to extra
> additional information. Some of the annoyances can be streamlined
> like tab stripping and handling the two space indentation to make
> this process faster. One of my next things is to get a simple tool
> going that reports problems with conversions, essentially what I
> said was needed to integrate with checkpatch, so this process of
> conversion is even faster. Trivial peripheral bindings like eeprom.txt
> can be done in 5 minutes or so right now.
>
> If we decide we must have tags like "type:" in the initial binding
> doc schema definition *and* we must add that content in each
> conversion, then this becomes more time consuming to validate that
> information against working dts files. IMHO, we'd be better off
> to get the base format straight, addressing missing pieces like
> all the compatible permutations, and convert them all with
> just that content. After that, we come back and add new content
> features like type: tagging. I'm trying to find a reasonable
> place to do this incrementally since the volume of bindings to
> convert is enormous.
>
> But to answer your question, if we get a format I'll do
> conversions and hope I'm not alone.

I'm sure others will help out.  I will, and I'm pretty sure we can get
some conversion sprints set up at conferences (I know I can set aside
some time or resources at ELC in the spring - it might be too late for
ELCE in October to set up a scheduled block of time, but we can start
getting the word out.)  As I said in my other e-mail, one doesn't have
to be a kernel coder to do this, and the conversions should be pretty
straight-forward.
 -- Tim

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Matt Porter]

On Fri, Aug 28, 2015 at 01:55:11PM -0500, Rob Herring wrote:
> On Fri, Aug 28, 2015 at 12:49 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
> >> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >> > Add a skeleton DT binding document that serves as the canonical
> >> > example for implementing YAML-based DT bindings documentation.
> >> > The skeleton binding illustrates use of all fields and variations
> >> > described in the dt-binding-format.txt documentation.
> >> >
> >> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> >> > ---
> >> >  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
> >> >  1 file changed, 98 insertions(+)
> >> >  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> >> >
> >> > diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
> >> > new file mode 100644
> >> > index 0000000..175965f
> >> > --- /dev/null
> >> > +++ b/Documentation/devicetree/bindings/skeleton.yaml
> >> > @@ -0,0 +1,98 @@
> >> > +%YAML 1.2
> >> > +---
> >> > +id: skel-device
> >> > +
> >> > +title: Skeleton Device
> >> > +
> >> > +maintainer:
> >> > +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >>
> >> We'd want to tie this into get_maintainers.pl obviously.
> >
> > Right, I broke my rule of "no new tags for content we don't already
> > have" by adding this. It stems out of the discussion at LPC where
> > Mark suggested we could avoid the core bindings being moved by adding
> > maintainer info into the binding itself.
> 
> If you stated that rule, I missed it...

Mental list of rules I made for my initial cut at this. I was trying
really hard to have this be a cut and paste reformatting to start and
then see where comments take us.

> > This is an area where more docs are needed. Ideally, on the first pass
> > conversion we would not attempt to populate these. I had considered
> > that we *could* only add maintainer: tags on core bindings to and
> > adjust get_maintainers.pl to use that info to override the standard
> > directory-based info if it exists. I don't think it's necessary to add
> > a specific maintainer for all of the peripheral bindings if the default
> > is the subsystem maintainer and the dt list. We could at least start
> > directing core binding discussion to the -spec list which is where I
> > think you'd like it.
> 
> Yes, certainly this can solve my problem around handling core bindings.
> 
> Some subsystem maintainers require a driver have a maintainer. We
> could certainly enforce that too. We could probably automatically
> populate this with the original author as a starting point. OTOH,
> bindings should not change frequently.

Populating with the original author seems easy enough.

> >> > +description: >
> >> > +  The Skeleton Device binding represents the SK11 device produced by
> >> > +  the Skeleton Corporation. The binding can also support compatible
> >> > +  clones made by second source vendors.
> >> > +
> >> > +compatible:
> >> > +  - name: "skel,sk11"
> >> > +  - name: "faux,fx11"
> >>
> >> Is this an OR or AND? We need both.
> >
> > True, this only covers the OR case atm.
> >
> >> The complicated case is "one of {specific names} followed by {generic name}."
> >
> > I need to rethink these. I do have deprecated: tag for that case and
> > possibly "name:" gets split to "generic:" and "specific:" and we can
> > then do the right thing.
> >
> > For the above I would have:
> >
> > compatible:
> >   - specific: "skel,sk11"
> >   - specific: "faux,fx11"
> 
> This can still be an AND or OR relationship.

Ok, so this might need to be sequences of and: and or:
key value pairs to reflect the combinations.

> > and something like the Allwinner simple framebuffer would be:
> >
> > compatible:
> >   - generic: "simple-framebuffer"
> >   - specific: "allwinner,simple-framebuffer"
> >
> > where our validator would insist on seeing one of the specific: tags
> > along with the generic: tag. A lot of bindings, given current doc
> > patterns would just have two tags like above.
> 
> We could follow Pantelis' suggestion and do C style logic expressions:
> 
> name: ("chip1-ipblock" || "chip2-ipblock") && "generic-ipblock"

That might work. More specialized parsing, but it probably is needed
to handle property values ranges anyway.

> >> > +    description: A clone of the original sk11 device
> >> > +
> >> > +required:
> >> > +  - name: "reg"
> >>
> >> We definitely need type info from the start.
> >
> > Are you sure? Ugh.
> >
> > My big contention here is that we don't carry that content in the
> > docs today so we shouldn't try to add it in the initial conversion.
> 
> What? We have that all over the place. Boolean properties, phandles, etc.

I mean that it's not always explicitly called out in peripheral
DT bindings...it's inconsistently documented. I think that by properly
documenting this in the generic bindings as you are suggesting and
getting those properties by reference, the task of documenting these
won't be so bad.

> > I think you are right in that we should have it documented in the
> > schema but I'm concerned that we make the starting conversion
> > effort too large by adding type info to all converted docs.
> 
> How we do the conversion certainly needs to be worked out. However,
> first we need some sort of agreement that yes this looks promising
> before we spend too much time on it. We need to wait for all the YAML
> haters to come out of hiding. :) Once the direction is settled, then

I was expecting the YAML haters by now, I'll give it another few days.
:)

> we can iron out the details of the structure. Finally, then we can
> determine how to do the conversion. Just because we define how type
> information looks doesn't mean it has to be there day 1.

Agreed.
 
> > I think we gain a lot even without at the start, but I understand
> > why we need it and how it will help reduce the review firehose
> > with the associated validation tools.
> >
> >>
> >> > +    description: chip select address of skeleton device
> >> > +    reference: spi-slave
> >>
> >> I would like to not have to list properties if the inherited binding
> >> lists it. The problem is we need to say how many cells and the order
> >> (not a problem here, but for mmio devices).
> >
> > Yeah, make sense.
> >
> >> Perhaps we can list the reference at the top level for the node
> >> instead of for every property.
> >
> > That's a good point. I was wondering if per prop references would
> > get unwieldy once we actually add them into all the converted docs.
> > There's a huge number of existing docs without proper references.
> 
> Yes, in many cases they are implied. It should be simple enough to
> generate the list though whether they point to the doc file or just
> list the properties.

Ok.

> 
> >
> > Ok, I'll take a look at collecting references per group of properties.
> >
> >>
> >> > +  - name: "spi-max-frequency"
> >> > +    description: >
> >> > +      Maximum SPI clocking speed of skeleton device in Hz, must be
> >> > +      1000000
> >> > +    reference: spi-slave
> >>
> >> Rather than listing the property and having constraint in description,
> >> perhaps we could add constraints like this:
> >>
> >> - spi-max-frequency-range: 1000000 1000000
> >>
> >> Or groups of constraints:
> >>
> >> - spi-max-frequency-constraints:
> >>   range: 1000000 1000000
> >>   some-other-constraint: <value>
> >
> > I was hoping to avoid this to start due to my argument for keeping it
> > simple for the first pass at conversion. However, the latter looks
> > flexible enough. We have cases with enumerated values as well to handle
> > the require some thought.
> 
> We can always add constraints as a second step, but I'd like to define
> the structure at least.

Ok, this sounds reasonable, I'm goling to take a stab at reorganizing
the format based on the big areas of references, property types, and
property constraints.

-Matt

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Sun, Aug 30, 2015 at 03:04:20PM -0700, Frank Rowand wrote:
> On 8/27/2015 10:23 PM, Matt Porter wrote:
> > During the Device Tree microconference at Linux Plumbers 2015, we had
> > a short discussion about how to improve DT Binding Documentation. A
> > number of issues were raised (again, as these things have been
> > discussed in the past) including:
> > 
> > 	* Inconsistency between binding documents due to prose text
> > 	  format.
> > 	* Inability to reliably machine read bindings for mass update
> >           or search.
> >         * Bit rot of bindings as new conventions are agreed upon but
> > 	  only new bindings are changed.
> > 
> > Grant Likely probably summed up the issue best with "...as long as
> > bindings are human readable, we'll have issues...". The context
> > of that comment was, of course, regarding our current documents
> > written in very inconsistent prose style. When the topic of needing
> > the bindings in a rigid format was raised, there was general head
> > nodding that this was needed. It was noted that this has been
> > discussed many times before and nothing has been done.
> > 
> > My proposed solution to the problem is to convert all DT bindings
> > a rigid text markup format. In choosing a text markup language my
> > requirements were:
> > 
> > 	1) Human readable
> > 	2) Well documented
> > 	3) Easy to translate to other data formats
> > 	4) Well supported by tools and libraries
> 
> Thanks for taking this on!  The proposal looks like a great start.
> The requirements look good.
> 
> 
> > After looking at a number of markup options, YAML stood out as the
> > one that meets all of these requirements. The YAML syntax is adopted
> > in many projects specifically because of the high level of readability.
> > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
> > There's a number of tools to convert between YAML and other popular
> > data formats such as JSON and XML. XML was cited by Behan Webster
> > during the microconference as an important data format as the type
> > of developers that may produce comprehensive DTS Binding validation
> > tools will want to use XML. Every major scripting language has a
> > high level binding to the low level libyaml C library to facilitate
> > handling of YAML data files.
> > 
> > One caveat with YAML is it does not tolerate tabs. Yes, I said it.
> > No tabs! This can be managed with proper editor modes and also with
> > helper scripts to strip tabs to aid in people passing planned
> > checkpatch.pl checks that would run YAML DT Binding specific tag
> > validators for new bindings.
> 
> I'm just a casual user of some markup languages, nowhere near knowledgeable
> or an expert, so will be listening to reasoned opinions on the choice of
> YAML.  But a quick look at the spec you referenced and the examples you
> provided gives me some confidence that YAML will work.
> 
> 
> > The scope of the initial YAML DT Binding format was specifically
> > limited to supporting *only* the content we have in bindings today.
> > The idea here is to propose and agree on something that will take
> > us just a few steps in the right direction. If we move *all* current
> 
> Yes, this is a great approach so we can start making reasonable progress.
> (Caveat below.)
> 
> > binding content to a machine parseable format, additional features
> > can be added with more automation and scripting. As it stands today,
> > because of the inconsistency of the wording of the files, we can't
> > add a lot of new features to the content until we convert what we
> > have today into a standard format.
> > 
> > With that said, it should be noted that some new features such as
> > "type" tags to indicate cell types could be added to support
> > additional DTS validation beyond what the current content supports.
> > Another possibility is adding "range" type information to validate
> > the legal values for a cell.
> 
> The caveat: I think we should look at some obvious features (such as
> "type" that you mention) to make sure that YAML and the chosen
> structuring will easily and effectively support the features.
> 
> Thinking about how some features might be implemented led to comments
> that I will make in replies to the other patches.
> 
> I agree that we should not attempt to add the additional information
> enabled by the new features if the information is not in the present
> bindings.  Adding the new information in a subsequent pass seems to be
> a good strategy.

Ok, I think Rob had a good point that we need to get the format for
these additional pieces of information in place even if we don't
populate them to start. Sounds like there's some agreement on this
approach.

> 
> > 
> > This series is broken up into three major parts:
> > 
> > 1) The documentation defining the YAML DT binding format
> > 2) A skeleton device binding example illustrating use of this format
> > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
> >    ti-phy.txt
> > 
> > As a proof of concept of what can be done with a proper machine
> > readable DT binding source file, there's a simple markdown document
> > generator at https://github.com/konsulko/dtgendoc. Also, to see
> > actual output from the generator, the generated markdown from those
> > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki
> > 
> > There's a lot of other possibilities for validation tools using
> > only the data we have today in the bindings. In addition, Frank
> > Rowand covered some DT debug techniques that would benefit from
> > the binding documentation being 100% reliably searchable.
> > 
> > I found it useful to see a side-by-side view of a converted doc
> > versus the original content, so here's a screenshot of eeprom.txt
> > vs. eeprom.yaml:
> > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml
> 
> This binding is your patch 3, but the side by side example you provide
> makes it much easier to follow along, so I'll comment here.
> 
> Part of the information about compatible got lost in the conversion.
> The part that is lost is:
> 
>    If there is no specific driver for <manufacturer>, a generic
>    driver based on <type> is selected.
> 
> I will discuss this a little more in response to patch 1, where I will
> ask whether compatible should be treated differently than other
> properties.  So maybe any further discussion of this should be in
> reply to that email.

Ok, yes, this is an area the needs additional work as Rob pointed out
with handling various AND and OR type constraints. I'll move discussion
to that patch though.

> > When we decide on a text markup format that is acceptable, then the
> > next step is to convert all the bindings. That process would start
> > with the complete set of generic bindings as they will be referenced
> > by the actual device bindings.
> > 
> > If the RFC wasn't explicit enough...comments welcome.
> > 
> > -Matt
> > 
> > Matt Porter (5):
> >   Documentation: dt-bindings: add documentation on new DT binding format
> >   Documentation: dt-bindings: add example DT binding document
> >   Documentation: dt-bindings: add YAML eeprom binding
> >   Documentation: dt-bindings: phy: add YAML generic PHY binding
> >   Documentation: dt-bindings: phy: add YAML TI PHY binding
> > 
> >  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++
> >  Documentation/devicetree/bindings/eeprom.yaml      |  44 ++++++
> >  .../devicetree/bindings/phy/phy-bindings.yaml      |  89 +++++++++++
> >  Documentation/devicetree/bindings/phy/ti-phy.yaml  | 166 +++++++++++++++++++++
> >  Documentation/devicetree/bindings/skeleton.yaml    |  98 ++++++++++++
> >  5 files changed, 503 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> >  create mode 100644 Documentation/devicetree/bindings/eeprom.yaml
> >  create mode 100644 Documentation/devicetree/bindings/phy/phy-bindings.yaml
> >  create mode 100644 Documentation/devicetree/bindings/phy/ti-phy.yaml
> >  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> > 
> 

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Frank Rowand]

On 9/1/2015 10:14 AM, Tim Bird wrote:
> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:

< snip >

>>
>> But to answer your question, if we get a format I'll do
>> conversions and hope I'm not alone.
> 
> I'm sure others will help out.  I will, and I'm pretty sure we can get
> some conversion sprints set up at conferences (I know I can set aside
> some time or resources at ELC in the spring - it might be too late for
> ELCE in October to set up a scheduled block of time, but we can start
> getting the word out.)  As I said in my other e-mail, one doesn't have
> to be a kernel coder to do this, and the conversions should be pretty
> straight-forward.
>  -- Tim
> 

A conversion sprint at ELCE sounds like a good idea if we can find a
good time to schedule it.  I can help, so there will be at least two
of us who can help people as they encounter issues.

-Frank

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Tim Bird]

On Fri, Aug 28, 2015 at 7:53 AM, Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
...
>> +example:
>> +  - dts: |
>> +      sk11@0 {
>> +              compatible = "skel,sk11";
>> +              reg = <0>;
>> +              spi-max-frequency = <1000000>;
>> +              spi-cs-high;
>> +      };
>
> At least in this example, we could generate it. Examples are nice, but
> we have dts files full of examples already. I get a fair number of
> "fix the example" patches, so maybe we should eliminate the simple
> ones.

I would hesitate to eliminate examples.  I've been saved by them on
a few occasions, when the dts files only had one or two instances
of a type of binding, somewhat different from each other, and the example helped
break the tie.  If there's something wrong with the example, it's a sign
of an out-of-date binding doc, just as much as if the text were wrong.
It ought to be possible to validate the example versus the binding doc
(as Pantelis says), so ultimately we should be able to catch errors here
as well.
 -- Tim

Re: [RFC PATCH 5/5] Documentation: dt-bindings: phy: add YAML TI PHY binding

[Tim Bird]

On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> Convert the ti-phy.txt binding to standard YAML DT binding
> format. This binding references the YAML generic PHY binding.
>
> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> ---
>  Documentation/devicetree/bindings/phy/ti-phy.yaml | 166 ++++++++++++++++++++++
>  1 file changed, 166 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/phy/ti-phy.yaml
>
> diff --git a/Documentation/devicetree/bindings/phy/ti-phy.yaml b/Documentation/devicetree/bindings/phy/ti-phy.yaml
> new file mode 100644
> index 0000000..ec93501
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/phy/ti-phy.yaml
> @@ -0,0 +1,166 @@
> +%YAML 1.2
> +---
> +id: omap-control-phy
> +
> +title: OMAP Control PHY
> +
> +compatible:
> +  - name: "ti,control-phy-otghs"
> +    description: if it has otghs_control mailbox register as on OMAP4.
> +  - name: "ti,control-phy-usb2"
> +    description: if it has Power down bit in control_dev_conf register
> +                 e.g. USB2_PHY on OMAP5.
> +  - name: "ti,control-phy-pipe3"
> +    description: if it has DPLL and individual Rx & Tx power control
> +                 e.g. USB3 PHY and SATA PHY on OMAP5.
> +  - name: "ti,control-phy-pcie"
> +    description: for pcie to support external clock for pcie and to
> +                 set PCS delay value. e.g. PCIE PHY in DRA7x
> +  - name: "ti,control-phy-usb2-dra7"
> +    description: if it has power down register like USB2 PHY on DRA7
> +                 DRA7 platform.
> +  - name: "ti,control-phy-usb2-am437"
> +    description: if it has power down register like USB2 PHY on AM437
> +                 platform.

I'm confused.  If these descriptions are multi-line, don't they need the '>'
syntax?

Also, I'm still confused by the "name:" field in this section.  These are
possible *values* for the compatible property, not the name of the field,
as all other property descriptions are.

...
 -- Tim

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Tim Bird]

On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand <frowand.list-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On 9/1/2015 10:14 AM, Tim Bird wrote:
>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>
> < snip >
>
>>>
>>> But to answer your question, if we get a format I'll do
>>> conversions and hope I'm not alone.
>>
>> I'm sure others will help out.  I will, and I'm pretty sure we can get
>> some conversion sprints set up at conferences (I know I can set aside
>> some time or resources at ELC in the spring - it might be too late for
>> ELCE in October to set up a scheduled block of time, but we can start
>> getting the word out.)  As I said in my other e-mail, one doesn't have
>> to be a kernel coder to do this, and the conversions should be pretty
>> straight-forward.
>>  -- Tim
>>
>
> A conversion sprint at ELCE sounds like a good idea if we can find a
> good time to schedule it.  I can help, so there will be at least two
> of us who can help people as they encounter issues.

Even if we don't find a block of time, we can do something like
announce a "contest", ask people to do something in their spare time,
and find some way to get them a raffle ticket if they submit a patch
with a conversion.  Then hold an extra prize drawing during the
closing session, with just those raffle tickets, and give someone a
nice award for contributing.  Of course, there's always the adage that
you should be careful what you measure and reward...  We don't want a
flood of crappy conversions, with a time constraint on the review.
I'll think some more about this.  An alternative would be to have a
contest announced ahead of the event, with enough time for people to
submit and get reviewed.

By the way - I presume the new docs will replace the existing ones,
but I could imagine wanting to have them live side-by-side
temporarily.  Any thoughts on this?  Will file name or location
changes be allowed during the conversion?
 -- Tim

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Rob Herring]

On Tue, Sep 1, 2015 at 1:03 PM, Tim Bird <tbird20d-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand <frowand.list-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On 9/1/2015 10:14 AM, Tim Bird wrote:
>>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>
>> < snip >
>>
>>>>
>>>> But to answer your question, if we get a format I'll do
>>>> conversions and hope I'm not alone.
>>>
>>> I'm sure others will help out.  I will, and I'm pretty sure we can get
>>> some conversion sprints set up at conferences (I know I can set aside
>>> some time or resources at ELC in the spring - it might be too late for
>>> ELCE in October to set up a scheduled block of time, but we can start
>>> getting the word out.)  As I said in my other e-mail, one doesn't have
>>> to be a kernel coder to do this, and the conversions should be pretty
>>> straight-forward.
>>>  -- Tim
>>>
>>
>> A conversion sprint at ELCE sounds like a good idea if we can find a
>> good time to schedule it.  I can help, so there will be at least two
>> of us who can help people as they encounter issues.
>
> Even if we don't find a block of time, we can do something like
> announce a "contest", ask people to do something in their spare time,
> and find some way to get them a raffle ticket if they submit a patch
> with a conversion.  Then hold an extra prize drawing during the
> closing session, with just those raffle tickets, and give someone a
> nice award for contributing.  Of course, there's always the adage that
> you should be careful what you measure and reward...  We don't want a
> flood of crappy conversions, with a time constraint on the review.
> I'll think some more about this.  An alternative would be to have a
> contest announced ahead of the event, with enough time for people to
> submit and get reviewed.

Sounds like a review nightmare. That's another reason why as much
automated conversion we can do, the better.

> By the way - I presume the new docs will replace the existing ones,
> but I could imagine wanting to have them live side-by-side
> temporarily.  Any thoughts on this?  Will file name or location
> changes be allowed during the conversion?

I proposed some ideas earlier in the thread. Either we can have both
side by side or do a mass conversion to YAML making the existing doc a
comment (add # prefix).

Any renames/moving should be separate (there's some clean-up I'd like
to there as well). Exact rules depend on the approach, but we need to
be able to tell which bindings conversions are not started, in
progress, or complete.

Rob

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Nicolas Ferre]

Le 01/09/2015 19:35, Tim Bird a écrit :
> On Fri, Aug 28, 2015 at 7:53 AM, Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> ...
>>> +example:
>>> +  - dts: |
>>> +      sk11@0 {
>>> +              compatible = "skel,sk11";
>>> +              reg = <0>;
>>> +              spi-max-frequency = <1000000>;
>>> +              spi-cs-high;
>>> +      };
>>
>> At least in this example, we could generate it. Examples are nice, but
>> we have dts files full of examples already. I get a fair number of
>> "fix the example" patches, so maybe we should eliminate the simple
>> ones.
> 
> I would hesitate to eliminate examples.  I've been saved by them on
> a few occasions, when the dts files only had one or two instances
> of a type of binding, somewhat different from each other, and the example helped
> break the tie.  If there's something wrong with the example, it's a sign
> of an out-of-date binding doc, just as much as if the text were wrong.
> It ought to be possible to validate the example versus the binding doc
> (as Pantelis says), so ultimately we should be able to catch errors here
> as well.

I back Tim's advice.
Example are so important that I regularly find myself thinking "ah-ha,
okay that's supposed to work like that" after having read the binding
documentation.

Bye,
-- 
Nicolas Ferre

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Nicolas Ferre]

Le 01/09/2015 19:35, Tim Bird a écrit :
> On Fri, Aug 28, 2015 at 7:53 AM, Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> ...
>>> +example:
>>> +  - dts: |
>>> +      sk11@0 {
>>> +              compatible = "skel,sk11";
>>> +              reg = <0>;
>>> +              spi-max-frequency = <1000000>;
>>> +              spi-cs-high;
>>> +      };
>>
>> At least in this example, we could generate it. Examples are nice, but
>> we have dts files full of examples already. I get a fair number of
>> "fix the example" patches, so maybe we should eliminate the simple
>> ones.
> 
> I would hesitate to eliminate examples.  I've been saved by them on
> a few occasions, when the dts files only had one or two instances
> of a type of binding, somewhat different from each other, and the example helped
> break the tie.  If there's something wrong with the example, it's a sign
> of an out-of-date binding doc, just as much as if the text were wrong.
> It ought to be possible to validate the example versus the binding doc
> (as Pantelis says), so ultimately we should be able to catch errors here
> as well.

I back Tim's advice.
Example are so important that I regularly find myself thinking "ah-ha,
okay that's supposed to work like that" after having read the binding
documentation.

Bye,
-- 
Nicolas Ferre

Re: [RFC PATCH 1/5] Documentation: dt-bindings: add documentation on new DT binding format

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 7836 bytes --]

On Fri, Aug 28, 2015 at 01:23:49AM -0400, Matt Porter wrote:
> Documentation explaining the syntax and format of the YAML-based DT binding
> documentation.

Sorry I've taken so long to comment; I've had sickness and a backlog
of day-job work to deal with.

> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> ---
>  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
>  1 file changed, 106 insertions(+)
>  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> 
> diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> new file mode 100644
> index 0000000..f9acc22
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> @@ -0,0 +1,106 @@
> +--------------------------
> +Device Tree Binding Format
> +--------------------------
> +
> +Background
> +----------
> +
> +DT bindings historically were written as text in prose format which
> +led to issues in usability of that source documentation. Some of
> +these issues include the need to programmatically process binding
> +source documentation to do DTS validation, perform mass updates to
> +format/style, and to generate publishable documentation in HTML or
> +PDF form.
> +
> +Overview
> +--------
> +
> +The DT binding format is based on the YAML text markup language.
> +Although there are many text markup options available, YAML
> +fulfills all requirements considered for a DT binding source format
> +which include:
> +
> +1) Must be human readable
> +2) Must be easily translated to other data formats (XML, JSON, etc).
> +3) Must have sufficient tools and libraries to enable developers to
> +   build new tools for DT binding processing
> +4) Must have a complete spec to refer to syntax
> +
> +YAML is documentated in the specification found at
> +http://www.yaml.org/spec/1.2/spec.html

I've read through this.  At least tentatively, YAML seems like a
pretty good choice to me.  It seems to have a pretty good balance
between human readability and machine parsability.


> +The required YAML DT binding tag format and syntax are defined in
> +the following sections.
> +
> +YAML DT Binding Syntax
> +----------------------
> +
> +* Lines starting with "#" are comments and not part of the binding itself
> +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> +* "---" starts a binding document
> +* "..." ends a binding document
> +* Multiple binding documents may exist in a single file
> +* Tabs are not permitted
> +* Scope is denoted by indentation of two spaces

I think the recommended indentation should be increased.  At just 2
spaces it's easy to lose track of indentation level if you scroll down
a page.

8 would match the kernel code style, but might get cumbersome since I
think our YAML structures will probably be more deeply nested in
practice than code blocks generally should be.

So, I'd suggest 4.

> +* Key value pairs are denoted by "key: value"
> +* Sequences are denoted by "- "
> +* Scalar values may convert newlines to spaces and preserve blank
> +  lines for long description formatting using ">"
> +* Scalar values may escape all reserved characters and preserve
> +  newlines by using "|" to denote literal style
> +
> +For additional information on YAML syntax, refer to the specification
> +at http://www.yaml.org/spec/1.2/spec.html
> +
> +YAML DT Binding Format
> +----------------------
> +
> +The following YAML types are supported in the DT binding format:

I find the wording above really confusing.  Mentioning "types" makes
me thinnk you're talking about the actual YAML tags, which usually
have typing information, but actually you seem to be talking about the
expected members of the top level YAML mapping.

Speaking of YAML tags, we really should specify the YAML schema we're
using here.  From the rest of the stuff, I'm assuming it's the "Core"
schema described in section 10.3 of the YAML spec.  That seems a
reasonable first choice to me, although eventually we may want to
define our own YAML schema.

> +* [R] id: unique identifier in property form (e.g. skel-device)
> +
> +* [R] title: title of the binding
> +
> +* [O] maintainer: sequence of maintainers
> +      [R] name: name and email of maintainer or mailing list in RFC822
> +                form.
> +
> +* [O] description: full description of the binding
> +
> +* [O] compatible: sequence of valid compatible descriptors
> +      [R] name: the compatible string surrounded in double quotes
> +      [O] deprecated: a deprecated compatible string surrounded in
> +                      double quotes
> +      [O] description: description of the compatible string

So, when thinking about DT validation, I like to make a distinction
between the binding's "key" and its requirements.  By the "key"
(because I haven't thought of a better term) I mean the thing that
tells you this binding is in play.  From a validator's point of view,
given a DT node and a library of bindings, it needs to check that if
the node matches the key, then it must match all the binding's requirements.

For most bindings that will be a particular value listed in
'compatible', but there are other cases.  I'm fine if 'compatible' is
the only style of "key" we support initially, but I'd like to see a
higher level structural break between the "key" part of the binding
and the requirements.

> +* [O] required: sequence of required properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] optional: sequence of optional properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id
> +
> +* [O] deprecated: sequence of deprecated properties:
> +      [R] name: name of the property surrounded in double quotes
> +      [R] description: description of the property
> +      [O] reference: optional reference to a binding id

I'm not sure if it will work better to put require / optional
etc. properties into different sections as you have here, or if it
would be better to tag each property with "required", "optional",
"required-when" and so forth.

> +* [R] example: sequence of examples:
> +      [R] dts: DT source of example usage. The example text must use
> +               literal style ("|") so that it retains indentation and
> +               newlines.
> +      [O] description: description of the example
> +
> +Note: [R] and [O] denote required and optional fields, respectively.
> +
> +Skeleton Binding
> +----------------
> +
> +The skeleton.yaml binding found in the top of the DT binding tree
> +is the canonical example of syntax and format to use when writing
> +a DT binding document. It is maintained with the latest formatting
> +conventions, making it the best starting point when writing a new DT
> +binding.

So the other thing I'd like to see in our very first implementation is
at least minimal checking of the binding itself.  Initially, just
checking that it is valid YAML would be enough, I think.  But the
quoting and nesting rules in YAML are pretty subtle, so I could easily
see someone writing something based on existing examples that looks ok
to a human, but doesn't quite meet the YAML spec rules.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [RFC PATCH 1/5] Documentation: dt-bindings: add documentation on new DT binding format

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 9429 bytes --]

On Tue, Sep 01, 2015 at 09:59:14AM -0700, Tim Bird wrote:
> On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Documentation explaining the syntax and format of the YAML-based DT binding
> > documentation.
> >
> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> > ---
> >  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
> >  1 file changed, 106 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> >
> > diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> > new file mode 100644
> > index 0000000..f9acc22
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> > @@ -0,0 +1,106 @@
> > +--------------------------
> > +Device Tree Binding Format
> > +--------------------------
> > +
> > +Background
> > +----------
> > +
> > +DT bindings historically were written as text in prose format which
> > +led to issues in usability of that source documentation. Some of
> > +these issues include the need to programmatically process binding
> > +source documentation to do DTS validation, perform mass updates to
> > +format/style, and to generate publishable documentation in HTML or
> > +PDF form.
> > +
> > +Overview
> > +--------
> > +
> > +The DT binding format is based on the YAML text markup language.
> > +Although there are many text markup options available, YAML
> > +fulfills all requirements considered for a DT binding source format
> > +which include:
> > +
> > +1) Must be human readable
> > +2) Must be easily translated to other data formats (XML, JSON, etc).
> > +3) Must have sufficient tools and libraries to enable developers to
> > +   build new tools for DT binding processing
> > +4) Must have a complete spec to refer to syntax
> > +
> > +YAML is documentated in the specification found at
> > +http://www.yaml.org/spec/1.2/spec.html
> > +
> > +The required YAML DT binding tag format and syntax are defined in
> > +the following sections.
> > +
> > +YAML DT Binding Syntax
> > +----------------------
> > +
> > +* Lines starting with "#" are comments and not part of the binding itself
> > +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> 
> I think it would be good to specify a DT Binding Format version as well.
> I would expect we may expand the syntax, or we may alter it, so it
> would be good to have a number for our specific syntax.  This would
> help in some circumstances, if we make multiple passes over the
> documents to achieve our conversion.  Maybe this should be added
> below.

Good point.  In fact, I think it's absolutely essential that we
include that.  This is a complicated business and we're bound to make
mistakes in the binding definition format - a version number gives us
a way to fix them without requiring changing all the existing bindings
at once.

[snip]
> > +* [O] compatible: sequence of valid compatible descriptors
> > +      [R] name: the compatible string surrounded in double quotes
> > +      [O] deprecated: a deprecated compatible string surrounded in
> > +                      double quotes
> > +      [O] description: description of the compatible string
> 
> compatible is a property like all the others, and I think it's confusing
> to have it in a separate section.  For example, when listing other
> properties, the "name:" field specifies the property name, but for
> the compatible section the "name:" field specifies the possible values
> for the compatible property.  This is (at least to me) quite confusing.
> 
> Can we make this a regular property, with an option 'deprecated' attribute.
> I guess we may need to distinguish between deprecated properties and
> deprecated vales of properties.
> The above 'deprecated' specifies deprecated values for the compatible
> property, while
> the section below specifies deprecated properties.  I don't see a way
> to indicate a deprecated
> value for a still-existing property, but that may be too much detail for now.

So, I think folding compatible in with the rest is a mistake.  There
are two important distinctions between compatible and the other
property descriptions:

1) (See my other post for context) The most important is that
compatible is the "key" for the binding.  It's what lets you know the
binding is in play, rather than just saying what the requirements are
once you know it's a relevant binding.

2) The condition we're typically listing for compatible is that it
must contain the given compatible string - it's valid and common to
also have additional compatible strings (e.g. for the future,
backwards-compatible revision of this device).  For most other
properties we're describing the *whole* value it should have.

Eventually I think we'll want a general way of expressing such
"equals" versus "contains" conditions, but that's a pretty hard
problem, so for the first cut I think it's reasonable to special case
'compatible'.


Incidentally, note that it would also be quite common for 'compatible'
to appear in *both* the "key" and the requirements.  e.g. the binding
for "foo,bardev-revision-b" could require that compatible also list
"foo,bardev-revision-a" if it's backwards compatible.

> > +* [O] required: sequence of required properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> 
> I would recommend we add types and value constraints to the
> binding schema:
> 
> [O] type: type of value for this property, and if specified must be
> one of the following (without quotes):
> "string", "number", "array", "phandle"

It might be worth re-using the type names used in old OF documents
here.  Although "prop-encoded-array" tends to cover an awful lot there.

> [O] value constraint: specifies a limit on the value for a property.
> This can be specified as English text, or be a range or choice constraint.
> The latter two should be expressed with the following exact english
> phrases (without quotes):
> "must be in the range x to y, inclusive"
> where x and y are numbers
> "must be one of <choice 1>, <choice 2>, ..."
> where <choice x> is a string, which should be quoted
> if it includes a comma.
> 
> Any other constraint is expressed in free-form English text

Requiring exact english phrases when we're already have a flexible
encoding scheme with YAML is pretty horrid.

I'd suggest instead that English constraints just go in the
description.  We can add new fields for specific types of machine
verifiable constraints as we implement them (e.g. "max", "min",
"allowed-values").

> I would recommend adding these to this binding schema doc, but
> not requiring them on the first conversion pass of the docs.
> 
> Basically my idea is that you want the each pass over the binding
> documents to be as painless as possible.  Every time a user encounters
> something they're not sure about, it will slow them down.  There will
> be lots of ambiguous situations, and we want the process to be as
> lossless as possible, so I would recommend a strategy of doing as
> much as people can, and if in doubt just put any outstanding text
> in the descriptions.  The descriptions will always be free-form text,
> and we can come along later and convert these to types, constraints,
> as we see fit.
> 
> > +* [O] optional: sequence of optional properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] deprecated: sequence of deprecated properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [R] example: sequence of examples:
> > +      [R] dts: DT source of example usage. The example text must use
> > +               literal style ("|") so that it retains indentation and
> > +               newlines.
> > +      [O] description: description of the example
> > +
> 
> I'd recommend having a top-level notes section:
> * [O] note: any other notes about this node that are not covered by
> preceding property descriptions.

Good call.  Although I'd suggest "description" to match the name used
elsewhere.

> > +Note: [R] and [O] denote required and optional fields, respectively.
> 
> I'd put this note before the format description.
> 
>  -- Tim
> 
> > +
> > +Skeleton Binding
> > +----------------
> > +
> > +The skeleton.yaml binding found in the top of the DT binding tree
> > +is the canonical example of syntax and format to use when writing
> > +a DT binding document. It is maintained with the latest formatting
> > +conventions, making it the best starting point when writing a new DT
> > +binding.
> > --

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Tue, Sep 01, 2015 at 09:14:24AM -0700, Tim Bird wrote:
> On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > During the Device Tree microconference at Linux Plumbers 2015, we had
> > a short discussion about how to improve DT Binding Documentation. A
> > number of issues were raised (again, as these things have been
> > discussed in the past) including:
> >
> >         * Inconsistency between binding documents due to prose text
> >           format.
> >         * Inability to reliably machine read bindings for mass update
> >           or search.
> >         * Bit rot of bindings as new conventions are agreed upon but
> >           only new bindings are changed.
> >
> > Grant Likely probably summed up the issue best with "...as long as
> > bindings are human readable, we'll have issues...".
> I'd rephrase that to "... as long as bindings are not machine readable,
> we'll have issues...".  Human readable and machine readable are not
> mutually exclusive.  XML is not, IMHO, human readable, but YAML looks
> pretty good, so I think it's a good choice.

Yes, that's exactly what I hoped to find with this experiment...that
we can have both.

> > The context
> > of that comment was, of course, regarding our current documents
> > written in very inconsistent prose style. When the topic of needing
> > the bindings in a rigid format was raised, there was general head
> > nodding that this was needed. It was noted that this has been
> > discussed many times before and nothing has been done.
> >
> > My proposed solution to the problem is to convert all DT bindings
> > a rigid text markup format. In choosing a text markup language my
> > requirements were:
> >
> >         1) Human readable
> >         2) Well documented
> >         3) Easy to translate to other data formats
> >         4) Well supported by tools and libraries
> >
> > After looking at a number of markup options, YAML stood out as the
> > one that meets all of these requirements. The YAML syntax is adopted
> > in many projects specifically because of the high level of readability.
> > A comprehensive spec is at http://www.yaml.org/spec/1.2/spec.html.
> > There's a number of tools to convert between YAML and other popular
> > data formats such as JSON and XML. XML was cited by Behan Webster
> > during the microconference as an important data format as the type
> > of developers that may produce comprehensive DTS Binding validation
> > tools will want to use XML. Every major scripting language has a
> > high level binding to the low level libyaml C library to facilitate
> > handling of YAML data files.
> >
> > One caveat with YAML is it does not tolerate tabs. Yes, I said it.
> > No tabs! This can be managed with proper editor modes and also with
> > helper scripts to strip tabs to aid in people passing planned
> > checkpatch.pl checks that would run YAML DT Binding specific tag
> > validators for new bindings.
> 
> This will be an endless source of pain.  Based on your examples,
> it looks like we only need a very small subset of YAML.  I'd be willing
> to write a parser for the language, but we might lose the ability to
> do conversions to other formats.  It depends on how comprehensive
> and automatic the helper scripts to strip tabs would be.

We need something to validate that a binding meets our binding
schema so it shouldn't a huge problem to have it warn and by option,
reformat the binding with tabs stripped. Having an appropriate mode
available for vim/emacs would go a long way to helping here too.

> > The scope of the initial YAML DT Binding format was specifically
> > limited to supporting *only* the content we have in bindings today.
> > The idea here is to propose and agree on something that will take
> > us just a few steps in the right direction. If we move *all* current
> > binding content to a machine parseable format, additional features
> > can be added with more automation and scripting. As it stands today,
> > because of the inconsistency of the wording of the files, we can't
> > add a lot of new features to the content until we convert what we
> > have today into a standard format.
> >
> > With that said, it should be noted that some new features such as
> > "type" tags to indicate cell types could be added to support
> > additional DTS validation beyond what the current content supports.
> > Another possibility is adding "range" type information to validate
> > the legal values for a cell.
> 
> I would recommend we add type and something for value constraints.
> Even if we don't add populate those immediately, I think thinking
> through some of the possible uses will be helpful to avoid doing
> something dumb.
> 'choices' or 'enums' seem frequently used.  I see lots of "must be one
> of..." in the binding docs I read.

Yes, so definitely after initial feedback, we're going to define some kind
of type and constraint tags. Whether we use C-like expressions for
constraints or something else is still up for debate.

And as you point out (and Rob also did) we need some type of AND and
OR expressions for constraint enums which also include the regular
case of defining how compatibles must be handled (e.g. one of the
specific strings AND the generic string).

> 
> Another important thing to document would be a 'superclass' or
> 'inherit-from' attribute, for things that are of a specific type (i2c,
> regulator, clock, etc.)

There's an attempt at that with the reference tag. Based on this and
other comments we'll expand that usage to inherit all the properties
from a generic binding (a 'superclass' as used above) rather than
repeating those within each peripheral binding document.

> > This series is broken up into three major parts:
> >
> > 1) The documentation defining the YAML DT binding format
> > 2) A skeleton device binding example illustrating use of this format
> > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
> >    ti-phy.txt
> >
> > As a proof of concept of what can be done with a proper machine
> > readable DT binding source file, there's a simple markdown document
> > generator at https://github.com/konsulko/dtgendoc. Also, to see
> > actual output from the generator, the generated markdown from those
> > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki
> >
> > There's a lot of other possibilities for validation tools using
> > only the data we have today in the bindings. In addition, Frank
> > Rowand covered some DT debug techniques that would benefit from
> > the binding documentation being 100% reliably searchable.
> >
> > I found it useful to see a side-by-side view of a converted doc
> > versus the original content, so here's a screenshot of eeprom.txt
> > vs. eeprom.yaml:
> > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml
> >
> > When we decide on a text markup format that is acceptable, then the
> > next step is to convert all the bindings. That process would start
> > with the complete set of generic bindings as they will be referenced
> > by the actual device bindings.
> 
> Although there are a lot of files to convert, I think the job is manageable,
> and can be done with conversion sprints at conferences and the like.
> You don't have to be a kernel 'C' coder to do a conversion, so I think
> a lot of non-traditional contributers can help out.  I'm already
> thinking about what could be done at an Embedded Linux Conference.

I think this would be great and completely agree that it allows a lot of
people to get involved. If we can get to a tolerable schema in the next
month then we should be ready for some f2f conversion hacking at ELCE.

> One thing that would help would be a conversion guide.  It would have
> things like: "if you see this type of language, put that in the
> description, some examples are ..."

Ok, that's very good. I'll add to the TODO list that we need to have
this is our guidelines document. I think most of this will fall out
initially in agreeing on our format/schema and then whatever we
miss can be added as we do the large scale conversion of bindings.

> I'll have more comments on individual patches.

Thanks for reviewing!

-Matt

Re: [RFC PATCH 5/5] Documentation: dt-bindings: phy: add YAML TI PHY binding

[Matt Porter]

On Tue, Sep 01, 2015 at 10:42:25AM -0700, Tim Bird wrote:
> On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Convert the ti-phy.txt binding to standard YAML DT binding
> > format. This binding references the YAML generic PHY binding.
> >
> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> > ---
> >  Documentation/devicetree/bindings/phy/ti-phy.yaml | 166 ++++++++++++++++++++++
> >  1 file changed, 166 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/phy/ti-phy.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/phy/ti-phy.yaml b/Documentation/devicetree/bindings/phy/ti-phy.yaml
> > new file mode 100644
> > index 0000000..ec93501
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/phy/ti-phy.yaml
> > @@ -0,0 +1,166 @@
> > +%YAML 1.2
> > +---
> > +id: omap-control-phy
> > +
> > +title: OMAP Control PHY
> > +
> > +compatible:
> > +  - name: "ti,control-phy-otghs"
> > +    description: if it has otghs_control mailbox register as on OMAP4.
> > +  - name: "ti,control-phy-usb2"
> > +    description: if it has Power down bit in control_dev_conf register
> > +                 e.g. USB2_PHY on OMAP5.
> > +  - name: "ti,control-phy-pipe3"
> > +    description: if it has DPLL and individual Rx & Tx power control
> > +                 e.g. USB3 PHY and SATA PHY on OMAP5.
> > +  - name: "ti,control-phy-pcie"
> > +    description: for pcie to support external clock for pcie and to
> > +                 set PCS delay value. e.g. PCIE PHY in DRA7x
> > +  - name: "ti,control-phy-usb2-dra7"
> > +    description: if it has power down register like USB2 PHY on DRA7
> > +                 DRA7 platform.
> > +  - name: "ti,control-phy-usb2-am437"
> > +    description: if it has power down register like USB2 PHY on AM437
> > +                 platform.
> 
> I'm confused.  If these descriptions are multi-line, don't they need the '>'
> syntax?

Consider that a bug ;) Yes, they should have '>'. Since I'm just using
the default core yaml schema validator this is perfectly legal when I
run it through yaml. It would mess up formating when output to a
document.


> Also, I'm still confused by the "name:" field in this section.  These are
> possible *values* for the compatible property, not the name of the field,
> as all other property descriptions are.

Ok, so that's a legacy artifact of several incarnations of these tags
before I posted initially. I agree, it should be "string" or "value",
probably the latter as you suggest.

-Matt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Mon, Aug 31, 2015 at 09:05:00AM -0500, Rob Herring wrote:
> On Fri, Aug 28, 2015 at 12:13 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > On Fri, Aug 28, 2015 at 09:26:17AM -0500, Rob Herring wrote:
> >> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >> > During the Device Tree microconference at Linux Plumbers 2015, we had
> >> > a short discussion about how to improve DT Binding Documentation. A
> 
> [...]
> 
> >> > One caveat with YAML is it does not tolerate tabs. Yes, I said it.
> >> > No tabs! This can be managed with proper editor modes and also with
> >> > helper scripts to strip tabs to aid in people passing planned
> >> > checkpatch.pl checks that would run YAML DT Binding specific tag
> >> > validators for new bindings.
> >>
> >> What do parsers do with tabs? Throw an error?
> >
> > Yes, they throw an error. Keep in mind that most of what I used to start
> > are general purpose conversion tools on top of a particular scripting
> > languages's high level binding to libyaml. The error output leaves a
> > bit to be desired for our use case. In any case, when I was developing
> > the skeleton.yaml I used the yaml script from
> > https://github.com/ryo/yamltools to catch all these syntax errors I
> > was inserting..like tabs. the PyYaml binding being used in my PoC
> > dtgendoc does the same thing but I don't gracefully handle those
> > errors like we could.
> >
> >> Beyond tabs, how do we check files can be parsed both generically and
> >> for any binding specific requirements. We now need a schema for
> >> checking the schema. We need some equivalent to compile testing.
> >
> > Right. So, I think what you are touching on is something I should
> > have expanded on in the TODO list. Basically, we need a scripted
> > tool that we run from checkpatch.pl that 1) reads the .yaml and
> > validates the YAML itself (that comes for free in the high level
> > parsers) reporting errors in a sensical manner 2) validates our
> > DT binding specific tags
> 
> We all know that no one runs checkpatch.pl. ;) I really want the basic
> checking of the doc files to run from make (and run by 0-day). The
> tool dependency could be an issue though. However, DocBook builds from
> make and I don't think many people check that regularly. Then there is
> using the binding docs to check dts files. That should probably be
> part of the dtb building.

Ok, makes sense. We can do that. I get this uneasy feeling about what
happens to all of this *when* bindings and dts files move out of the
kernel.

> > Now, I would caution about trying to do too much on Day 1 or we
> > could end up back at the "never doing anything" stage.
> 
> Certainly, but I would like to have a plan for what Day 2 and 3 look like.

Sure.

> 
> > It would
> > be an improvement to simply check that the basic tags exist as
> > shown in the [R] or [O] fields in the documentation. One thing
> > I should point out is that I carefully avoided marking some tags
> > as [R] where existing bindings don't have them...even if logically,
> > a description should be required on every binding. The idea here
> > is to avoid updating content at the same time that we are updating
> > the format. Rather, I think it would be better to get the base
> > format updated, then come back with a janitorial team and add
> > descriptions (since now we can generate a worklist of those
> > bindings missing a top-level description) and systematically
> > fix those and review with the appropriate maintainers.
> 
> Yes. Any checking would be a great improvement.
> 
> [...]
> 
> >> > When we decide on a text markup format that is acceptable, then the
> >> > next step is to convert all the bindings. That process would start
> >> > with the complete set of generic bindings as they will be referenced
> >> > by the actual device bindings.
> >>
> >> You are going to do that for everyone, right? ;)
> >
> > Let's just say that I'm banking on others helping here once we have
> > a format agreed upon. If we can hold the binding doc schema definition
> > initially to just define tags for content that already exists in our
> > textual binding docs, the effort for conversion is tolerable. To give
> > an example, that phy-bindings.txt, it took 15 minutes to convert and
> > and pass through the yaml parser and dtgendoc. The reason is that it's
> > pure reformatting work. It doesn't take any special knowledge of the
> > hardware and it doesn't involve reviewing dts files to extra
> > additional information. Some of the annoyances can be streamlined
> > like tab stripping and handling the two space indentation to make
> > this process faster. One of my next things is to get a simple tool
> > going that reports problems with conversions, essentially what I
> > said was needed to integrate with checkpatch, so this process of
> > conversion is even faster. Trivial peripheral bindings like eeprom.txt
> > can be done in 5 minutes or so right now.
> 
> What if instead of using the docs as a starting point, we use dts
> files as a starting point? They give us something parse-able that we
> can do some automation with.

That's doable, and we know that the dts files actually work.

> Additionaly, perhaps we can do a mass conversion of all the doc files
> from txt to yaml where all the current text is converted to comments
> and we fill in boilerplate and whatever we can convert with some
> automation? The downside here would be it will be hard to tell which
> conversions are complete.

Yes, basically we get a template and just have some human editing of
the text into proper tags to do.

 
> Putting both together, we could then fairly easily for example extract
> compatible strings from dts files, lookup which doc file they are in
> and fill in the compatible string info.

Ok.

> 
> > If we decide we must have tags like "type:" in the initial binding
> > doc schema definition *and* we must add that content in each
> > conversion, then this becomes more time consuming to validate that
> > information against working dts files. IMHO, we'd be better off
> > to get the base format straight, addressing missing pieces like
> > all the compatible permutations, and convert them all with
> > just that content. After that, we come back and add new content
> > features like type: tagging. I'm trying to find a reasonable
> > place to do this incrementally since the volume of bindings to
> > convert is enormous.
> 
> I wouldn't say we have to add it, but we need to maintain type info.

You already changed my opinion here. I think from generating the
template via working dts files, also have type info will not be
challenging. The other factor is we've already agreed that we need
to inhereit generic bindings so the volume of properties that have
to have types defined is lower as most properties will be inherited
from generic bindings.

> Are you proposing we can actually validate dts files on day 1? That I
> would expect to come later.

No, I expect that to come later. I was speaking only of "human
validation" of types while creating docs but I think now it's not
a big burden.

-Matt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Tue, Sep 01, 2015 at 10:14:06AM -0700, Tim Bird wrote:
> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> 
> > Now, I would caution about trying to do too much on Day 1 or we
> > could end up back at the "never doing anything" stage.
> 
> I strongly agree with that (despite my having just sent a bunch of comments
> on the binding format document).
> 
> > It would
> > be an improvement to simply check that the basic tags exist as
> > shown in the [R] or [O] fields in the documentation. One thing
> > I should point out is that I carefully avoided marking some tags
> > as [R] where existing bindings don't have them...even if logically,
> > a description should be required on every binding. The idea here
> > is to avoid updating content at the same time that we are updating
> > the format.
> 
> I strongly agree with this as well.  I think that the goal for the first
> pass should be "no lost text".  That is, if it doesn't convert into something
> in the binding format, it should still be preserved somewhere, for evaluation
> and formatting on a subsequent pass.
> 
> > Rather, I think it would be better to get the base
> > format updated, then come back with a janitorial team and add
> > descriptions (since now we can generate a worklist of those
> > bindings missing a top-level description) and systematically
> > fix those and review with the appropriate maintainers.
> 
> Agreed.  It will be very useful, once we get our first pass done, to
> be able to find: 1) documents not converted yet, 2) property definitions
> that are missing certain fields, 3) things that could still be formatted,
> from text that is remaining in the description strings or other free-form text.
> This should be easy to script.

Exactly, if we were to avoid populating type: and constraints: flags
then it is very easy to identify those bindings not yet converted to
using those tags.

> >> An example such as checking that compatible strings are documented as
> >> checkpatch.pl does would be nice. Roughly, that would be just list all
> >> compatible values.
> >
> > Ok, so my comments above were strictly about a validator for the
> > binding doc submission itself. I can add an example based on your
> > checkpatch.pl to adapt it to the .yaml compatible tags.
> >
> >> > The scope of the initial YAML DT Binding format was specifically
> >> > limited to supporting *only* the content we have in bindings today.
> >> > The idea here is to propose and agree on something that will take
> >> > us just a few steps in the right direction. If we move *all* current
> >> > binding content to a machine parseable format, additional features
> >> > can be added with more automation and scripting. As it stands today,
> >> > because of the inconsistency of the wording of the files, we can't
> >> > add a lot of new features to the content until we convert what we
> >> > have today into a standard format.
> >> >
> >> > With that said, it should be noted that some new features such as
> >> > "type" tags to indicate cell types could be added to support
> >> > additional DTS validation beyond what the current content supports.
> >> > Another possibility is adding "range" type information to validate
> >> > the legal values for a cell.
> >> >
> >> > This series is broken up into three major parts:
> >> >
> >> > 1) The documentation defining the YAML DT binding format
> >> > 2) A skeleton device binding example illustrating use of this format
> >> > 3) Some real binding conversions (eeprom.txt, phy-bindings.txt, and
> >> >    ti-phy.txt
> >> >
> >> > As a proof of concept of what can be done with a proper machine
> >> > readable DT binding source file, there's a simple markdown document
> >> > generator at https://github.com/konsulko/dtgendoc. Also, to see
> >> > actual output from the generator, the generated markdown from those
> >> > bindings is viewable at https://github.com/konsulko/dtgendoc/wiki
> >>
> >> Nice.
> >>
> >> > There's a lot of other possibilities for validation tools using
> >> > only the data we have today in the bindings. In addition, Frank
> >> > Rowand covered some DT debug techniques that would benefit from
> >> > the binding documentation being 100% reliably searchable.
> >> >
> >> > I found it useful to see a side-by-side view of a converted doc
> >> > versus the original content, so here's a screenshot of eeprom.txt
> >> > vs. eeprom.yaml:
> >> > https://github.com/konsulko/dtgendoc/wiki#eepromtxt-vs-eepromyaml
> >> >
> >> > When we decide on a text markup format that is acceptable, then the
> >> > next step is to convert all the bindings. That process would start
> >> > with the complete set of generic bindings as they will be referenced
> >> > by the actual device bindings.
> >>
> >> You are going to do that for everyone, right? ;)
> >
> > Let's just say that I'm banking on others helping here once we have
> > a format agreed upon. If we can hold the binding doc schema definition
> > initially to just define tags for content that already exists in our
> > textual binding docs, the effort for conversion is tolerable. To give
> > an example, that phy-bindings.txt, it took 15 minutes to convert and
> > and pass through the yaml parser and dtgendoc. The reason is that it's
> > pure reformatting work. It doesn't take any special knowledge of the
> > hardware and it doesn't involve reviewing dts files to extra
> > additional information. Some of the annoyances can be streamlined
> > like tab stripping and handling the two space indentation to make
> > this process faster. One of my next things is to get a simple tool
> > going that reports problems with conversions, essentially what I
> > said was needed to integrate with checkpatch, so this process of
> > conversion is even faster. Trivial peripheral bindings like eeprom.txt
> > can be done in 5 minutes or so right now.
> >
> > If we decide we must have tags like "type:" in the initial binding
> > doc schema definition *and* we must add that content in each
> > conversion, then this becomes more time consuming to validate that
> > information against working dts files. IMHO, we'd be better off
> > to get the base format straight, addressing missing pieces like
> > all the compatible permutations, and convert them all with
> > just that content. After that, we come back and add new content
> > features like type: tagging. I'm trying to find a reasonable
> > place to do this incrementally since the volume of bindings to
> > convert is enormous.
> >
> > But to answer your question, if we get a format I'll do
> > conversions and hope I'm not alone.
> 
> I'm sure others will help out.  I will, and I'm pretty sure we can get
> some conversion sprints set up at conferences (I know I can set aside
> some time or resources at ELC in the spring - it might be too late for
> ELCE in October to set up a scheduled block of time, but we can start
> getting the word out.)  As I said in my other e-mail, one doesn't have
> to be a kernel coder to do this, and the conversions should be pretty
> straight-forward.

Agreed, a f2f session would probably kick out a number of things
we hadn't yet considered as well. We'll see if we're actually ready
with a yaml schema that people are happy with by then though...

-Matt

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Tue, Sep 01, 2015 at 10:35:30AM -0700, Frank Rowand wrote:
> On 9/1/2015 10:14 AM, Tim Bird wrote:
> > On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> 
> < snip >
> 
> >>
> >> But to answer your question, if we get a format I'll do
> >> conversions and hope I'm not alone.
> > 
> > I'm sure others will help out.  I will, and I'm pretty sure we can get
> > some conversion sprints set up at conferences (I know I can set aside
> > some time or resources at ELC in the spring - it might be too late for
> > ELCE in October to set up a scheduled block of time, but we can start
> > getting the word out.)  As I said in my other e-mail, one doesn't have
> > to be a kernel coder to do this, and the conversions should be pretty
> > straight-forward.
> >  -- Tim
> > 
> 
> A conversion sprint at ELCE sounds like a good idea if we can find a
> good time to schedule it.  I can help, so there will be at least two
> of us who can help people as they encounter issues.

I'll be there as well.

-Matt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Tue, Sep 01, 2015 at 11:03:50AM -0700, Tim Bird wrote:
> On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand <frowand.list-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > On 9/1/2015 10:14 AM, Tim Bird wrote:
> >> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >
> > < snip >
> >
> >>>
> >>> But to answer your question, if we get a format I'll do
> >>> conversions and hope I'm not alone.
> >>
> >> I'm sure others will help out.  I will, and I'm pretty sure we can get
> >> some conversion sprints set up at conferences (I know I can set aside
> >> some time or resources at ELC in the spring - it might be too late for
> >> ELCE in October to set up a scheduled block of time, but we can start
> >> getting the word out.)  As I said in my other e-mail, one doesn't have
> >> to be a kernel coder to do this, and the conversions should be pretty
> >> straight-forward.
> >>  -- Tim
> >>
> >
> > A conversion sprint at ELCE sounds like a good idea if we can find a
> > good time to schedule it.  I can help, so there will be at least two
> > of us who can help people as they encounter issues.
> 
> Even if we don't find a block of time, we can do something like
> announce a "contest", ask people to do something in their spare time,
> and find some way to get them a raffle ticket if they submit a patch
> with a conversion.  Then hold an extra prize drawing during the
> closing session, with just those raffle tickets, and give someone a
> nice award for contributing.  Of course, there's always the adage that
> you should be careful what you measure and reward...  We don't want a
> flood of crappy conversions, with a time constraint on the review.
> I'll think some more about this.  An alternative would be to have a
> contest announced ahead of the event, with enough time for people to
> submit and get reviewed.
> 
> By the way - I presume the new docs will replace the existing ones,
> but I could imagine wanting to have them live side-by-side
> temporarily.  Any thoughts on this?  Will file name or location
> changes be allowed during the conversion?

Good question. As a starting point I simply changed the file extension
and they live in the same directory structure with the .txt versions.
I can't think of an advantage to moving locations or file names
initially. Certainly some file names could use improvement but
changing them introduces confusion.

-Matt

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Matt Porter]

On Tue, Sep 01, 2015 at 01:24:17PM -0500, Rob Herring wrote:
> On Tue, Sep 1, 2015 at 1:03 PM, Tim Bird <tbird20d-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand <frowand.list-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> On 9/1/2015 10:14 AM, Tim Bird wrote:
> >>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >>
> >> < snip >
> >>
> >>>>
> >>>> But to answer your question, if we get a format I'll do
> >>>> conversions and hope I'm not alone.
> >>>
> >>> I'm sure others will help out.  I will, and I'm pretty sure we can get
> >>> some conversion sprints set up at conferences (I know I can set aside
> >>> some time or resources at ELC in the spring - it might be too late for
> >>> ELCE in October to set up a scheduled block of time, but we can start
> >>> getting the word out.)  As I said in my other e-mail, one doesn't have
> >>> to be a kernel coder to do this, and the conversions should be pretty
> >>> straight-forward.
> >>>  -- Tim
> >>>
> >>
> >> A conversion sprint at ELCE sounds like a good idea if we can find a
> >> good time to schedule it.  I can help, so there will be at least two
> >> of us who can help people as they encounter issues.
> >
> > Even if we don't find a block of time, we can do something like
> > announce a "contest", ask people to do something in their spare time,
> > and find some way to get them a raffle ticket if they submit a patch
> > with a conversion.  Then hold an extra prize drawing during the
> > closing session, with just those raffle tickets, and give someone a
> > nice award for contributing.  Of course, there's always the adage that
> > you should be careful what you measure and reward...  We don't want a
> > flood of crappy conversions, with a time constraint on the review.
> > I'll think some more about this.  An alternative would be to have a
> > contest announced ahead of the event, with enough time for people to
> > submit and get reviewed.
> 
> Sounds like a review nightmare. That's another reason why as much
> automated conversion we can do, the better.
 
I don't want to discount the value of interested people getting together
f2f to review these and potentially clean them up for submission. That
depends on what we thinking is the minimal "in progress" conversion
that can be place upstream. i.e. is it simply compatible strings
autoconverted to tags and the entire current document in comments?

> > By the way - I presume the new docs will replace the existing ones,
> > but I could imagine wanting to have them live side-by-side
> > temporarily.  Any thoughts on this?  Will file name or location
> > changes be allowed during the conversion?
> 
> I proposed some ideas earlier in the thread. Either we can have both
> side by side or do a mass conversion to YAML making the existing doc a
> comment (add # prefix).
> 

Were you thinking that this automated conversion would be sufficient
for an initial commit? I'm not sure if I misunderstood in your separate
comments and was looking at this as something that would be hand edited
to move the existing doc (# prefix) into description tags where
appropriate.

> Any renames/moving should be separate (there's some clean-up I'd like
> to there as well). Exact rules depend on the approach, but we need to
> be able to tell which bindings conversions are not started, in
> progress, or complete.

If we add .yaml in place we can identify what's in progress by the fact
that a .yaml exists with the same filename and then based on which
tags have been populated (such as type: and constraints: not yet
populated) then we know the state.

-Matt

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Matt Porter]

On Sun, Aug 30, 2015 at 03:18:19PM -0700, Frank Rowand wrote:
> On 8/28/2015 11:55 AM, Rob Herring wrote:
> > On Fri, Aug 28, 2015 at 12:49 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >> On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
> >>> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >>>> Add a skeleton DT binding document that serves as the canonical
> >>>> example for implementing YAML-based DT bindings documentation.
> >>>> The skeleton binding illustrates use of all fields and variations
> >>>> described in the dt-binding-format.txt documentation.
> >>>>
> >>>> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> >>>> ---
> >>>>  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
> >>>>  1 file changed, 98 insertions(+)
> >>>>  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> >>>>
> >>>> diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
> >>>> new file mode 100644
> >>>> index 0000000..175965f
> >>>> --- /dev/null
> >>>> +++ b/Documentation/devicetree/bindings/skeleton.yaml
> >>>> @@ -0,0 +1,98 @@
> >>>> +%YAML 1.2
> >>>> +---
> >>>> +id: skel-device
> >>>> +
> >>>> +title: Skeleton Device
> >>>> +
> >>>> +maintainer:
> >>>> +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >>>
> >>> We'd want to tie this into get_maintainers.pl obviously.
> >>
> >> Right, I broke my rule of "no new tags for content we don't already
> >> have" by adding this. It stems out of the discussion at LPC where
> >> Mark suggested we could avoid the core bindings being moved by adding
> >> maintainer info into the binding itself.
> > 
> > If you stated that rule, I missed it...
> > 
> >> This is an area where more docs are needed. Ideally, on the first pass
> >> conversion we would not attempt to populate these. I had considered
> >> that we *could* only add maintainer: tags on core bindings to and
> >> adjust get_maintainers.pl to use that info to override the standard
> >> directory-based info if it exists. I don't think it's necessary to add
> >> a specific maintainer for all of the peripheral bindings if the default
> >> is the subsystem maintainer and the dt list. We could at least start
> >> directing core binding discussion to the -spec list which is where I
> >> think you'd like it.
> 
> I think that adding the maintainer: tags for core bindings in the conversion
> process (and optionally add the tag for other bindings in the person
> converting has the driver and subsystem knowledge to intelligently
> do so -- not just a blind list from get_maintainer) would be a valid
> exception to your desire to not add new information on the conversion
> step.  As much as other uses of information from bindings is near and
> dear to me, I think that throttling the binding review fire hose is
> the highest priority use of additional information.

Ok, and the other suggestion is to just populate it with the original
author so that's straightforward enough to do.

> 
> 
> > Yes, certainly this can solve my problem around handling core bindings.
> > 
> > Some subsystem maintainers require a driver have a maintainer. We
> > could certainly enforce that too. We could probably automatically
> > populate this with the original author as a starting point. OTOH,
> > bindings should not change frequently.
> > 
> >>
> >>>> +
> >>>> +description: >
> >>>> +  The Skeleton Device binding represents the SK11 device produced by
> >>>> +  the Skeleton Corporation. The binding can also support compatible
> >>>> +  clones made by second source vendors.
> >>>> +
> >>>> +compatible:
> >>>> +  - name: "skel,sk11"
> >>>> +  - name: "faux,fx11"
> >>>
> >>> Is this an OR or AND? We need both.
> >>
> >> True, this only covers the OR case atm.
> >>
> >>> The complicated case is "one of {specific names} followed by {generic name}."
> >>
> >> I need to rethink these. I do have deprecated: tag for that case and
> >> possibly "name:" gets split to "generic:" and "specific:" and we can
> >> then do the right thing.
> >>
> >> For the above I would have:
> >>
> >> compatible:
> >>   - specific: "skel,sk11"
> >>   - specific: "faux,fx11"
> > 
> > This can still be an AND or OR relationship.
> > 
> >>
> >> and something like the Allwinner simple framebuffer would be:
> >>
> >> compatible:
> >>   - generic: "simple-framebuffer"
> >>   - specific: "allwinner,simple-framebuffer"
> >>
> >> where our validator would insist on seeing one of the specific: tags
> >> along with the generic: tag. A lot of bindings, given current doc
> >> patterns would just have two tags like above.
> > 
> > We could follow Pantelis' suggestion and do C style logic expressions:
> > 
> > name: ("chip1-ipblock" || "chip2-ipblock") && "generic-ipblock"
> > 
> > 
> >>
> >>>> +    description: A clone of the original sk11 device
> >>>> +
> >>>> +required:
> >>>> +  - name: "reg"
> >>>
> >>> We definitely need type info from the start.
> >>
> >> Are you sure? Ugh.
> >>
> >> My big contention here is that we don't carry that content in the
> >> docs today so we shouldn't try to add it in the initial conversion.
> > 
> > What? We have that all over the place. Boolean properties, phandles, etc.
> 
> Piling on...  If the type information is obvious in the current binding
> doc, then it should be captured in the conversion and not lost.  If the
> information is not currently present then it should not be required in
> the conversion step.

Ok.

> Which means that we need to define the how to specify the type information.

Agreed, I'm adding that to the format in v2.

> >> I think you are right in that we should have it documented in the
> >> schema but I'm concerned that we make the starting conversion
> >> effort too large by adding type info to all converted docs.
> > 
> > How we do the conversion certainly needs to be worked out. However,
> > first we need some sort of agreement that yes this looks promising
> > before we spend too much time on it. We need to wait for all the YAML
> > haters to come out of hiding. :) Once the direction is settled, then
> > we can iron out the details of the structure. Finally, then we can
> > determine how to do the conversion. Just because we define how type
> > information looks doesn't mean it has to be there day 1.
> > 
> >> I think we gain a lot even without at the start, but I understand
> >> why we need it and how it will help reduce the review firehose
> >> with the associated validation tools.
> >>
> >>>
> >>>> +    description: chip select address of skeleton device
> >>>> +    reference: spi-slave
> >>>
> >>> I would like to not have to list properties if the inherited binding
> >>> lists it. The problem is we need to say how many cells and the order
> >>> (not a problem here, but for mmio devices).
> >>
> >> Yeah, make sense.
> >>
> >>> Perhaps we can list the reference at the top level for the node
> >>> instead of for every property.
> >>
> >> That's a good point. I was wondering if per prop references would
> >> get unwieldy once we actually add them into all the converted docs.
> >> There's a huge number of existing docs without proper references.
> > 
> > Yes, in many cases they are implied. It should be simple enough to
> > generate the list though whether they point to the doc file or just
> > list the properties.
> > 
> >>
> >> Ok, I'll take a look at collecting references per group of properties.
> >>
> >>>
> >>>> +  - name: "spi-max-frequency"
> >>>> +    description: >
> >>>> +      Maximum SPI clocking speed of skeleton device in Hz, must be
> >>>> +      1000000
> >>>> +    reference: spi-slave
> >>>
> >>> Rather than listing the property and having constraint in description,
> >>> perhaps we could add constraints like this:
> >>>
> >>> - spi-max-frequency-range: 1000000 1000000
> >>>
> >>> Or groups of constraints:
> >>>
> >>> - spi-max-frequency-constraints:
> >>>   range: 1000000 1000000
> >>>   some-other-constraint: <value>
> >>
> >> I was hoping to avoid this to start due to my argument for keeping it
> >> simple for the first pass at conversion. However, the latter looks
> >> flexible enough. We have cases with enumerated values as well to handle
> >> the require some thought.
> > 
> > We can always add constraints as a second step, but I'd like to define
> > the structure at least.
> 
> Agreed, this is something that I would like to see defined, just to make
> sure that we can do it in a reasonable way with the proposed tool.

Will do, adding that in v2.

-Matt
--
To unsubscribe from this list: send the line "unsubscribe devicetree" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Matt Porter]

On Tue, Sep 01, 2015 at 10:35:44AM -0700, Tim Bird wrote:
> On Fri, Aug 28, 2015 at 7:53 AM, Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> ...
> >> +example:
> >> +  - dts: |
> >> +      sk11@0 {
> >> +              compatible = "skel,sk11";
> >> +              reg = <0>;
> >> +              spi-max-frequency = <1000000>;
> >> +              spi-cs-high;
> >> +      };
> >
> > At least in this example, we could generate it. Examples are nice, but
> > we have dts files full of examples already. I get a fair number of
> > "fix the example" patches, so maybe we should eliminate the simple
> > ones.
> 
> I would hesitate to eliminate examples.  I've been saved by them on
> a few occasions, when the dts files only had one or two instances
> of a type of binding, somewhat different from each other, and the example helped
> break the tie.  If there's something wrong with the example, it's a sign
> of an out-of-date binding doc, just as much as if the text were wrong.
> It ought to be possible to validate the example versus the binding doc
> (as Pantelis says), so ultimately we should be able to catch errors here
> as well.

I think part of the suggestion is that we can simply generate the common
examples from the tags (as part of formatted doc generation) rather than it
being a verbatim paste from a dts file. This has the benefit of not
getting that verbatim text of of sync with a dts (because we'll
eventually validating the dts files versus the binding doc). Does that
still achieve what you want? Keep in mind that we aren't talking about
taking away the ability to put an explicit verbatim example, but rather
making it an exception case to demonstrate some variants where an author
wants to point out different common use cases.

-Matt

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Matt Porter]

On Wed, Sep 02, 2015 at 10:45:13AM +0200, Nicolas Ferre wrote:
> Le 01/09/2015 19:35, Tim Bird a écrit :
> > On Fri, Aug 28, 2015 at 7:53 AM, Rob Herring <robherring2-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/HQFizaE/u3fw@public.gmane.orgm> wrote:
> > ...
> >>> +example:
> >>> +  - dts: |
> >>> +      sk11@0 {
> >>> +              compatible = "skel,sk11";
> >>> +              reg = <0>;
> >>> +              spi-max-frequency = <1000000>;
> >>> +              spi-cs-high;
> >>> +      };
> >>
> >> At least in this example, we could generate it. Examples are nice, but
> >> we have dts files full of examples already. I get a fair number of
> >> "fix the example" patches, so maybe we should eliminate the simple
> >> ones.
> > 
> > I would hesitate to eliminate examples.  I've been saved by them on
> > a few occasions, when the dts files only had one or two instances
> > of a type of binding, somewhat different from each other, and the example helped
> > break the tie.  If there's something wrong with the example, it's a sign
> > of an out-of-date binding doc, just as much as if the text were wrong.
> > It ought to be possible to validate the example versus the binding doc
> > (as Pantelis says), so ultimately we should be able to catch errors here
> > as well.
> 
> I back Tim's advice.
> Example are so important that I regularly find myself thinking "ah-ha,
> okay that's supposed to work like that" after having read the binding
> documentation.

See my comments on Tim's post. In summary, we'd retain the complex type
examples but allow the trivial ones to not bother with it in explicit
source. OTOH, this would have to be a final change in the conversion
process, most likely, because we need all the type: and constraints:
tags present in order to generate a working example.

Of course, that's the same info necessary to validate a dts as well.

Perhaps it's best to look at this as the automated conversion process
that Rob suggested. That is, we'll have the current text (including
all examples) brought into the .yaml file as comments. So we aren't
eliminating examples. Later, if those examples are too trivial to
bother capturing as above, we simply remove and one can dump an example
via a doc generator not unlike my markdown example (which can spit out
pasteable text).

-Matt

Re: [RFC PATCH 1/5] Documentation: dt-bindings: add documentation on new DT binding format

[Matt Porter]

On Sun, Aug 30, 2015 at 03:04:33PM -0700, Frank Rowand wrote:
> On 8/27/2015 10:23 PM, Matt Porter wrote:
> > Documentation explaining the syntax and format of the YAML-based DT binding
> > documentation.
> > 
> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> > ---
> >  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
> >  1 file changed, 106 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> > 
> > diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> > new file mode 100644
> > index 0000000..f9acc22
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> > @@ -0,0 +1,106 @@
> > +--------------------------
> > +Device Tree Binding Format
> > +--------------------------
> > +
> > +Background
> > +----------
> > +
> > +DT bindings historically were written as text in prose format which
> > +led to issues in usability of that source documentation. Some of
> > +these issues include the need to programmatically process binding
> > +source documentation to do DTS validation, perform mass updates to
> > +format/style, and to generate publishable documentation in HTML or
> > +PDF form.
> > +
> > +Overview
> > +--------
> > +
> > +The DT binding format is based on the YAML text markup language.
> > +Although there are many text markup options available, YAML
> > +fulfills all requirements considered for a DT binding source format
> > +which include:
> > +
> > +1) Must be human readable
> > +2) Must be easily translated to other data formats (XML, JSON, etc).
> > +3) Must have sufficient tools and libraries to enable developers to
> > +   build new tools for DT binding processing
> > +4) Must have a complete spec to refer to syntax
> > +
> > +YAML is documentated in the specification found at
> > +http://www.yaml.org/spec/1.2/spec.html
> > +
> > +The required YAML DT binding tag format and syntax are defined in
> > +the following sections.
> > +
> > +YAML DT Binding Syntax
> > +----------------------
> > +
> > +* Lines starting with "#" are comments and not part of the binding itself
> > +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> > +* "---" starts a binding document
> > +* "..." ends a binding document
> > +* Multiple binding documents may exist in a single file
> > +* Tabs are not permitted
> > +* Scope is denoted by indentation of two spaces
> > +* Key value pairs are denoted by "key: value"
> > +* Sequences are denoted by "- "
> > +* Scalar values may convert newlines to spaces and preserve blank
> > +  lines for long description formatting using ">"
> > +* Scalar values may escape all reserved characters and preserve
> > +  newlines by using "|" to denote literal style
> > +
> > +For additional information on YAML syntax, refer to the specification
> > +at http://www.yaml.org/spec/1.2/spec.html
> > +
> > +YAML DT Binding Format
> > +----------------------
> > +
> > +The following YAML types are supported in the DT binding format:
> > +
> > +* [R] id: unique identifier in property form (e.g. skel-device)
> > +
> > +* [R] title: title of the binding
> > +
> > +* [O] maintainer: sequence of maintainers
> > +      [R] name: name and email of maintainer or mailing list in RFC822
> > +                form.
> > +
> > +* [O] description: full description of the binding
> > +
> > +* [O] compatible: sequence of valid compatible descriptors
> > +      [R] name: the compatible string surrounded in double quotes
> > +      [O] deprecated: a deprecated compatible string surrounded in
> > +                      double quotes
> > +      [O] description: description of the compatible string
> > +
> > +* [O] required: sequence of required properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] optional: sequence of optional properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] deprecated: sequence of deprecated properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> 
> I commented in reply to patch 0 that we should think through how
> this structure will support adding new features as the next step
> after converting existing bindings.
> 
> The specific case I started thinking about was the distinction between
> required, optional, required-if, and optional-if.  A property might
> be required in all cases, optional in all cases, required in some
> specified cases, and/or optional in some specified cases.  So a
> property could be both "required-if" and "optional-if".  Or it
> might just be one of the two.
 
Ok, so this is similar to the issue Rob points out about AND and
ORs of compatible strings.

> Here is where my YAML ignorance jumps out.  I really do not know the
> best way to approach it.  And I do not know if the syntax I am using
> is even remotely correct - I am trying to extrapolate from the examples
> in the spec, without having read past the examples yet.  My naive knee
> jerk alternatives where (by example, because I can explain it more
> concisely that way instead of the way it should be described in this file):

I will confess that I'm not an expert either so we can learn together.
:)

> 
> ---- (1)
> 
> required:
>   - &P1
>     name: "prop1"
>     required-if: < some condition >
> 
> optional:
>   - *P1
>     optional-if: < some other condition>
>   - name: "prop2"
>     required: true
>   - name: "prop3"
>     optional: true
> 

Couple concerns here. First, we have to consider properties being
inherited from a generic binding rather than repeating them in
a peripheral binding. The conditions in which a generic property
may be required can be different in a child peripheral binding.
So you may have a property inhereit that is optional in the generic
binding, but is required in a particular child peripheral binding.

Now, having said that, if we make use of the anchor/alias syntax
as above, now we have to have unique anchor names across all generic
bindings. Further, we have this issue of yaml parsers not having
the ability now to directly include other files. Certainly we could
make a wrapper to pull in each referenced generic binding into a
single yaml document to be run through the parser.

The second thing is that in the example above we still have
the required-if tag populated within the optional section which
is not ideal.

> ---- (2)
> 
> property:
>   - name: "prop1"
>     required-if: < some condition >
>     optional-if: < some other condition >
>   - name: "prop2"
>     required: true
>   - name: "prop3"
>     optional: true

This looks better to me. If coupled with the C like constraint
expressions it should work well for most cases.

I'm thinking of the resource-names.txt common case such as:

property:
  - name: "reg"
    required-if: iface(mmio)
  - name "reg-names"
    required-if: prop(reg)

Keep in mind I'm not sure yet what we'll allow in these conditional
statements. I just believe that we'll want the same syntax for
constraints and property conditionals

> 
> 
> In a related question, I commented in my reply to patch 0 that some of
> the information about the compatible got lost in the conversion from
> the existing binding to the YAML format.  That led me to wonder why
> the compatible property should be at the YAML top level (same as
> "optional" and "required").  My (again naive, YAML amateur) thought
> is that the form should be something like:
> 
> [O] property:
>     [R] optional: | required: | required-if: | optional-if: | required-if: optional-if:
>     [O] deprecated: a deprecated compatible string surrounded in
>                     double quotes | ???
>     [R] name: name of the property surrounded in double quotes
>     [R] description: description of the property
>     [O] reference: optional reference to a binding id

TBH, compatible started getting treated separately from other properties
simply because of my initial take on readability in the yaml source.
There's technically no reason it can't be folded back in as yet another
property as above. However, we need a way to show ands and ors for
compatibles in a sane manner. Consider a device where the generic
compatible string from the original part in the family is gendev1001
and subsequent family members requiring tweaks exist.

property:
  - name: "foo,gendev1001"
    required: true
  - name: "foo,gendev1005"
    required-or: true
  - name: "foo,gendev1006"
    required-or: true
  - name: "foo,gendev1007"
    required-or: true

Not sure here, but that's one way to do it with the keys driving it
versus a conditional expression.

or

property:
  - name: "foo,gendev1001"
    required: true
  - name: "foo,gendev1005" || "foo,gendev1006" || "foo,gendev1007"
    required-if: !"foo,gendev1001"

which seems to look a lot better to me. A
not-yet-existing-smart-validator would check that a dts had:

compatible = "foo,gendev100[5|6|7]", "foo,gendev1001"

or simply

compatible = "foo,gendev1001"

> Apologies for bizarre format for "optional:", but for the purposes of this
> comment, hopefully my intent will be clear.
> 
> Then "compatible" becomes just another "property:", just as it is in the
> .dts, but any processing or analysis tool knows that the name "compatible"
> is used in a very specific way and has magical meaning for .dts.
> 
> Then if compatible is just another "property:" then the information that
> lost was in the conversion process would not be lost, it would appear
> in "description:".

Sounds good.

> 
> 
> > +
> > +* [R] example: sequence of examples:
> > +      [R] dts: DT source of example usage. The example text must use
> > +               literal style ("|") so that it retains indentation and
> > +               newlines.
> > +      [O] description: description of the example
> > +
> > +Note: [R] and [O] denote required and optional fields, respectively.
> > +
> > +Skeleton Binding
> > +----------------
> > +
> > +The skeleton.yaml binding found in the top of the DT binding tree
> > +is the canonical example of syntax and format to use when writing
> > +a DT binding document. It is maintained with the latest formatting
> > +conventions, making it the best starting point when writing a new DT
> > +binding.
> 
> It would be good to have a comment of YAML style.  Looking at the spec,
> there appear to be multiple ways to format content that is functionally
> equivalent.  For example:
> 
>    # a flow sequence
>    - [name        , hr, avg  ]
>    - [Mark McGwire, 65, 0.278]
>    - [Sammy Sosa  , 63, 0.288]
> 
> vs.
> 
>    # a flow mapping
>    Mark McGwire: {hr: 65, avg: 0.278}
>    Sammy Sosa: {
>        hr: 63,
>        avg: 0.288
>      }
> 
> This is my first quick reading of the YAML spec, so maybe a flow sequence and
> a flow mapping are not equivalent, but that was my naive impression.

The examples section in the intro is incomplete and glosses over some
stuff such that it's expected to get confused by this part. I got lost
there at first as well.

The quick summary is that YAML allows either Flow Style or Block Style
of describing nodes. The two example above use flow style as described
in Chapter 7 to show a sequence construct and separately a mapping
construct. I chose to use block style through the proposed YAML dt
binding format as it's my personal preference for human readability.
Compare, for example, the format of doing sequences in flow style
(section 7.4.1) versus block style (section 8.2.1) and you'll see that
I'm making heavy use of block style sequences throughout this format.

> I think that in the interest of human readability (requirement 1), that a
> specific formatting style should be specified.  That will make life easier
> for people who do not know YAML, but are (1) merely copying from existing binding
> docs to create new ones, or (2) trying to understand how different bindings
> differ.

Agreed. I'll explicitly mention that we use block style formatting now.
Also, David asked that we explicitly mention the YAML schema used (which
is the default core schema that also parsers have to implement).

-Matt

Re: [RFC PATCH 0/5] DT binding documents using text markup

[Rob Herring]

On Tue, Sep 8, 2015 at 9:36 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> On Tue, Sep 01, 2015 at 01:24:17PM -0500, Rob Herring wrote:
>> On Tue, Sep 1, 2015 at 1:03 PM, Tim Bird <tbird20d-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > On Tue, Sep 1, 2015 at 10:35 AM, Frank Rowand <frowand.list-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> >> On 9/1/2015 10:14 AM, Tim Bird wrote:
>> >>> On Fri, Aug 28, 2015 at 10:13 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>> >>
>> >> < snip >
>> >>
>> >>>>
>> >>>> But to answer your question, if we get a format I'll do
>> >>>> conversions and hope I'm not alone.
>> >>>
>> >>> I'm sure others will help out.  I will, and I'm pretty sure we can get
>> >>> some conversion sprints set up at conferences (I know I can set aside
>> >>> some time or resources at ELC in the spring - it might be too late for
>> >>> ELCE in October to set up a scheduled block of time, but we can start
>> >>> getting the word out.)  As I said in my other e-mail, one doesn't have
>> >>> to be a kernel coder to do this, and the conversions should be pretty
>> >>> straight-forward.
>> >>>  -- Tim
>> >>>
>> >>
>> >> A conversion sprint at ELCE sounds like a good idea if we can find a
>> >> good time to schedule it.  I can help, so there will be at least two
>> >> of us who can help people as they encounter issues.
>> >
>> > Even if we don't find a block of time, we can do something like
>> > announce a "contest", ask people to do something in their spare time,
>> > and find some way to get them a raffle ticket if they submit a patch
>> > with a conversion.  Then hold an extra prize drawing during the
>> > closing session, with just those raffle tickets, and give someone a
>> > nice award for contributing.  Of course, there's always the adage that
>> > you should be careful what you measure and reward...  We don't want a
>> > flood of crappy conversions, with a time constraint on the review.
>> > I'll think some more about this.  An alternative would be to have a
>> > contest announced ahead of the event, with enough time for people to
>> > submit and get reviewed.
>>
>> Sounds like a review nightmare. That's another reason why as much
>> automated conversion we can do, the better.
>
> I don't want to discount the value of interested people getting together
> f2f to review these and potentially clean them up for submission. That
> depends on what we thinking is the minimal "in progress" conversion
> that can be place upstream. i.e. is it simply compatible strings
> autoconverted to tags and the entire current document in comments?

That is sufficient for me (I reserve the right to change my mind).
Logistically, it needs to be a script that can be run before/during
the merge window and perhaps again after. I'd guess the long pole here
is how we validate the .yaml files.

>> > By the way - I presume the new docs will replace the existing ones,
>> > but I could imagine wanting to have them live side-by-side
>> > temporarily.  Any thoughts on this?  Will file name or location
>> > changes be allowed during the conversion?
>>
>> I proposed some ideas earlier in the thread. Either we can have both
>> side by side or do a mass conversion to YAML making the existing doc a
>> comment (add # prefix).
>>
>
> Were you thinking that this automated conversion would be sufficient
> for an initial commit? I'm not sure if I misunderstood in your separate
> comments and was looking at this as something that would be hand edited
> to move the existing doc (# prefix) into description tags where
> appropriate.

Yes, I think it is more important to have infrastructure in place to
enable others than how much is converted initially. Certainly any
changes based on existing docs will have to be manual, but we may be
able to do multiple passes of automated conversions. Sharing any
conversion scripts is important to enable others for that as well.

>> Any renames/moving should be separate (there's some clean-up I'd like
>> to there as well). Exact rules depend on the approach, but we need to
>> be able to tell which bindings conversions are not started, in
>> progress, or complete.
>
> If we add .yaml in place we can identify what's in progress by the fact
> that a .yaml exists with the same filename and then based on which
> tags have been populated (such as type: and constraints: not yet
> populated) then we know the state.

You mean keeping the .txt and .yaml until done? If we are copying in
the current doc, then I'd rather not do that. I don't think looking at
the tags will work as it would be hard to distinguish incomplete from
done. However, we could simply have an "in-progress" tag that is
removed when done (might as well take advantage of our new found
structured docs).

Rob

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4628 bytes --]

On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Add a skeleton DT binding document that serves as the canonical
> > example for implementing YAML-based DT bindings documentation.
> > The skeleton binding illustrates use of all fields and variations
> > described in the dt-binding-format.txt documentation.
> >
> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> > ---
> >  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
> >  1 file changed, 98 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
> > new file mode 100644
> > index 0000000..175965f
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/skeleton.yaml
> > @@ -0,0 +1,98 @@
> > +%YAML 1.2
> > +---
> > +id: skel-device
> > +
> > +title: Skeleton Device
> > +
> > +maintainer:
> > +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> 
> We'd want to tie this into get_maintainers.pl obviously.
> 
> > +
> > +description: >
> > +  The Skeleton Device binding represents the SK11 device produced by
> > +  the Skeleton Corporation. The binding can also support compatible
> > +  clones made by second source vendors.
> > +
> > +compatible:
> > +  - name: "skel,sk11"
> > +  - name: "faux,fx11"
> 
> Is this an OR or AND? We need both.

Hrm.  So, from the example below it's an OR.  But.. I think the
considerations here change when you separate out "tag" information as
I've suggested elsewhere.

The command AND case, I can think of is where you want both
"foo,new-model-device" and "foo,old-model-device".  I think the first
belongs in the "tag", but the second in the body of the requirements.

The OR case demonstrated here strikes me as very poor binding design -
but that doesn't mean we can totally avoid it, since we do have poor
bindings we want to convert.

Fwiw, I think the right way to encode "OR" at least in the "tag" is to
pick just the preferred form, then have additional bindings tagged on
the alternative forms and inheriting the main binding.  I don't know
if that's practical in the short term though.

> The complicated case is "one of {specific names} followed by {generic name}."
> 
> > +    description: A clone of the original sk11 device
> > +
> > +required:
> > +  - name: "reg"
> 
> We definitely need type info from the start.

It's interesting you should mention that here, because 'reg' is
actually a hard case for describint the type: because it's format is
determined as much by the parent bus binding as this node's binding.

I suspect it will be worth special-casing "reg" in order to make
common bindings more compact, but again, probably not in the first
pass.

> > +    description: chip select address of skeleton device
> > +    reference: spi-slave
> 
> I would like to not have to list properties if the inherited binding
> lists it. The problem is we need to say how many cells and the order
> (not a problem here, but for mmio devices).
> 
> Perhaps we can list the reference at the top level for the node
> instead of for every property.

Yeah, I think it would be worth having a top-level "inherits" field.

> > +  - name: "spi-max-frequency"
> > +    description: >
> > +      Maximum SPI clocking speed of skeleton device in Hz, must be
> > +      1000000
> > +    reference: spi-slave
> 
> Rather than listing the property and having constraint in description,
> perhaps we could add constraints like this:
> 
> - spi-max-frequency-range: 1000000 1000000
> 
> Or groups of constraints:
> 
> - spi-max-frequency-constraints:
>   range: 1000000 1000000
>   some-other-constraint: <value>

Not for the first pass I think.  What I would suggest is having
"description" for pure informational stuff, and "constraint" or maybe
just "extra" for normative constraints expressed in English.

The idea here is that over time we can add new ways of expressing
constraints.  In the meantime tools can use the "extra" field to
preseve any difficult text in the current versions and at least let
tools know that they won't be able to check everything.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Rob Herring]

On 09/10/2015 02:08 AM, David Gibson wrote:
> On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
>> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
>>> Add a skeleton DT binding document that serves as the canonical
>>> example for implementing YAML-based DT bindings documentation.
>>> The skeleton binding illustrates use of all fields and variations
>>> described in the dt-binding-format.txt documentation.
>>>
>>> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
>>> ---
>>>  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
>>>  1 file changed, 98 insertions(+)
>>>  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
>>>
>>> diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
>>> new file mode 100644
>>> index 0000000..175965f
>>> --- /dev/null
>>> +++ b/Documentation/devicetree/bindings/skeleton.yaml
>>> @@ -0,0 +1,98 @@
>>> +%YAML 1.2
>>> +---
>>> +id: skel-device
>>> +
>>> +title: Skeleton Device
>>> +
>>> +maintainer:
>>> +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
>>
>> We'd want to tie this into get_maintainers.pl obviously.
>>
>>> +
>>> +description: >
>>> +  The Skeleton Device binding represents the SK11 device produced by
>>> +  the Skeleton Corporation. The binding can also support compatible
>>> +  clones made by second source vendors.
>>> +
>>> +compatible:
>>> +  - name: "skel,sk11"
>>> +  - name: "faux,fx11"
>>
>> Is this an OR or AND? We need both.
> 
> Hrm.  So, from the example below it's an OR.  But.. I think the
> considerations here change when you separate out "tag" information as
> I've suggested elsewhere.
> 
> The command AND case, I can think of is where you want both
> "foo,new-model-device" and "foo,old-model-device".  I think the first
> belongs in the "tag", but the second in the body of the requirements.
> 
> The OR case demonstrated here strikes me as very poor binding design -
> but that doesn't mean we can totally avoid it, since we do have poor
> bindings we want to convert.

I don't think so. Often "old" is a singleton, and "new" is one of
several newer chips. It is also very common for licensed IP blocks. All
the vendors have different quirks and possibly different versions of the
IP. In the better cases, we can have a vendor compat for each vendor and
then a generic compat.


> Fwiw, I think the right way to encode "OR" at least in the "tag" is to
> pick just the preferred form, then have additional bindings tagged on
> the alternative forms and inheriting the main binding.  I don't know
> if that's practical in the short term though.
> 
>> The complicated case is "one of {specific names} followed by {generic name}."
>>
>>> +    description: A clone of the original sk11 device
>>> +
>>> +required:
>>> +  - name: "reg"
>>
>> We definitely need type info from the start.
> 
> It's interesting you should mention that here, because 'reg' is
> actually a hard case for describint the type: because it's format is
> determined as much by the parent bus binding as this node's binding.

And reg is well defined, so that's one I'm least worried about.

> I suspect it will be worth special-casing "reg" in order to make
> common bindings more compact, but again, probably not in the first
> pass.

Yes, I would fully expect reg to just be flagged as used and how many
addresses.

> 
>>> +    description: chip select address of skeleton device
>>> +    reference: spi-slave
>>
>> I would like to not have to list properties if the inherited binding
>> lists it. The problem is we need to say how many cells and the order
>> (not a problem here, but for mmio devices).
>>
>> Perhaps we can list the reference at the top level for the node
>> instead of for every property.
> 
> Yeah, I think it would be worth having a top-level "inherits" field.
> 
>>> +  - name: "spi-max-frequency"
>>> +    description: >
>>> +      Maximum SPI clocking speed of skeleton device in Hz, must be
>>> +      1000000
>>> +    reference: spi-slave
>>
>> Rather than listing the property and having constraint in description,
>> perhaps we could add constraints like this:
>>
>> - spi-max-frequency-range: 1000000 1000000
>>
>> Or groups of constraints:
>>
>> - spi-max-frequency-constraints:
>>   range: 1000000 1000000
>>   some-other-constraint: <value>
> 
> Not for the first pass I think.  What I would suggest is having
> "description" for pure informational stuff, and "constraint" or maybe
> just "extra" for normative constraints expressed in English.

At least for simple constraints, I don't see the point of manually
converting to something still not machine parse-able. We should be able
to figure out a syntax. Sure, there may be complex examples we just punt
on, but we can address most common cases. But if we can't define a
syntax, then lets do nothing until we do.

> The idea here is that over time we can add new ways of expressing
> constraints.  In the meantime tools can use the "extra" field to
> preseve any difficult text in the current versions and at least let
> tools know that they won't be able to check everything.

Just keeping the existing doc text will preserve things.

Rob

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 8073 bytes --]

On Thu, Sep 10, 2015 at 11:46:13PM -0500, Rob Herring wrote:
> On 09/10/2015 02:08 AM, David Gibson wrote:
> > On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
> >> On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> >>> Add a skeleton DT binding document that serves as the canonical
> >>> example for implementing YAML-based DT bindings documentation.
> >>> The skeleton binding illustrates use of all fields and variations
> >>> described in the dt-binding-format.txt documentation.
> >>>
> >>> Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> >>> ---
> >>>  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
> >>>  1 file changed, 98 insertions(+)
> >>>  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> >>>
> >>> diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
> >>> new file mode 100644
> >>> index 0000000..175965f
> >>> --- /dev/null
> >>> +++ b/Documentation/devicetree/bindings/skeleton.yaml
> >>> @@ -0,0 +1,98 @@
> >>> +%YAML 1.2
> >>> +---
> >>> +id: skel-device
> >>> +
> >>> +title: Skeleton Device
> >>> +
> >>> +maintainer:
> >>> +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> >>
> >> We'd want to tie this into get_maintainers.pl obviously.
> >>
> >>> +
> >>> +description: >
> >>> +  The Skeleton Device binding represents the SK11 device produced by
> >>> +  the Skeleton Corporation. The binding can also support compatible
> >>> +  clones made by second source vendors.
> >>> +
> >>> +compatible:
> >>> +  - name: "skel,sk11"
> >>> +  - name: "faux,fx11"
> >>
> >> Is this an OR or AND? We need both.
> > 
> > Hrm.  So, from the example below it's an OR.  But.. I think the
> > considerations here change when you separate out "tag" information as
> > I've suggested elsewhere.
> > 
> > The command AND case, I can think of is where you want both
> > "foo,new-model-device" and "foo,old-model-device".  I think the first
> > belongs in the "tag", but the second in the body of the requirements.
> > 
> > The OR case demonstrated here strikes me as very poor binding design -
> > but that doesn't mean we can totally avoid it, since we do have poor
> > bindings we want to convert.
> 
> I don't think so. Often "old" is a singleton, and "new" is one of
> several newer chips.

Yeah, so the trouble is bindings like that are kind of unclear
(although you can generally figure it out by convention).  Giving AND
and OR conditions gives more detail, but still doesn't really express
what you want here.

Supposeyou have the SK33 device, and its backwards compatible
revisions the SK34 and SK35.  There's also the SK38, which is
compatible with the original SK33, but not the revised versions.

So really, you have a cluster of related bindings:

SK33 binding:
	Applies when compatible contains "skel,sk33"
	Requires compatible contain "skel,sk33"
	Describes all the basic properties of the device
SK34 binding:
	Applies when compatible contains "skel,sk34"
	Requires compatible contain "skel,sk34" AND "skel,sk33"
	Inherits the SK33 binding
	Describes any properties necessary for SK34 extensions
SK35 binding
	Applies when compatible contains "skel,sk35"
	Requires compatible contains "skel,sk35", "skel,sk34" AND "skel,sk33"
	Inherits the SK34 binding
	Describes any properties for SK35 extensions
SK38 binding
	Applies when compatible contains "skel,sk38"
	Requires compatible contains "skel,sk38" AND "skel,sk33"
	Requires compatible DOES NOT contain "skel,sk34" or
	  "skel,sk35" (arguably you don't need to specify that, though)
	Inherits the SK33 binding
	Describes any properties for SK38 extensions

Plus, for backwards compatibility you might have to have the:

SK3x binding
	Deprecated
	Applies when compatible contains "skel,sk3x"
	Inherits the SK33 binding

> It is also very common for licensed IP blocks. All
> the vendors have different quirks and possibly different versions of the
> IP. In the better cases, we can have a vendor compat for each vendor and
> then a generic compat.

So, similar considerations here.  You really have a bunch of at least
potentially different bindings for each licensed variant, but all
based on the core common binding.

> > Fwiw, I think the right way to encode "OR" at least in the "tag" is to
> > pick just the preferred form, then have additional bindings tagged on
> > the alternative forms and inheriting the main binding.  I don't know
> > if that's practical in the short term though.

But, yeah, these are common cases so we need a less verbose way of
describing it than the full version above.  At least when the
differences between the devices don't require any new properties at
all.

I think we want to special case this for the compatible property
though.  I thinkAND and OR conditions will be complex, while still not
really expressing what we need for compatible.


> > 
> >> The complicated case is "one of {specific names} followed by {generic name}."
> >>
> >>> +    description: A clone of the original sk11 device
> >>> +
> >>> +required:
> >>> +  - name: "reg"
> >>
> >> We definitely need type info from the start.
> > 
> > It's interesting you should mention that here, because 'reg' is
> > actually a hard case for describint the type: because it's format is
> > determined as much by the parent bus binding as this node's binding.
> 
> And reg is well defined, so that's one I'm least worried about.

True/

> > I suspect it will be worth special-casing "reg" in order to make
> > common bindings more compact, but again, probably not in the first
> > pass.
> 
> Yes, I would fully expect reg to just be flagged as used and how many
> addresses.
> 
> > 
> >>> +    description: chip select address of skeleton device
> >>> +    reference: spi-slave
> >>
> >> I would like to not have to list properties if the inherited binding
> >> lists it. The problem is we need to say how many cells and the order
> >> (not a problem here, but for mmio devices).
> >>
> >> Perhaps we can list the reference at the top level for the node
> >> instead of for every property.
> > 
> > Yeah, I think it would be worth having a top-level "inherits" field.
> > 
> >>> +  - name: "spi-max-frequency"
> >>> +    description: >
> >>> +      Maximum SPI clocking speed of skeleton device in Hz, must be
> >>> +      1000000
> >>> +    reference: spi-slave
> >>
> >> Rather than listing the property and having constraint in description,
> >> perhaps we could add constraints like this:
> >>
> >> - spi-max-frequency-range: 1000000 1000000
> >>
> >> Or groups of constraints:
> >>
> >> - spi-max-frequency-constraints:
> >>   range: 1000000 1000000
> >>   some-other-constraint: <value>
> > 
> > Not for the first pass I think.  What I would suggest is having
> > "description" for pure informational stuff, and "constraint" or maybe
> > just "extra" for normative constraints expressed in English.
> 
> At least for simple constraints, I don't see the point of manually
> converting to something still not machine parse-able. We should be able
> to figure out a syntax. Sure, there may be complex examples we just punt
> on, but we can address most common cases. But if we can't define a
> syntax, then lets do nothing until we do.

Sure.

> > The idea here is that over time we can add new ways of expressing
> > constraints.  In the meantime tools can use the "extra" field to
> > preseve any difficult text in the current versions and at least let
> > tools know that they won't be able to check everything.
> 
> Just keeping the existing doc text will preserve things.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [RFC PATCH 2/5] Documentation: dt-bindings: add example DT binding document

[Matt Porter]

[-- Attachment #1: Type: text/plain, Size: 5651 bytes --]

On Thu, Sep 10, 2015 at 05:08:50PM +1000, David Gibson wrote:
> On Fri, Aug 28, 2015 at 09:53:06AM -0500, Rob Herring wrote:
> > On Fri, Aug 28, 2015 at 12:23 AM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > > Add a skeleton DT binding document that serves as the canonical
> > > example for implementing YAML-based DT bindings documentation.
> > > The skeleton binding illustrates use of all fields and variations
> > > described in the dt-binding-format.txt documentation.
> > >
> > > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> > > ---
> > >  Documentation/devicetree/bindings/skeleton.yaml | 98 +++++++++++++++++++++++++
> > >  1 file changed, 98 insertions(+)
> > >  create mode 100644 Documentation/devicetree/bindings/skeleton.yaml
> > >
> > > diff --git a/Documentation/devicetree/bindings/skeleton.yaml b/Documentation/devicetree/bindings/skeleton.yaml
> > > new file mode 100644
> > > index 0000000..175965f
> > > --- /dev/null
> > > +++ b/Documentation/devicetree/bindings/skeleton.yaml
> > > @@ -0,0 +1,98 @@
> > > +%YAML 1.2
> > > +---
> > > +id: skel-device
> > > +
> > > +title: Skeleton Device
> > > +
> > > +maintainer:
> > > +  - name: Skeleton Person <skel-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > 
> > We'd want to tie this into get_maintainers.pl obviously.
> > 
> > > +
> > > +description: >
> > > +  The Skeleton Device binding represents the SK11 device produced by
> > > +  the Skeleton Corporation. The binding can also support compatible
> > > +  clones made by second source vendors.
> > > +
> > > +compatible:
> > > +  - name: "skel,sk11"
> > > +  - name: "faux,fx11"
> > 
> > Is this an OR or AND? We need both.
> 
> Hrm.  So, from the example below it's an OR.  But.. I think the
> considerations here change when you separate out "tag" information as
> I've suggested elsewhere.
> 
> The command AND case, I can think of is where you want both
> "foo,new-model-device" and "foo,old-model-device".  I think the first
> belongs in the "tag", but the second in the body of the requirements.
> 
> The OR case demonstrated here strikes me as very poor binding design -
> but that doesn't mean we can totally avoid it, since we do have poor
> bindings we want to convert.
> 
> Fwiw, I think the right way to encode "OR" at least in the "tag" is to
> pick just the preferred form, then have additional bindings tagged on
> the alternative forms and inheriting the main binding.  I don't know
> if that's practical in the short term though.

I reworked this as a constraint key with C-like syntax to express
allowable compatible strings. It's got some warts but let's take
a look at v2 and see if good constraint syntax can handle all these
cases. I use some macro like constructs to denote a generic compatible
string that must be included in addition to the part-specific string.

> 
> > The complicated case is "one of {specific names} followed by {generic name}."
> > 
> > > +    description: A clone of the original sk11 device
> > > +
> > > +required:
> > > +  - name: "reg"
> > 
> > We definitely need type info from the start.
> 
> It's interesting you should mention that here, because 'reg' is
> actually a hard case for describint the type: because it's format is
> determined as much by the parent bus binding as this node's binding.
> 
> I suspect it will be worth special-casing "reg" in order to make
> common bindings more compact, but again, probably not in the first
> pass.

I've added type in v2 and in comments note the "reg" is indeed a
special case and derived not so much from the inherited binding but
from the actual parent node as used in an implementation.

> 
> > > +    description: chip select address of skeleton device
> > > +    reference: spi-slave
> > 
> > I would like to not have to list properties if the inherited binding
> > lists it. The problem is we need to say how many cells and the order
> > (not a problem here, but for mmio devices).
> > 
> > Perhaps we can list the reference at the top level for the node
> > instead of for every property.
> 
> Yeah, I think it would be worth having a top-level "inherits" field.
 
Added in v2

> > > +  - name: "spi-max-frequency"
> > > +    description: >
> > > +      Maximum SPI clocking speed of skeleton device in Hz, must be
> > > +      1000000
> > > +    reference: spi-slave
> > 
> > Rather than listing the property and having constraint in description,
> > perhaps we could add constraints like this:
> > 
> > - spi-max-frequency-range: 1000000 1000000
> > 
> > Or groups of constraints:
> > 
> > - spi-max-frequency-constraints:
> >   range: 1000000 1000000
> >   some-other-constraint: <value>
> 
> Not for the first pass I think.  What I would suggest is having
> "description" for pure informational stuff, and "constraint" or maybe
> just "extra" for normative constraints expressed in English.
> 
> The idea here is that over time we can add new ways of expressing
> constraints.  In the meantime tools can use the "extra" field to
> preseve any difficult text in the current versions and at least let
> tools know that they won't be able to check everything.

I added the definitions of these and an example of use. This by no
means indicates that we plan to add constraints for every converted
binding. However, in some cases it's very easy to add with a C-like
constraint syntax. I'm sure there's cases where this will fall over
and needs to be extended but let's review that in the context of the
v2 posting.

-Matt

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [RFC PATCH 1/5] Documentation: dt-bindings: add documentation on new DT binding format

[Matt Porter]

On Tue, Sep 01, 2015 at 09:59:14AM -0700, Tim Bird wrote:
> On Thu, Aug 27, 2015 at 10:23 PM, Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org> wrote:
> > Documentation explaining the syntax and format of the YAML-based DT binding
> > documentation.
> >
> > Signed-off-by: Matt Porter <mporter-OWPKS81ov/FWk0Htik3J/w@public.gmane.org>
> > ---
> >  .../devicetree/bindings/dt-binding-format.txt      | 106 +++++++++++++++++++++
> >  1 file changed, 106 insertions(+)
> >  create mode 100644 Documentation/devicetree/bindings/dt-binding-format.txt
> >
> > diff --git a/Documentation/devicetree/bindings/dt-binding-format.txt b/Documentation/devicetree/bindings/dt-binding-format.txt
> > new file mode 100644
> > index 0000000..f9acc22
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/dt-binding-format.txt
> > @@ -0,0 +1,106 @@
> > +--------------------------
> > +Device Tree Binding Format
> > +--------------------------
> > +
> > +Background
> > +----------
> > +
> > +DT bindings historically were written as text in prose format which
> > +led to issues in usability of that source documentation. Some of
> > +these issues include the need to programmatically process binding
> > +source documentation to do DTS validation, perform mass updates to
> > +format/style, and to generate publishable documentation in HTML or
> > +PDF form.
> > +
> > +Overview
> > +--------
> > +
> > +The DT binding format is based on the YAML text markup language.
> > +Although there are many text markup options available, YAML
> > +fulfills all requirements considered for a DT binding source format
> > +which include:
> > +
> > +1) Must be human readable
> > +2) Must be easily translated to other data formats (XML, JSON, etc).
> > +3) Must have sufficient tools and libraries to enable developers to
> > +   build new tools for DT binding processing
> > +4) Must have a complete spec to refer to syntax
> > +
> > +YAML is documentated in the specification found at
> > +http://www.yaml.org/spec/1.2/spec.html
> > +
> > +The required YAML DT binding tag format and syntax are defined in
> > +the following sections.
> > +
> > +YAML DT Binding Syntax
> > +----------------------
> > +
> > +* Lines starting with "#" are comments and not part of the binding itself
> > +* "%YAML 1.2" starts a file, indicating the version of YAML in use
> 
> I think it would be good to specify a DT Binding Format version as well.
> I would expect we may expand the syntax, or we may alter it, so it
> would be good to have a number for our specific syntax.  This would
> help in some circumstances, if we make multiple passes over the
> documents to achieve our conversion.  Maybe this should be added
> below.

Added this in v2, thanks.

> 
> > +* "---" starts a binding document
> > +* "..." ends a binding document
> > +* Multiple binding documents may exist in a single file
> > +* Tabs are not permitted
> > +* Scope is denoted by indentation of two spaces
> > +* Key value pairs are denoted by "key: value"
> > +* Sequences are denoted by "- "
> > +* Scalar values may convert newlines to spaces and preserve blank
> > +  lines for long description formatting using ">"
> > +* Scalar values may escape all reserved characters and preserve
> > +  newlines by using "|" to denote literal style
> > +
> > +For additional information on YAML syntax, refer to the specification
> > +at http://www.yaml.org/spec/1.2/spec.html
> > +
> > +YAML DT Binding Format
> > +----------------------
> > +
> > +The following YAML types are supported in the DT binding format:
> > +
> 
> Based on my comment above, I recommend:
> [R] binding-format: 1
> 
> > +* [R] id: unique identifier in property form (e.g. skel-device)
> > +
> > +* [R] title: title of the binding
> > +
> > +* [O] maintainer: sequence of maintainers
> > +      [R] name: name and email of maintainer or mailing list in RFC822
> > +                form.
> > +
> > +* [O] description: full description of the binding
> > +
> > +* [O] compatible: sequence of valid compatible descriptors
> > +      [R] name: the compatible string surrounded in double quotes
> > +      [O] deprecated: a deprecated compatible string surrounded in
> > +                      double quotes
> > +      [O] description: description of the compatible string
> 
> compatible is a property like all the others, and I think it's confusing
> to have it in a separate section.  For example, when listing other
> properties, the "name:" field specifies the property name, but for
> the compatible section the "name:" field specifies the possible values
> for the compatible property.  This is (at least to me) quite confusing.
> 
> Can we make this a regular property, with an option 'deprecated' attribute.
> I guess we may need to distinguish between deprecated properties and
> deprecated vales of properties.
> The above 'deprecated' specifies deprecated values for the compatible
> property, while
> the section below specifies deprecated properties.  I don't see a way
> to indicate a deprecated
> value for a still-existing property, but that may be too much detail for now.

I've made an attempt at this in v2. There's some ugly fall-out I haven't
decided the best way to handle. That is, I can show a allowable
compatible strings including a specific, generic type syntax using
C-like expressions. However, showing deprecated compatible requires
a special key. I'll be interested if anybody can suggestion something
cleaner. There's definitely a trade-off in folding compatibles into
properties

> 
> 
> > +* [O] required: sequence of required properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> 
> I would recommend we add types and value constraints to the
> binding schema:
> 
> [O] type: type of value for this property, and if specified must be
> one of the following (without quotes):
> "string", "number", "array", "phandle"

I modified this a bit to be "string", "int", "array", "phandle", and
"empty". I find "empty" to be useful for those properties and a
validator will find that useful to validate those boolean type
property switches it encounters i.e. skip constraint checking

> [O] value constraint: specifies a limit on the value for a property.
> This can be specified as English text, or be a range or choice constraint.
> The latter two should be expressed with the following exact english
> phrases (without quotes):
> "must be in the range x to y, inclusive"
> where x and y are numbers
> "must be one of <choice 1>, <choice 2>, ..."
> where <choice x> is a string, which should be quoted
> if it includes a comma.

Got them, but Pantelis' C syntax suggestion for constraints seemed
popular and flexible so I went with that initially, please let me
know what you think of that.

> 
> Any other constraint is expressed in free-form English text

Yes, I believe the description field can hold this information
until bindings are fully converted with constraint info in the 
C-syntax (if we stick with that).

> I would recommend adding these to this binding schema doc, but
> not requiring them on the first conversion pass of the docs.

Sounds good.

> Basically my idea is that you want the each pass over the binding
> documents to be as painless as possible.  Every time a user encounters
> something they're not sure about, it will slow them down.  There will
> be lots of ambiguous situations, and we want the process to be as
> lossless as possible, so I would recommend a strategy of doing as
> much as people can, and if in doubt just put any outstanding text
> in the descriptions.  The descriptions will always be free-form text,
> and we can come along later and convert these to types, constraints,
> as we see fit.

Agreed!

> > +* [O] optional: sequence of optional properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [O] deprecated: sequence of deprecated properties:
> > +      [R] name: name of the property surrounded in double quotes
> > +      [R] description: description of the property
> > +      [O] reference: optional reference to a binding id
> > +
> > +* [R] example: sequence of examples:
> > +      [R] dts: DT source of example usage. The example text must use
> > +               literal style ("|") so that it retains indentation and
> > +               newlines.
> > +      [O] description: description of the example
> > +
> 
> I'd recommend having a top-level notes section:
> * [O] note: any other notes about this node that are not covered by
> preceding property descriptions.

Added in v2, thanks.

> 
> > +Note: [R] and [O] denote required and optional fields, respectively.
> 
> I'd put this note before the format description.

Done, thanks.

> 
>  -- Tim
> 
> > +
> > +Skeleton Binding
> > +----------------
> > +
> > +The skeleton.yaml binding found in the top of the DT binding tree
> > +is the canonical example of syntax and format to use when writing
> > +a DT binding document. It is maintained with the latest formatting
> > +conventions, making it the best starting point when writing a new DT
> > +binding.
> > --