[OE-core][RFC] Layer Setup JSON schema and example

[OE-core][RFC] Layer Setup JSON schema and example

[Joshua Watt]

Defines a common schema for layer setup that can be consumed by tools to
know how to fetch and assemble layers for end users. Also includes an
example of the layer setup that constructs poky for reference.

The schema can be used to validate a layer setup file with the commands:

 $ python3 -m pip install jsonschema
 $ jsonschema -i meta/lib/layers.example.json meta/lib/layers.schema.json

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/layers.example.json |  35 +++++++++++
 meta/lib/layers.schema.json  | 113 +++++++++++++++++++++++++++++++++++
 2 files changed, 148 insertions(+)
 create mode 100644 meta/lib/layers.example.json
 create mode 100644 meta/lib/layers.schema.json

diff --git a/meta/lib/layers.example.json b/meta/lib/layers.example.json
new file mode 100644
index 0000000000..33264dac9e
--- /dev/null
+++ b/meta/lib/layers.example.json
@@ -0,0 +1,35 @@
+{
+    "version": "1.0",
+    "sources": [
+        {
+            "name": "poky",
+            "path": "poky",
+            "description": "Poky reference distribution",
+            "remote": {
+                "type": "git",
+                "branch": "master",
+                "rev": "bc6d96e69684253a7236594cb0af2738be06b7a9",
+                "remotes": [
+                    {
+                        "name": "origin",
+                        "uri": "git://git.yoctoproject.org/poky"
+                    }
+                ]
+            },
+            "layers": [
+                {
+                    "name": "meta",
+                    "subpath": "meta"
+                },
+                {
+                    "name": "meta-poky",
+                    "subpath": "meta-poky"
+                },
+                {
+                    "name": "meta-yocto-bsp",
+                    "subpath": "meta-yocto-bsp"
+                }
+            ]
+        }
+    ]
+}
diff --git a/meta/lib/layers.schema.json b/meta/lib/layers.schema.json
new file mode 100644
index 0000000000..19df89d38b
--- /dev/null
+++ b/meta/lib/layers.schema.json
@@ -0,0 +1,113 @@
+{
+    "description": "OpenEmbedder Layer Setup Manifest",
+    "type": "object",
+    "additionalProperties": false,
+    "required": [
+        "version"
+    ],
+    "properties": {
+        "version": {
+            "description": "The version of this document; currently '1.0'",
+            "enum": ["1.0"]
+        },
+        "sources": {
+            "description": "The list of layer sources",
+            "type": "array",
+            "items": {
+                "type": "object",
+                "description": "The upstream source from which a set of layers may be fetched",
+                "additionalProperties": false,
+                "required": [
+                    "name",
+                    "path"
+                ],
+                "properties": {
+                    "name": {
+                        "description": "The name of this layer source",
+                        "type": "string"
+                    },
+                    "path": {
+                        "description": "The path where this layer source will be placed when fetching",
+                        "type": "string"
+                    },
+                    "description": {
+                        "description": "A description of this layer source",
+                        "type": "string"
+                    },
+                    "layers": {
+                        "description": "The list of layers to be used from this upstream source",
+                        "type": "array",
+                        "items": {
+                            "description": "A layer from the upstream source",
+                            "type": "object",
+                            "additionalProperties": false,
+                            "required": [
+                                "name"
+                            ],
+                            "properties": {
+                                "name": {
+                                    "description": "The name of the layer",
+                                    "type": "string"
+                                },
+                                "subpath": {
+                                    "description": "The subpath (relative to the source root) for this layer. Omit if the source root is the layer path",
+                                    "type": "string"
+                                }
+                            }
+                        }
+                    },
+                    "remote": {
+                        "oneOf": [
+                            {
+                                "description": "A remote git source from which to fetch",
+                                "type": "object",
+                                "additionalProperties": false,
+                                "required": [
+                                    "type",
+                                    "rev"
+                                ],
+                                "properties": {
+                                    "type": {
+                                        "description": "This is a git source",
+                                        "enum": ["git"]
+                                    },
+                                    "branch": {
+                                        "description": "The git branch to fetch (optional)",
+                                        "type": "string"
+                                    },
+                                    "rev": {
+                                        "description": "The git revision to checkout",
+                                        "type": "string"
+                                    },
+                                    "remotes": {
+                                        "description": "The list of git remotes to add to this repository",
+                                        "type": "array",
+                                        "items": {
+                                            "description": "A git remote",
+                                            "type": "object",
+                                            "addtionalProperties": false,
+                                            "required": [
+                                                "name",
+                                                "uri"
+                                            ],
+                                            "properties": {
+                                                "name": {
+                                                    "description": "The name of the remote",
+                                                    "type": "string"
+                                                },
+                                                "uri": {
+                                                    "description": "The URI for the remote",
+                                                    "type": "string"
+                                                }
+                                            }
+                                        }
+                                    }
+                                }
+                            }
+                        ]
+                    }
+                }
+            }
+        }
+    }
+}
-- 
2.33.0

Re: [OE-core][RFC] Layer Setup JSON schema and example

[Alexander Kanavin]

On Wed, 6 Jul 2022 at 21:54, Joshua Watt <JPEWhacker@gmail.com> wrote:
>  $ python3 -m pip install jsonschema
>  $ jsonschema -i meta/lib/layers.example.json meta/lib/layers.schema.json

Since jsonschema is not a standard module, we need to think for a bit
about how and when validation should occur with tools available
directly in core. I would like it to happen every time json is
produced or (about to be) consumed, but then the module should be
coming from somewhere, either from the host or from a native recipe,
both look like not a great idea. Thoughts? Is this where 'oe-setup
installed via pip' approach starts to show benefits? :)

Alex

Re: [OE-core][RFC] Layer Setup JSON schema and example

[Joshua Watt]

On Wed, Jul 6, 2022 at 4:11 PM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> On Wed, 6 Jul 2022 at 21:54, Joshua Watt <JPEWhacker@gmail.com> wrote:
> >  $ python3 -m pip install jsonschema
> >  $ jsonschema -i meta/lib/layers.example.json meta/lib/layers.schema.json
>
> Since jsonschema is not a standard module, we need to think for a bit
> about how and when validation should occur with tools available
> directly in core. I would like it to happen every time json is
> produced or (about to be) consumed, but then the module should be
> coming from somewhere, either from the host or from a native recipe,
> both look like not a great idea. Thoughts? Is this where 'oe-setup
> installed via pip' approach starts to show benefits? :)

Ya, for a 100% "standalone" tool, I'm not sure if we can do any real
validation in python without external modules. For testing, can either
add the requirement for the `python3-jsonschema` system package, or
maybe use python3-jsonschema-native?

>
> Alex

Re: [OE-core][RFC] Layer Setup JSON schema and example

[Ross Burton]

> On 6 Jul 2022, at 20:53, Joshua Watt via lists.openembedded.org <JPEWhacker=gmail.com@lists.openembedded.org> wrote:
>
> Defines a common schema for layer setup that can be consumed by tools to
> know how to fetch and assemble layers for end users. Also includes an
> example of the layer setup that constructs poky for reference.

JSON is terrible for humans to read or edit, is this intended to be purely for the machine, or are humans meant to read it too?  If humans are meant to work on the file I’d really prefer something readable like YAML.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [OE-core][RFC] Layer Setup JSON schema and example

[Alexander Kanavin]

On Thu, 7 Jul 2022 at 12:03, Ross Burton <ross.burton@arm.com> wrote:
> JSON is terrible for humans to read or edit, is this intended to be purely for the machine, or are humans meant to read it too?  If humans are meant to work on the file I’d really prefer something readable like YAML.

Wait, how is YAML more readable, if JSON is valid YAML?

Alex

Re: [OE-core][RFC] Layer Setup JSON schema and example

[Ross Burton]

> Wait, how is YAML more readable, if JSON is valid YAML?

But YAML is not valid JSON, and YAML is the one which can be readable, unlike JSON.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [OE-core][RFC] Layer Setup JSON schema and example

[Alexander Kanavin]

On Thu, 7 Jul 2022 at 12:36, Ross Burton <Ross.Burton@arm.com> wrote:
> > Wait, how is YAML more readable, if JSON is valid YAML?
>
> But YAML is not valid JSON, and YAML is the one which can be readable, unlike JSON.

YAML is also not an included python battery, while JSON is. I'll stick
with JSON for now, if you provide specific examples of how the JSON
that we're defining would look more readable rendered in YAML, that
would be appreciated.

Alex

Re: [OE-core][RFC] Layer Setup JSON schema and example

[Joshua Watt]

On Thu, Jul 7, 2022 at 5:40 AM Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> On Thu, 7 Jul 2022 at 12:36, Ross Burton <Ross.Burton@arm.com> wrote:
> > > Wait, how is YAML more readable, if JSON is valid YAML?
> >
> > But YAML is not valid JSON, and YAML is the one which can be readable, unlike JSON.
>
> YAML is also not an included python battery, while JSON is. I'll stick
> with JSON for now, if you provide specific examples of how the JSON
> that we're defining would look more readable rendered in YAML, that
> would be appreciated.

I generally agree that YAML is better for humans, but using JSON now
allows us to make progress without having to hash out that discussion,
and won't exclude using YAML in the future (since YAML is a strict
superset of JSON).

>
> Alex