[PATCH][hardknott 1/3] arm-bsp/tc0-artifacts-image: restrict to TC0

[PATCH][hardknott 1/3] arm-bsp/tc0-artifacts-image: restrict to TC0

[Ross Burton]

From: Ross Burton <ross@burtonini.com>

This recipe is for TC0, so set COMPATIBLE_MACHINE appropriately.

This is both logically correct, and fixes a Yocto Compatible test failure
where tc0-artifacts-image isn't buildable for arbitrary machines.

Change-Id: I6381fc08076d4ddea254f7efc84f90c4a7f65d26
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jon Mason <jon.mason@arm.com>
---
 meta-arm-bsp/recipes-bsp/images/tc0-artifacts-image.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta-arm-bsp/recipes-bsp/images/tc0-artifacts-image.bb b/meta-arm-bsp/recipes-bsp/images/tc0-artifacts-image.bb
index 43c1eda..b2ad356 100644
--- a/meta-arm-bsp/recipes-bsp/images/tc0-artifacts-image.bb
+++ b/meta-arm-bsp/recipes-bsp/images/tc0-artifacts-image.bb
@@ -6,6 +6,8 @@ SUMMARY = "Total Compute Images"
 DESCRIPTION = "Build all the images required for Total Compute platform"
 LICENSE = "Apache-2.0"
 
+COMPATIBLE_MACHINE = "tc0"
+
 inherit nopackages
 
 # The last image to be built is trusted-firmware-a
-- 
2.25.1

[PATCH][hardknott 2/3] arm-bsp/linux: guard FILESEXTRAPATHS changes in overrides

[Ross Burton]

From: Ross Burton <ross@burtonini.com>

Because of how the kernel class searches for files any changes to
FILESEXTRAPATHS should be done in an override so it doesn't potentially
impact unrelated builds.

Issue spotted by running yocto-check-layer.

Change-Id: Id3b741d184a024d0cc6d2d5031e150a8f0ae4b0d
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Jon Mason <jon.mason@arm.com>
---
 .../recipes-kernel/linux/linux-arm-platforms.inc     | 12 +++++++++++-
 .../recipes-kernel/linux/linux-yocto-corstone700.inc |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta-arm-bsp/recipes-kernel/linux/linux-arm-platforms.inc b/meta-arm-bsp/recipes-kernel/linux/linux-arm-platforms.inc
index 1737a05..17a036e 100644
--- a/meta-arm-bsp/recipes-kernel/linux/linux-arm-platforms.inc
+++ b/meta-arm-bsp/recipes-kernel/linux/linux-arm-platforms.inc
@@ -5,7 +5,11 @@
 # in this file. Update SRC_URI and do_patch for building images with custom dts
 #
 
-FILESEXTRAPATHS_prepend := "${THISDIR}:${THISDIR}/files:"
+# We can't set FILESEXTRAPATHS once because of how the kernel classes search for
+# config fragments. Discussion is ongoing as to whether this is the correct
+# solution, or a workaround.
+# https://bugzilla.yoctoproject.org/show_bug.cgi?id=14154
+ARMBSPFILESPATHS := "${THISDIR}:${THISDIR}/files:"
 
 # Arm platforms kmeta
 SRC_URI_KMETA = "file://arm-platforms-kmeta;type=kmeta;name=arm-platforms-kmeta;destsuffix=arm-platforms-kmeta"
@@ -26,6 +30,7 @@ COMPATIBLE_MACHINE_corstone500 = "corstone500"
 #
 COMPATIBLE_MACHINE_fvp-base = "fvp-base"
 KMACHINE_fvp-base = "fvp"
+FILESEXTRAPATHS_prepend_fvp-base := "${ARMBSPFILESPATHS}"
 SRC_URI_append_fvp-base = " file://fvp-base-dts.patch"
 
 #
@@ -33,6 +38,7 @@ SRC_URI_append_fvp-base = " file://fvp-base-dts.patch"
 #
 COMPATIBLE_MACHINE_fvp-base-arm32 = "fvp-base-arm32"
 KMACHINE_fvp-base-arm32 = "fvp-arm32"
+FILESEXTRAPATHS_prepend_fvp-base-arm32 := "${ARMBSPFILESPATHS}"
 SRC_URI_append_fvp-base-arm32 = " file://fvp-base-arm32-dts.patch \
                                   file://0001-ARM-vexpress-enable-GICv3.patch"
 
@@ -43,6 +49,7 @@ COMPATIBLE_MACHINE_juno = "juno"
 KMACHINE_juno = "juno"
 KBUILD_DEFCONFIG_juno = "defconfig"
 KCONFIG_MODE_juno = "--alldefconfig"
+FILESEXTRAPATHS_prepend_juno := "${ARMBSPFILESPATHS}"
 SRC_URI_append_juno = " \
     file://0001-mailbox-add-support-for-doorbell-signal-mode-control.patch \
     file://0002-dt-bindings-mailbox-add-bindings-to-support-ARM-MHU-.patch \
@@ -68,6 +75,7 @@ KCONFIG_MODE_sgi575 = "--alldefconfig"
 COMPATIBLE_MACHINE_tc0 = "tc0"
 KMACHINE_tc0 = "tc0"
 KCONFIG_MODE_tc0 = "--alldefconfig"
+FILESEXTRAPATHS_prepend_tc0 := "${ARMBSPFILESPATHS}"
 SRC_URI_append_tc0 = " \
     file://defconfig \
     file://0001-drm-Add-component-aware-simple-encoder.patch \
@@ -94,6 +102,7 @@ FILESEXTRAPATHS_prepend_n1sdp := "${THISDIR}/linux-yocto-5.10/n1sdp:"
 COMPATIBLE_MACHINE_n1sdp = "n1sdp"
 KBUILD_DEFCONFIG_n1sdp = "defconfig"
 KCONFIG_MODE_n1sdp = "--alldefconfig"
+FILESEXTRAPATHS_prepend_n1sdp := "${ARMBSPFILESPATHS}"
 SRC_URI_append_n1sdp = " \
     file://0001-pci_quirk-add-acs-override-for-PCI-devices.patch \
     file://0002-pcie-Add-quirk-for-the-Arm-Neoverse-N1SDP-platform.patch \
@@ -119,4 +128,5 @@ COMPATIBLE_MACHINE_musca-s1 = "(^$)"
 # FVP BaseR AEMv8r64 Machine
 #
 COMPATIBLE_MACHINE_fvp-baser-aemv8r64 = "fvp-baser-aemv8r64"
+FILESEXTRAPATHS_prepend_fvp-baser-aemv8r64 := "${ARMBSPFILESPATHS}"
 SRC_URI_append_fvp-baser-aemv8r64 = " file://fvp-baser-aemv8r64.dts;subdir=git/arch/arm64/boot/dts/arm"
diff --git a/meta-arm-bsp/recipes-kernel/linux/linux-yocto-corstone700.inc b/meta-arm-bsp/recipes-kernel/linux/linux-yocto-corstone700.inc
index f2a9a54..eb584e5 100644
--- a/meta-arm-bsp/recipes-kernel/linux/linux-yocto-corstone700.inc
+++ b/meta-arm-bsp/recipes-kernel/linux/linux-yocto-corstone700.inc
@@ -1,4 +1,4 @@
-FILESEXTRAPATHS_prepend := "${THISDIR}/files/corstone700:"
+FILESEXTRAPATHS_prepend := "${THISDIR}:${THISDIR}/files:"
 
 COMPATIBLE_MACHINE = "${MACHINE}"
 
-- 
2.25.1

[PATCH][hardknott 3/3] ci: add yocto-check-layers step

[Ross Burton]

Add a job to run yocto-check-layers to validate a selection of layers
are Yocto Project Compatible.

Right now we validate:
- meta-arm
- meta-arm-bsp
- meta-arm-toolchain
- meta-gem5

meta-atp and meta-arm-autonomy are currently skipped: meta-atp fails and
meta-arm-autonomy has a larger set of dependant layers and will be added
later.

Change-Id: Ia0c8c16e4228eeb461bd213b30703e950ede656f
Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 .gitlab-ci.yml      |  8 ++++++++
 ci/check-layers.py  | 43 +++++++++++++++++++++++++++++++++++++++++++
 ci/check-layers.yml |  7 +++++++
 3 files changed, 58 insertions(+)
 create mode 100755 ci/check-layers.py
 create mode 100644 ci/check-layers.yml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b97d106..7ad141f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -71,6 +71,14 @@ machine-coverage:
 # Build stage, the actual build jobs
 #
 
+# Validate layers are Yocto Project Compatible
+check-layers:
+  extends: .setup
+  coverage: '/Coverage: \d+/'
+  script:
+  - kas shell --update --force-checkout ci/base.yml:ci/meta-python.yml --command \
+    "$CI_PROJECT_DIR/ci/check-layers.py $CI_PROJECT_DIR/ci/check-layers.yml $CI_PROJECT_DIR $KAS_WORK_DIR"
+
 corstone500:
   extends: .build
 
diff --git a/ci/check-layers.py b/ci/check-layers.py
new file mode 100755
index 0000000..7b607ae
--- /dev/null
+++ b/ci/check-layers.py
@@ -0,0 +1,43 @@
+#! /usr/bin/env python3
+
+import argparse
+import pathlib
+import re
+import subprocess
+import sys
+
+import yaml
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("config", type=argparse.FileType())
+    parser.add_argument("metaarm", type=pathlib.Path, help="Path to meta-arm")
+    parser.add_argument("others", type=pathlib.Path, help="Path to parent of dependencies")
+    args = parser.parse_args()
+
+    config = yaml.safe_load(args.config)
+    layers = config["layers"]
+    dependencies = config["dependencies"]
+
+    found_layers = [p for p in args.metaarm.glob("meta-*") if p.is_dir()]
+    print(f"Testing {len(layers)} layers: {', '.join(layers)}.")
+    print(f"Found {len(found_layers)} layers in meta-arm.")
+    print()
+
+    cli = ["yocto-check-layer-wrapper",]
+    cli.extend([args.metaarm / layer for layer in layers])
+    cli.append("--dependency")
+    cli.extend([args.others / layer for layer in dependencies])
+
+    passed = 0
+    process = subprocess.Popen(cli, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
+    while True:
+        line = process.stdout.readline()
+        if process.poll() is not None:
+            break
+        print(line.strip())
+        if re.search(r"meta-.+ PASS", line):
+            passed += 1
+
+    print(f"Coverage: {int(passed / len(found_layers) * 100)}%")
+    sys.exit(process.returncode)
diff --git a/ci/check-layers.yml b/ci/check-layers.yml
new file mode 100644
index 0000000..7d2820c
--- /dev/null
+++ b/ci/check-layers.yml
@@ -0,0 +1,7 @@
+layers:
+  - meta-arm
+  - meta-arm-bsp
+  - meta-arm-toolchain
+  - meta-gem5
+dependencies:
+  - meta-openembedded/meta-oe
-- 
2.25.1

Re: [meta-arm] [PATCH][hardknott 3/3] ci: add yocto-check-layers step

[Denys Dmytriyenko]

Ross,

Thanks! Do you need these changes on dunfell as well?


On Wed, Jun 23, 2021 at 01:28:14PM +0100, Ross Burton wrote:
> Add a job to run yocto-check-layers to validate a selection of layers
> are Yocto Project Compatible.
> 
> Right now we validate:
> - meta-arm
> - meta-arm-bsp
> - meta-arm-toolchain
> - meta-gem5
> 
> meta-atp and meta-arm-autonomy are currently skipped: meta-atp fails and
> meta-arm-autonomy has a larger set of dependant layers and will be added
> later.
> 
> Change-Id: Ia0c8c16e4228eeb461bd213b30703e950ede656f
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  .gitlab-ci.yml      |  8 ++++++++
>  ci/check-layers.py  | 43 +++++++++++++++++++++++++++++++++++++++++++
>  ci/check-layers.yml |  7 +++++++
>  3 files changed, 58 insertions(+)
>  create mode 100755 ci/check-layers.py
>  create mode 100644 ci/check-layers.yml
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index b97d106..7ad141f 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -71,6 +71,14 @@ machine-coverage:
>  # Build stage, the actual build jobs
>  #
>  
> +# Validate layers are Yocto Project Compatible
> +check-layers:
> +  extends: .setup
> +  coverage: '/Coverage: \d+/'
> +  script:
> +  - kas shell --update --force-checkout ci/base.yml:ci/meta-python.yml --command \
> +    "$CI_PROJECT_DIR/ci/check-layers.py $CI_PROJECT_DIR/ci/check-layers.yml $CI_PROJECT_DIR $KAS_WORK_DIR"
> +
>  corstone500:
>    extends: .build
>  
> diff --git a/ci/check-layers.py b/ci/check-layers.py
> new file mode 100755
> index 0000000..7b607ae
> --- /dev/null
> +++ b/ci/check-layers.py
> @@ -0,0 +1,43 @@
> +#! /usr/bin/env python3
> +
> +import argparse
> +import pathlib
> +import re
> +import subprocess
> +import sys
> +
> +import yaml
> +
> +if __name__ == "__main__":
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument("config", type=argparse.FileType())
> +    parser.add_argument("metaarm", type=pathlib.Path, help="Path to meta-arm")
> +    parser.add_argument("others", type=pathlib.Path, help="Path to parent of dependencies")
> +    args = parser.parse_args()
> +
> +    config = yaml.safe_load(args.config)
> +    layers = config["layers"]
> +    dependencies = config["dependencies"]
> +
> +    found_layers = [p for p in args.metaarm.glob("meta-*") if p.is_dir()]
> +    print(f"Testing {len(layers)} layers: {', '.join(layers)}.")
> +    print(f"Found {len(found_layers)} layers in meta-arm.")
> +    print()
> +
> +    cli = ["yocto-check-layer-wrapper",]
> +    cli.extend([args.metaarm / layer for layer in layers])
> +    cli.append("--dependency")
> +    cli.extend([args.others / layer for layer in dependencies])
> +
> +    passed = 0
> +    process = subprocess.Popen(cli, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
> +    while True:
> +        line = process.stdout.readline()
> +        if process.poll() is not None:
> +            break
> +        print(line.strip())
> +        if re.search(r"meta-.+ PASS", line):
> +            passed += 1
> +
> +    print(f"Coverage: {int(passed / len(found_layers) * 100)}%")
> +    sys.exit(process.returncode)
> diff --git a/ci/check-layers.yml b/ci/check-layers.yml
> new file mode 100644
> index 0000000..7d2820c
> --- /dev/null
> +++ b/ci/check-layers.yml
> @@ -0,0 +1,7 @@
> +layers:
> +  - meta-arm
> +  - meta-arm-bsp
> +  - meta-arm-toolchain
> +  - meta-gem5
> +dependencies:
> +  - meta-openembedded/meta-oe
> -- 
> 2.25.1
> 

Re: [meta-arm] [PATCH][hardknott 3/3] ci: add yocto-check-layers step

[Ross Burton]

On Wed, 23 Jun 2021 at 20:22, Denys Dmytriyenko <denis@denix.org> wrote:
> Thanks! Do you need these changes on dunfell as well?

Step at a time...  Dunfell passing in our CI needs some further
changes to oe-core (which fails to build on arm64 hosts).

Ross