[OE-core][PATCH v7 0/3] Add a new bbclass that abstracts the generation of FIT blobs

[OE-core][PATCH v7 0/3] Add a new bbclass that abstracts the generation of FIT blobs

[Nandor Han]

Description
----------
Add a new class and unittest for generating FIT blobs.


Testing
-------

    1. linux-yocto_5.4.bbappend was modified to have the following configuration:

    ```
    inherit fit-image

    KERNEL_IMAGE_NODE[name] = "kernel"
    KERNEL_IMAGE_NODE[description] = "${PF}"
    KERNEL_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/zImage")'
    KERNEL_IMAGE_NODE[type] = "kernel"
    KERNEL_IMAGE_NODE[arch] = "${ARCH}"
    KERNEL_IMAGE_NODE[os] = "linux"
    KERNEL_IMAGE_NODE[compression] = "none"
    KERNEL_IMAGE_NODE[load] = "${UBOOT_LOADADDRESS}"
    KERNEL_IMAGE_NODE[entry] = "${UBOOT_ENTRYPOINT}"
    KERNEL_IMAGE_NODE[hash] = "sha256"

    FDT_IMAGE_NODE[name] = "fdt"
    FDT_IMAGE_NODE[description] = "FDT blob"
    FDT_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/dts/am335x-bone.dtb")'
    FDT_IMAGE_NODE[type] = "flat_dt"
    FDT_IMAGE_NODE[arch] = "${ARCH}"
    FDT_IMAGE_NODE[compression] = "none"
    FDT_IMAGE_NODE[hash] = "sha256"

    CONF1_CONF_NODE[name] = "conf"
    CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
    CONF1_CONF_NODE[kernel] = "kernel"
    CONF1_CONF_NODE[fdt] = "fdt"

    FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
    FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
    FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_NODE', 'name') or ""}"
    ```
    2. Build the kernel: `bitbake virtual/kernel`
    3. Verify that `image-fit.itb` is present in the build directory: PASS
    4. Disassemble the image using the command: `dtc -I dtb -O dts image-fit.itb`
    5. Verify that the FIT source contains the expected configuration: PASS
    6. Run the unittest using the command: `oe-selftest --run-tests fit_image.FitImage`
    7. Verify that is successfully: PASS
    ```
    2020-05-26 16:54:34,996 - oe-selftest - INFO - SUMMARY:
    2020-05-26 16:54:34,996 - oe-selftest - INFO - oe-selftest () - Ran 13
    tests in 1956.639s
    2020-05-26 16:54:34,997 - oe-selftest - INFO - oe-selftest - OK - All
    required tests passed (successes=13, skipped=0, failures=0, errors=0)
    ```         
 

    Changes since v1:
    ----------------
    - Change the format of short-log to "<target>: <summary>"

    Changes since v2:
    ----------------
    - rename the file from `fit-image` to `fit_image` to successfully export the class functions.
    - adding new sanity checks.
    - add missing dependency.
    - fix a variable reference in a debug log.
    
    Changes since v3:
    ----------------
    - unit-test added
    - class updated to support also properties for U-Boot image

    Changes since v4:
    ----------------
    - remove a wrong patch

    Changes since v5:
    ----------------
    - something went wrong with generation of the patches. regenerate

    Changes since v6:
    ----------------
    - fix the shortlog for one of the patches

Nandor Han (3):
  python-fdt: add a recipe for `python3-fdt` package
  classes: Add a new bbclass that abstracts the generation of FIT blobs
  selftest: add a unit-test for fit-image bbclass

 .../fit-image-test/files/dt-fake.dtb          |   3 +
 .../fit-image-test/files/zImage-fake          |   3 +
 .../fit-image-test/fit-image-test.bb          |  17 +
 meta/classes/fit_image.bbclass                | 387 ++++++++++++++++++
 meta/lib/oeqa/selftest/cases/fit_image.py     | 212 ++++++++++
 .../python/python3-fdt_0.2.0.bb               |  14 +
 6 files changed, 636 insertions(+)
 create mode 100644 meta-selftest/recipes-test/fit-image-test/files/dt-fake.dtb
 create mode 100644 meta-selftest/recipes-test/fit-image-test/files/zImage-fake
 create mode 100644 meta-selftest/recipes-test/fit-image-test/fit-image-test.bb
 create mode 100644 meta/classes/fit_image.bbclass
 create mode 100644 meta/lib/oeqa/selftest/cases/fit_image.py
 create mode 100644 meta/recipes-devtools/python/python3-fdt_0.2.0.bb

-- 
2.24.1

[OE-core][PATCH v7 1/3] python-fdt: add a recipe for `python3-fdt` package

[Nandor Han]

The package `python3-fdt` is used for parsing and generating DTBs.

Signed-off-by: Nandor Han <nandor.han@vaisala.com>
---
 meta/recipes-devtools/python/python3-fdt_0.2.0.bb | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python3-fdt_0.2.0.bb

diff --git a/meta/recipes-devtools/python/python3-fdt_0.2.0.bb b/meta/recipes-devtools/python/python3-fdt_0.2.0.bb
new file mode 100644
index 0000000000..040ceba980
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-fdt_0.2.0.bb
@@ -0,0 +1,14 @@
+SUMMARY = "Library for manipulation of Device Tree Data"
+HOMEPAGE = "https://github.com/molejar/pyFDT"
+SECTION = "devel/python"
+LICENSE = "Apache-2.0"
+LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=9a6ea5b6c346a830f54cc95f6a2a9245"
+
+SRC_URI[md5sum] = "a91daa36b3216f54feeac74ea8e5a475"
+SRC_URI[sha256sum] = "b675139346946115513e27b5eed9aa882628ab74ed500bd5e25e122ee0afa2f6"
+
+PYPI_PACKAGE = "fdt"
+
+inherit pypi setuptools3
+
+BBCLASSEXTEND = "native"
-- 
2.24.1

[OE-core][PATCH v7 2/3] classes: Add a new bbclass that abstracts the generation of FIT blobs

[Nandor Han]

FIT format is very versatile allowing various combination of booting
sequences. In the same time different U-Boot boot stages can use FIT
blobs to pack various binaries (e.g. SPL supports reading U-Boot from a
FIT blob). Because of the allowed level of customization, the generation
of a FIT blob using a fixed image tree source, becomes challenging and
increase the level of complexity where different configurations and
combinations are needed.

This bbclass will know how to generate a FIT blob, leaving the mechanics
of the process (dependencies, task order...) to be handled by the users
of the bbclass. In the same time will allow to separate the knowledge of
the FIT format leaving the user code cleaner and more readable.

Signed-off-by: Nandor Han <nandor.han@vaisala.com>
---
 meta/classes/fit_image.bbclass | 387 +++++++++++++++++++++++++++++++++
 1 file changed, 387 insertions(+)
 create mode 100644 meta/classes/fit_image.bbclass

diff --git a/meta/classes/fit_image.bbclass b/meta/classes/fit_image.bbclass
new file mode 100644
index 0000000000..2d1451020c
--- /dev/null
+++ b/meta/classes/fit_image.bbclass
@@ -0,0 +1,387 @@
+#
+# The class will facilitate the generation of FIT blobs.
+#
+# Glossary
+#    FIT - Flattened uImage Tree
+#
+# Requirements:
+#
+#    * The user need to specify the image content using the format specified in the "FIT Image API" section.
+#
+# FIT Image API
+#
+# The bbclass is using variable and variable flags to declare the FIT image content.
+#
+#    * Sub-Images and Configuration Nodes
+#
+#       ** A sub-image node content is declared using the format `VAR_NODE[<property-name>] = <value>`.
+#         * VAR_NODE - freely selected name of the variable representing a node.
+#         * <property-name> - a sub-image property (e.g. description, type...).
+#         * <value> - the property value.
+#             Depending of the property the value can support different formats.
+#           ** Property Values Formats
+#
+#            string property
+#            ---------------
+#            format: "<text>" - in case the property expects a text.
+#                (e.g. IMAGE_NODE_KERNEL[type] = "kernel")
+#
+#            address property
+#            ----------------
+#            format: "<address>" - in case the property expects an address.
+#                (e.g. IMAGE_NODE_KERNEL[entry] = "0xABCDABCD")
+#
+#            hash property
+#            -------------
+#            format: "<hash type>" - for hash property the hash type needs to be specified.
+#                (e.g. IMAGE_NODE_KERNEL[hash] = "sha256")
+#
+#            sub-image signature property
+#            ----------------------------
+#            format: "<algo>;<key-name-hint>;" - for image signature node.
+#                Both algorithm and key name needs to be provided.
+#                (e.g. IMAGE_NODE_KERNEL[signature] = "sha256,rsa2048;kernel;"
+#
+#            configuration signature property
+#            --------------------------------
+#            format: "<algo>;<key-name-hint>;<sub-image list>" - for configuration signature properties algorithm,
+#                key name and sub-image nodes needs to be provided.
+#                (e.g. CONF_NODE_CONF1[signature] = "sha256,rsa2048;kernel;"kernel","fdt";")
+#
+#       ** Sub-Image and Configuration Nodes Flags
+#              See the code for supported flags.
+#
+#    * FIT_IMAGES_NODE - contains a list of variables used to declare the sub-images nodes, separated by space.
+#                    (e.g. FIT_IMAGES_NODE = "IMAGE_NODE_KERNEL IMAGE_NODE_FDT")
+#
+#    * FIT_CONFIGURATIONS_NODE - contains a list of variables used to declare the configuration nodes,
+#          separated by space. (e.g. FIT_CONFIGURATIONS_NODE = "CONF_NODE_CONF1")
+#        ** Flags
+#           - "default": used to configure the default configuration node.
+#                 (e.g. FIT_CONFIGURATIONS_NODE[default] = "conf@0")
+#
+# Example:
+# This is part of a linux_%.bbappend recipe.
+#
+# KERNEL_IMAGE_NODE[name] = "kernel"
+# KERNEL_IMAGE_NODE[description] = "${PF}"
+# KERNEL_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/zImage")'
+# KERNEL_IMAGE_NODE[type] = "kernel"
+# KERNEL_IMAGE_NODE[arch] = "${ARCH}"
+# KERNEL_IMAGE_NODE[os] = "linux"
+# KERNEL_IMAGE_NODE[compression] = "none"
+# KERNEL_IMAGE_NODE[load] = "${UBOOT_LOADADDRESS}"
+# KERNEL_IMAGE_NODE[entry] = "${UBOOT_ENTRYPOINT}"
+# KERNEL_IMAGE_NODE[hash] = "sha256"
+#
+# FDT_IMAGE_NODE[name] = "fdt"
+# FDT_IMAGE_NODE[description] = "FDT blob"
+# FDT_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/dts/am335x-bone.dtb")'
+# FDT_IMAGE_NODE[type] = "flat_dt"
+# FDT_IMAGE_NODE[arch] = "${ARCH}"
+# FDT_IMAGE_NODE[compression] = "none"
+# FDT_IMAGE_NODE[hash] = "sha256"
+#
+# CONF1_CONF_NODE[name] = "conf"
+# CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
+# CONF1_CONF_NODE[kernel] = "kernel"
+# CONF1_CONF_NODE[fdt] = "fdt"
+#
+# FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
+# FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
+# FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_NODE', 'name') or ""}"
+#
+
+DEPENDS += "\
+    dtc-native \
+    u-boot-mkimage-native \
+"
+
+FIT_IMAGE_DESCRIPTION ??= "Generic FIT image"
+FIT_IMAGE_FILENAME ??= "image-fit"
+FIT_IMAGE_UBOOT_MKIMAGE_OPTS ??= ""
+
+def get_subimage_node_rules():
+    """
+    Defines the properties format and validation for sub-image nodes.
+
+    :return: Return a dictionary with the format rules.
+    """
+
+    #
+    # Rules Format: [Mandatory, Template String, Dictionary keys]
+    # Mandatory: True, False - used to verify if this property is mandatory for generating the image.
+    #            Note: Doesn't take in consideration the conditionally mandatory property.
+    #                  (see: U-Boot/doc/uImage.FIT/source_file_format.txt)
+    # Template String: Property content format. Used to generate the parameter content.
+    # Dictionary Keys: Keys used to be replaced in the Template String.
+    #
+    from collections import OrderedDict
+
+    rules = OrderedDict()
+
+    rules['description'] = [True, '= "$value"', ["value"]]
+    rules['data'] = [True, '= $value', ["value"]]
+    rules['type'] = [True, '= "$value"', ["value"]]
+    rules['arch'] = [False, '= "$value"', ["value"]]
+    rules['os'] = [False, '= "$value"', ["value"]]
+    rules['compression'] = [True, '= "$value"', ["value"]]
+    rules['load'] = [False, '= <$value>', ["value"]]
+    rules['entry'] = [False, '= <$value>', ["value"]]
+    rules['hash'] = [False, '{ algo = "$algo"; }', ['algo']]
+    rules['signature'] = [False, '{ algo = "$algo"; key-name-hint = "$key"; }', ['algo', 'key']]
+
+    return rules
+
+
+def get_conf_node_rules():
+    """
+    Defines the properties format and validation for configuration nodes.
+
+    :return: Return a dictionary with the format rules.
+    """
+    #
+    # Rules Format: [Mandatory, Template String, Dictionary keys]
+    # Mandatory: True, False - used to verify if this property is mandatory for generating the image.
+    #            Note: Doesn't take in consideration the conditionally mandatory property.
+    #                  (see: U-Boot/doc/uImage.FIT/source_file_format.txt)
+    # Template String: Property content format. Used to generate the parameter content.
+    # Dictionary Keys: Keys used to be replaced in the Template String.
+    #
+    from collections import OrderedDict
+
+    rules = OrderedDict()
+
+    rules['description'] = [True, '= "$value"', ["value"]]
+    rules['kernel'] = [False, '= "$value"', ["value"]]
+    rules['ramdisk'] = [False, '= "$value"', ["value"]]
+    rules['firmware'] = [False, '= "$value"', ["value"]]
+    rules['standalone'] = [False, '= "$value"', ["value"]]
+    rules['fdt'] = [False, '= "$value"', ["value"]]
+    rules['loadables'] = [False, '= "$value"', ["value"]]
+    rules['signature'] = [False, '{ algo = "$algo"; key-name-hint = "$key"; sign-images = $images; }',
+                          ['algo', 'key', 'images']]
+
+    return rules
+
+
+def generate_node(name, params, rules):
+    """
+    Generates a node.
+
+    :param name: Node name.
+    :param params: A dictionary containing the properties values.
+    :param rules: A dictionary containing the properties values validation and format.
+
+    :return: A string containing the node, including the new line characters.
+    """
+    from string import Template
+
+    content = []
+
+    for rule in rules.keys():
+        if rule in params.keys():
+            content.append('{param} {value}; '.format(
+                param=rule,
+                value=Template(rules[rule][1]).substitute(
+                    dict(zip(rules[rule][2], params[rule].split(';'))))))
+        elif rules[rule][0]:
+            bb.fatal('Missing mandatory parameter "{param}" from "{section}" section'.format(param=rule, section=name))
+
+    content = """
+                     """.join(content)
+    node = """   {name} {{
+                     {content}
+              }};
+           """.format(name=name, content=content)
+
+    return node
+
+
+def get_section_configuration(var, d):
+    """
+    Generates a string build from variable's flags.
+
+    :param var: variable to extract the flags.
+    :param d: bitbake environment.
+
+    :return: A string with the format '<flag1> = <value1>; <flag2> = <value2>;...'.
+    """
+    flags = d.getVarFlags(var)
+    if flags is not None:
+        flags = dict((flag, d.expand(value))
+                     for flag, value in list(flags.items()))
+    else:
+        flags = {}
+
+    configuration = ''.join((
+              """{name} = "{value}";
+              """.format(name=name, value=value) for name, value in flags.items()))
+
+    return configuration
+
+
+def get_section_properties(var, d):
+    """
+    Extract the nodes and parameters for a section.
+
+    :param var: variable containing the variable names of the nodes that are part of this section.
+    :param d: bitbake environment.
+
+    :return: a list containing dictionaries with section nodes parameters.
+    """
+    nodes = []
+    parameters = {}
+
+    for node in d.getVar(var).split():
+        parameters = d.getVarFlags(node)
+        if parameters is not None:
+            parameters = dict((parameter, d.expand(value))
+                              for parameter, value in list(parameters.items()))
+            nodes.append(parameters)
+
+    return nodes
+
+
+def generate_section(var, rules, d):
+    """
+    Generates a section node (configuration or sub-image).
+
+    :param var: Variable to extract the node names.
+    :param rules: Rules to use for generating this section.
+    :param d: bitbake environment.
+
+    :return: A string containing the section, including the new line characters.
+    """
+
+    section = get_section_configuration(var, d)
+
+    nodes_parameters = get_section_properties(var, d)
+    for parameters in nodes_parameters:
+        try:
+            name = parameters.pop('name')
+            node = generate_node(name, parameters, rules)
+            section += node
+        except KeyError:
+            bb.fatal("Missing name property for node: {node}".format(node=var))
+
+    return section
+
+
+def get_fit_image_template():
+    """
+    Get the FIT format.
+
+    :return: A Template string containing the FIT image format.
+    """
+    from string import Template
+
+    template = Template("""/dts-v1/;
+        /{
+             description = "$description";
+             #address-cells = <1>;
+             images {
+                 $images_section
+             };
+             configurations {
+                 $configurations_section
+             };
+        };""")
+    return template
+
+
+def generate_image_tree_source(d):
+    """
+    Generates a string containing the image tree source.
+
+    :return: A string representing the image tree.
+    """
+    from string import Template
+
+    values = {}
+    values['description'] = d.getVar('FIT_IMAGE_DESCRIPTION')
+
+    image_rules = get_subimage_node_rules()
+    if d.getVar('FIT_IMAGES_NODE', False) is None:
+       bb.fatal("Please add the FIT image nodes to FIT_IMAGES_NODE variable.")
+    values['images_section'] = generate_section('FIT_IMAGES_NODE', image_rules, d)
+
+    conf_rules = get_conf_node_rules()
+    if d.getVar('FIT_CONFIGURATIONS_NODE', False) is None:
+       bb.fatal("Please add the FIT configuration nodes to FIT_CONFIGURATIONS_NODE variable.")
+    values['configurations_section'] = generate_section('FIT_CONFIGURATIONS_NODE', conf_rules, d)
+
+    image_tree_source = get_fit_image_template().substitute(values)
+
+    return image_tree_source
+
+
+def generate_image_blob(file_name, image, d):
+    """
+    Generates a FIT blob.
+
+    :param file_name: FIT blob file name.
+    :param image: String containing the image tree source.
+    :param d: Bitbake environment.
+    """
+    import tempfile
+    import subprocess
+
+    bb.debug(1, "Generated FIT source is:\n {image}".format(image=image))
+
+    builddir = d.getVar('B')
+    blob_file_name = file_name + '.itb'
+
+    try:
+        fd, faux = tempfile.mkstemp(dir=builddir, prefix=file_name, suffix=".its")
+        with os.fdopen(fd, "w") as f:
+            f.write(image)
+
+        mkimage_opts = d.getVar('FIT_IMAGE_UBOOT_MKIMAGE_OPTS').split() or ""
+        cmd = ['mkimage', '-f', faux]
+        cmd.extend(mkimage_opts)
+        cmd.append('{output_name}'.format(output_name=blob_file_name))
+
+        ret = subprocess.run(
+            cmd,
+            check=True,
+            universal_newlines=True,
+            cwd=builddir,
+            stderr=subprocess.STDOUT,
+            stdout=subprocess.PIPE)
+
+        bb.debug(1, "Command for generating the FIT blob is:\n {cmd}".format(cmd=" ".join(ret.args)))
+
+    except subprocess.CalledProcessError as e:
+        bb.fatal('Failed to generate the FIT blob: {message}: {output}'.format(
+            message=str(e), output=e.stdout))
+    finally:
+        os.remove(faux)
+
+
+def generate_fit_image(file_name, d):
+    """
+    Create and generate FIT blob.
+
+    :param file_name: FIT blob file name.
+    :param d: Bitbake environment.
+    """
+    image = generate_image_tree_source(d)
+    generate_image_blob(file_name, image, d)
+
+
+python fit_image_do_generate_fit_image() {
+    generate_fit_image(d.getVar('FIT_IMAGE_FILENAME'), d)
+}
+
+do_generate_fit_image[vardeps] += " \
+    ${FIT_CONFIGURATIONS_NODE} \
+    ${FIT_IMAGES_NODE} \
+    FIT_CONFIGURATIONS_NODE \
+    FIT_IMAGES_NODE \
+    FIT_IMAGE_FILENAME \
+"
+
+addtask do_generate_fit_image after do_compile before do_deploy
+
+EXPORT_FUNCTIONS do_generate_fit_image
-- 
2.24.1

[OE-core][PATCH v7 3/3] selftest: add a unit-test for fit-image bbclass

[Nandor Han]

The unit-test will test the basic functionality of `fit-image.bbclass`.

Signed-off-by: Nandor Han <nandor.han@vaisala.com>
---
 .../fit-image-test/files/dt-fake.dtb          |   3 +
 .../fit-image-test/files/zImage-fake          |   3 +
 .../fit-image-test/fit-image-test.bb          |  17 ++
 meta/lib/oeqa/selftest/cases/fit_image.py     | 212 ++++++++++++++++++
 4 files changed, 235 insertions(+)
 create mode 100644 meta-selftest/recipes-test/fit-image-test/files/dt-fake.dtb
 create mode 100644 meta-selftest/recipes-test/fit-image-test/files/zImage-fake
 create mode 100644 meta-selftest/recipes-test/fit-image-test/fit-image-test.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/fit_image.py

diff --git a/meta-selftest/recipes-test/fit-image-test/files/dt-fake.dtb b/meta-selftest/recipes-test/fit-image-test/files/dt-fake.dtb
new file mode 100644
index 0000000000..7fa871ff00
--- /dev/null
+++ b/meta-selftest/recipes-test/fit-image-test/files/dt-fake.dtb
@@ -0,0 +1,3 @@
+Z�\x1aP�acn\x0f�ZWD1\�\v!΄�l�V\x06
+t(��H�\x15�\x01��z��jb4.���݁�\x15W;I�ƶ����b�e��\x13�Ñ�5�\x03x�\x1d&\x16�9,��g
+�;\x02���!c\x17V
diff --git a/meta-selftest/recipes-test/fit-image-test/files/zImage-fake b/meta-selftest/recipes-test/fit-image-test/files/zImage-fake
new file mode 100644
index 0000000000..7fa871ff00
--- /dev/null
+++ b/meta-selftest/recipes-test/fit-image-test/files/zImage-fake
@@ -0,0 +1,3 @@
+Z�\x1aP�acn\x0f�ZWD1\�\v!΄�l�V\x06
+t(��H�\x15�\x01��z��jb4.���݁�\x15W;I�ƶ����b�e��\x13�Ñ�5�\x03x�\x1d&\x16�9,��g
+�;\x02���!c\x17V
diff --git a/meta-selftest/recipes-test/fit-image-test/fit-image-test.bb b/meta-selftest/recipes-test/fit-image-test/fit-image-test.bb
new file mode 100644
index 0000000000..c7f325ec8a
--- /dev/null
+++ b/meta-selftest/recipes-test/fit-image-test/fit-image-test.bb
@@ -0,0 +1,17 @@
+SUMMARY = "Recipe for testing the fit_image bbclass"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+DEPENDS += "\
+    python3-fdt-native \
+"
+
+SRC_URI = "\
+    file://zImage-fake \
+    file://dt-fake.dtb \
+"
+
+inherit fit_image
+
+include test_recipe.inc
+
diff --git a/meta/lib/oeqa/selftest/cases/fit_image.py b/meta/lib/oeqa/selftest/cases/fit_image.py
new file mode 100644
index 0000000000..866e48b20e
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/fit_image.py
@@ -0,0 +1,212 @@
+import os
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import get_bb_var, bitbake, get_bb_vars
+
+class FitImage(OESelftestTestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        super(FitImage, cls).setUpClass()
+        cls.recipe_name = 'fit-image-test'
+        cls.build_dir = get_bb_var('B', cls.recipe_name)
+        cls.fit_blob_path = os.path.join(cls.build_dir, 'image-fit.itb')
+
+        bitbake('python3-fdt-native')
+        fdt_sitepackage=get_bb_var("S", "python3-fdt-native")
+        os.sys.path.append(fdt_sitepackage)
+        import fdt
+
+        bitbake('{recipe_name} -c cleanall'.format(recipe_name=cls.recipe_name))
+
+    @classmethod
+    def tearDownClass(cls):
+        bitbake('{recipe_name} -c cleanall'.format(recipe_name=cls.recipe_name))
+        super(FitImage, cls).tearDownClass()
+
+    def _get_fit_configuration(self):
+        configuration = """
+KERNEL_IMAGE_NODE[name] = "kernel"
+KERNEL_IMAGE_NODE[description] = "${PF}"
+KERNEL_IMAGE_NODE[data] = '/incbin/("${WORKDIR}/zImage-fake")'
+KERNEL_IMAGE_NODE[type] = "kernel"
+KERNEL_IMAGE_NODE[arch] = "arm"
+KERNEL_IMAGE_NODE[os] = "linux"
+KERNEL_IMAGE_NODE[compression] = "none"
+KERNEL_IMAGE_NODE[load] = "0x84000000"
+KERNEL_IMAGE_NODE[entry] = "0x84000000"
+KERNEL_IMAGE_NODE[hash] = "sha256"
+
+FDT_IMAGE_NODE[name] = "fdt"
+FDT_IMAGE_NODE[description] = "FDT blob"
+FDT_IMAGE_NODE[data] = '/incbin/("${WORKDIR}/dt-fake.dtb")'
+FDT_IMAGE_NODE[type] = "flat_dt"
+FDT_IMAGE_NODE[arch] = "arm"
+FDT_IMAGE_NODE[compression] = "none"
+FDT_IMAGE_NODE[hash] = "sha256"
+
+CONF1_CONF_NODE[name] = "conf"
+CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
+CONF1_CONF_NODE[kernel] = "kernel"
+CONF1_CONF_NODE[fdt] = "fdt"
+
+FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
+FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
+FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_NODE', 'name') or ""}"
+"""
+        return configuration
+
+    def setUp(self):
+        super(FitImage, self).setUp()
+        self.write_recipeinc(self.recipe_name, self._get_fit_configuration())
+
+    def tearDown(self):
+        self.delete_recipeinc(self.recipe_name)
+        super(FitImage, self).tearDown()
+
+    def test_fit_source_is_generated_correctly(self):
+        ret = bitbake("{recipe} -D -f -c generate_fit_image".format(recipe=self.recipe_name)).output
+        self.logger.info('HN {log}'.format(log=ret))
+
+    def test_fit_blob_is_generated_successfully(self):
+        """
+        Summary:     Able to apply a single patch to the Linux kernel source
+        Expected:    The README file should exist and the patch changes should be
+                     displayed at the end of the file.
+        Product:     Kernel Development
+        Author:      Yeoh Ee Peng <ee.peng.yeoh@intel.com>
+        AutomatedBy: Mazliana Mohamad <mazliana.mohamad@intel.com>
+        """
+
+        bitbake("{recipe} -c generate_fit_image".format(recipe=self.recipe_name))
+        self.assertExists(self.fit_blob_path, "FIT Blob not generated")
+
+    def test_that_fit_blob_name_is_configurable(self):
+        """
+        Summary:     Able to apply a single patch to the Linux kernel source
+        Expected:    The README file should exist and the patch changes should be
+                     displayed at the end of the file.
+        """
+        fit_image_name = "custom-fit-blob-name"
+        fit_blob_path = os.path.join(self.build_dir, "{name}.itb".format(name=fit_image_name))
+
+        self.append_recipeinc(self.recipe_name, 'FIT_IMAGE_FILENAME = "{name}"'.format(name=fit_image_name))
+        bitbake("{recipe} -c generate_fit_image".format(recipe=self.recipe_name))
+
+        self.assertExists(fit_blob_path, "FIT Blob not generated")
+
+    def _fail_when_property_missing(self, property):
+        self.remove_recipeinc(self.recipe_name, property)
+        with self.assertRaises(AssertionError):
+            ret = bitbake("{recipe} -f -c generate_fit_image".format(recipe=self.recipe_name))
+
+    def test_fail_when_image_node_name_property_missing(self):
+        self._fail_when_property_missing('KERNEL_IMAGE_NODE[name] = "kernel"')
+
+    def test_fail_when_image_node_description_property_missing(self):
+        self._fail_when_property_missing('KERNEL_IMAGE_NODE[description] = "${PF}"')
+
+    def test_fail_when_image_node_data_property_missing(self):
+        self._fail_when_property_missing('KERNEL_IMAGE_NODE[data] = \'/incbin/("${WORKDIR}/zImage-fake")\'')
+
+    def test_fail_when_image_node_compression_property_missing(self):
+        self._fail_when_property_missing('KERNEL_IMAGE_NODE[type] = "kernel"')
+
+    def test_fail_when_conf_node_name_property_missing(self):
+        self._fail_when_property_missing('CONF1_CONF_NODE[name] = "conf"')
+
+    def test_fail_when_conf_node_description_property_missing(self):
+        self._fail_when_property_missing('CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"')
+
+    def test_that_number_of_nodes_is_correct(self):
+        pass
+
+    def _extract_fit_source(self, text):
+        import re
+        mylines = []
+        copy = False
+
+        for myline in text:
+            if re.match('^DEBUG:.+', myline):
+                copy = False
+
+            if copy:
+                mylines.append(myline.rstrip('\r\n').strip())
+
+            if re.match('^DEBUG:.+do_generate_fit_image: Generated FIT source is:$', myline):
+                copy = True
+
+        sf = ''.join(mylines)
+        self.logger.info('LOG: {log}\nFIT SOURCE {sf}'.format(log=text, sf=sf))
+
+    def _verify_node(self, var_name, path, properties):
+        import fdt
+
+        # I'm creating variables in the recipe that contain the flags value.
+        # I wasn't able to find a better way to read a flag of a variable in the test.
+        for p in properties:
+            self.append_recipeinc(
+                self.recipe_name, 'FIT_IMAGE_TEST_{pu} = "${{@d.getVarFlag(\'{var}\', \'{p}\')}}"'.format(
+                    pu=p.upper(), p=p, var=var_name))
+
+        ret = bitbake("{recipe} -D -c generate_fit_image".format(recipe=self.recipe_name))
+        self._extract_fit_source(ret.output)
+
+        with open(self.fit_blob_path, 'rb') as fb:
+            fit_blob = fb.read()
+
+        dt = fdt.parse_dtb(fit_blob)
+
+        try:
+            node = dt.get_node(path)
+        except ValueError:
+            self.assertTrue(False, '{node} node not generated'.format(node=path))
+
+        node_flags = get_bb_vars(['FIT_IMAGE_TEST_{p}'.format(p=p.upper()) for p in properties],
+            self.recipe_name)
+
+        for p in properties:
+            self.assertTrue(
+                node.exist_property(p),
+                'Missing property {p} from node {n}'.format(p=p.upper(), n=path))
+
+            self.assertTrue(
+                node_flags['FIT_IMAGE_TEST_{p}'.format(p=p.upper())],
+                node.get_property(p))
+
+
+    def _verify_image_node(self, var_name, name, properties):
+        self._verify_node(var_name, 'images/{name}'.format(name=name), properties)
+
+    def _verify_config_node(self, var_name, name, properties):
+        self._verify_node(var_name, 'configurations/{name}'.format(name=name), properties)
+
+    def test_that_kernel_node_is_generated_successfully(self):
+        properties = [
+            'description',
+            'data',
+            'type',
+            'arch',
+            'os',
+            'compression',
+            'load',
+            'entry'
+        ]
+        self._verify_image_node('KERNEL_IMAGE_NODE', 'kernel', properties)
+
+    def test_that_fdt_node_is_generated_successfully(self):
+        properties = [
+            'description',
+            'data',
+            'type',
+            'arch',
+            'compression'
+        ]
+        self._verify_image_node('FDT_IMAGE_NODE', 'fdt', properties)
+
+    def test_that_conf_node_is_generated_successfully(self):
+        properties = [
+            'description',
+            'kernel',
+            'fdt'
+        ]
+        self._verify_config_node('CONF1_CONF_NODE', 'conf', properties)
-- 
2.24.1

✗ patchtest: failure for Add a new bbclass that abstracts the generation of FIT blobs (rev8)

[Patchwork]

== Series Details ==

Series: Add a new bbclass that abstracts the generation of FIT blobs (rev8)
Revision: 8
URL   : https://patchwork.openembedded.org/series/23431/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Errors in your Python code were encountered [test_pylint] 
  Suggested fix    Correct the lines introduced by your patch
  Output           Please, fix the listed issues:
                   meta/lib/oeqa/selftest/cases/fit_image.py does not exist



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for Add a new bbclass that abstracts the generation of FIT blobs (rev8)

[Nandor Han]

On 2020-05-27 09:32, Patchwork wrote:
> == Series Details ==
> 
> Series: Add a new bbclass that abstracts the generation of FIT blobs (rev8)
> Revision: 8
> URL   : https://patchwork.openembedded.org/series/23431/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Issue             Errors in your Python code were encountered [test_pylint]
>    Suggested fix    Correct the lines introduced by your patch
>    Output           Please, fix the listed issues:
>                     meta/lib/oeqa/selftest/cases/fit_image.py does not exist
> 
> 

I don't know how to fix this issue. The file is part of the patchset:

   create mode 100644 meta/lib/oeqa/selftest/cases/fit_image.py

Any suggestions

> 
> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
> 
> ---
> Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
> 

Re: [OE-core][PATCH v7 0/3] Add a new bbclass that abstracts the generation of FIT blobs

[Richard Purdie]

On Wed, 2020-05-27 at 09:05 +0300, Nandor Han wrote:
> Description
> ----------
> Add a new class and unittest for generating FIT blobs.
> 
> 
> Testing
> -------
> 
>     1. linux-yocto_5.4.bbappend was modified to have the following
> configuration:
> 
>     ```
>     inherit fit-image
> 
>     KERNEL_IMAGE_NODE[name] = "kernel"
>     KERNEL_IMAGE_NODE[description] = "${PF}"
>     KERNEL_IMAGE_NODE[data] =
> '/incbin/("./arch/${ARCH}/boot/zImage")'
>     KERNEL_IMAGE_NODE[type] = "kernel"
>     KERNEL_IMAGE_NODE[arch] = "${ARCH}"
>     KERNEL_IMAGE_NODE[os] = "linux"
>     KERNEL_IMAGE_NODE[compression] = "none"
>     KERNEL_IMAGE_NODE[load] = "${UBOOT_LOADADDRESS}"
>     KERNEL_IMAGE_NODE[entry] = "${UBOOT_ENTRYPOINT}"
>     KERNEL_IMAGE_NODE[hash] = "sha256"
> 
>     FDT_IMAGE_NODE[name] = "fdt"
>     FDT_IMAGE_NODE[description] = "FDT blob"
>     FDT_IMAGE_NODE[data] = '/incbin/("./arch/${ARCH}/boot/dts/am335x-
> bone.dtb")'
>     FDT_IMAGE_NODE[type] = "flat_dt"
>     FDT_IMAGE_NODE[arch] = "${ARCH}"
>     FDT_IMAGE_NODE[compression] = "none"
>     FDT_IMAGE_NODE[hash] = "sha256"
> 
>     CONF1_CONF_NODE[name] = "conf"
>     CONF1_CONF_NODE[description] = "Linux kernel and FDT blob"
>     CONF1_CONF_NODE[kernel] = "kernel"
>     CONF1_CONF_NODE[fdt] = "fdt"
> 
>     FIT_IMAGES_NODE = "KERNEL_IMAGE_NODE FDT_IMAGE_NODE"
>     FIT_CONFIGURATIONS_NODE = "CONF1_CONF_NODE"
>     FIT_CONFIGURATIONS_NODE[default] = "${@d.getVarFlag('CONF1_CONF_N
> ODE', 'name') or ""}"
>     ```
>     2. Build the kernel: `bitbake virtual/kernel`
>     3. Verify that `image-fit.itb` is present in the build directory:
> PASS
>     4. Disassemble the image using the command: `dtc -I dtb -O dts
> image-fit.itb`
>     5. Verify that the FIT source contains the expected
> configuration: PASS
>     6. Run the unittest using the command: `oe-selftest --run-tests
> fit_image.FitImage`
>     7. Verify that is successfully: PASS
>     ```
>     2020-05-26 16:54:34,996 - oe-selftest - INFO - SUMMARY:
>     2020-05-26 16:54:34,996 - oe-selftest - INFO - oe-selftest () -
> Ran 13
>     tests in 1956.639s
>     2020-05-26 16:54:34,997 - oe-selftest - INFO - oe-selftest - OK -
> All
>     required tests passed (successes=13, skipped=0, failures=0,
> errors=0)
>     ```         
>  
> 
>     Changes since v1:
>     ----------------
>     - Change the format of short-log to "<target>: <summary>"
> 
>     Changes since v2:
>     ----------------
>     - rename the file from `fit-image` to `fit_image` to successfully
> export the class functions.
>     - adding new sanity checks.
>     - add missing dependency.
>     - fix a variable reference in a debug log.
>     
>     Changes since v3:
>     ----------------
>     - unit-test added
>     - class updated to support also properties for U-Boot image
> 
>     Changes since v4:
>     ----------------
>     - remove a wrong patch
> 
>     Changes since v5:
>     ----------------
>     - something went wrong with generation of the patches. regenerate
> 
>     Changes since v6:
>     ----------------
>     - fix the shortlog for one of the patches
> 
> Nandor Han (3):
>   python-fdt: add a recipe for `python3-fdt` package
>   classes: Add a new bbclass that abstracts the generation of FIT
> blobs
>   selftest: add a unit-test for fit-image bbclass
> 
>  .../fit-image-test/files/dt-fake.dtb          |   3 +
>  .../fit-image-test/files/zImage-fake          |   3 +
>  .../fit-image-test/fit-image-test.bb          |  17 +
>  meta/classes/fit_image.bbclass                | 387
> ++++++++++++++++++
>  meta/lib/oeqa/selftest/cases/fit_image.py     | 212 ++++++++++
>  .../python/python3-fdt_0.2.0.bb               |  14 +
>  6 files changed, 636 insertions(+)
>  create mode 100644 meta-selftest/recipes-test/fit-image-
> test/files/dt-fake.dtb
>  create mode 100644 meta-selftest/recipes-test/fit-image-
> test/files/zImage-fake
>  create mode 100644 meta-selftest/recipes-test/fit-image-test/fit-
> image-test.bb
>  create mode 100644 meta/classes/fit_image.bbclass
>  create mode 100644 meta/lib/oeqa/selftest/cases/fit_image.py
>  create mode 100644 meta/recipes-devtools/python/python3-fdt_0.2.0.bb

This did show up two issues in testing:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1022

(step2d)

Basically there is no maintainer listed for python3-fdt which is an
easy fix and secondly the tests need python3-fdt-native which isn't
present in the test environment.

Cheers,

Richard

Re: [OE-core][PATCH v7 0/3] Add a new bbclass that abstracts the generation of FIT blobs

[Nandor Han]

Hi Richard and thanks for answering.

<snip>


> 
> This did show up two issues in testing:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/1022
> 
> (step2d)
> 
> Basically there is no maintainer listed for python3-fdt which is an
> easy fix and secondly the tests need python3-fdt-native which isn't
> present in the test environment.
> 

Not sure who can be maintainer. I could add myself of course, but I just 
don't want to delay something if I have to react fast and I cannot do it 
at that point. If this is not an issue, I can add myself as maintainer, 
if that's ok with you.


I don't know exactly how to solve the missing `python3-fdt-native` 
issue. I've tried to add it as dependency for my test recipe and locally 
seemed to work fine. I've tried to find documentation and also ask on 
IRC about this issue, but wasn't able to find a clear answer. I'll 
really appreciate some guidance here.

Nandor