[Fuego] Kernel build fixes

[Fuego] Kernel build fixes

[Daniel Sangorrin]

Hi Tim,

Here is a bunch of patches for the fuego-core part
of Functional.kernel_build. I have tested the
default and arm specs.

[PATCH 01/10] kernel_build: remove arm64 build test
[PATCH 02/10] kernel_build: add spec for arm and arm64
[PATCH 03/10] kernel_build: fix regex
[PATCH 04/10] kernel_build: add some pre_checks
[PATCH 05/10] kernel_build: remove paranoic shell checks
[PATCH 06/10] kernel_build: use nproc to guess the number of cores
[PATCH 07/10] kernel_build: put the test in test_run not test_build
[PATCH 08/10] kernel_build: simplify testlog generation
[PATCH 09/10] kernel_build: implement kernel deployment
[PATCH 10/10] kernel_build: add test.yaml

Thanks,
Daniel

[Fuego] [PATCH 01/10] kernel_build: remove arm64 build test

[Daniel Sangorrin]

arm64 is not officially supported by CIP

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/spec.json | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/spec.json b/engine/tests/Functional.kernel_build/spec.json
index 292f648..0b6d20c 100644
--- a/engine/tests/Functional.kernel_build/spec.json
+++ b/engine/tests/Functional.kernel_build/spec.json
@@ -19,13 +19,6 @@
         "cip-x86": {
             "gitrepo": "https://github.com/cip-project/linux-cip.git"
         },
-        "cip-arm64": {
-            "gitrepo": "https://github.com/cip-project/linux-cip.git",
-            "platform": "aarch64",
-            "arch": "arm64",
-            "params": "-j8 Image dtbs modules",
-            "regex_p": "^  OBJCOPY arch/arm64/boot/Image"
-        },
         "template": {
             "gitrepo": "https://xxx/yyy.git",
             "gitref": "my_tag_branch_or_commit_id",
-- 
2.7.4

[Fuego] [PATCH 02/10] kernel_build: add spec for arm and arm64

[Daniel Sangorrin]

I have tested arm(hf) but not arm64 because of the dependency
problem.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/spec.json | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/engine/tests/Functional.kernel_build/spec.json b/engine/tests/Functional.kernel_build/spec.json
index 0b6d20c..df0e748 100644
--- a/engine/tests/Functional.kernel_build/spec.json
+++ b/engine/tests/Functional.kernel_build/spec.json
@@ -4,6 +4,18 @@
         "default": {
             "gitrepo": "https://github.com/torvalds/linux.git"
         },
+        "arm": {
+            "gitrepo": "https://github.com/torvalds/linux.git",
+            "platform": "debian-armhf",
+            "arch": "arm"
+        },
+        "arm64": {
+            "gitrepo": "https://github.com/torvalds/linux.git",
+            "platform": "aarch64",
+            "arch": "arm64",
+            "params": "-j8 Image dtbs modules",
+            "regex_p": "^  OBJCOPY arch/arm64/boot/Image"
+        },
         "lts-414y": {
             "gitrepo": "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git",
             "gitref": "linux-4.14.y"
-- 
2.7.4

[Fuego] [PATCH 03/10] kernel_build: fix regex

[Daniel Sangorrin]

On ARM, the string has some spaces before Kernel

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index daa7d24..d8b3980 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -46,8 +46,8 @@ function test_deploy {
 
 function test_processing {
     echo "Processing kernel build log"
-    if [ -z ${FUNCTIONAL_KERNEL_BUILD_REGEX_P+x} ]; then
-        log_compare "$TESTDIR" "1" "^Kernel: arch/.* is ready" "p"
+    if [ -z "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" ]; then
+        log_compare "$TESTDIR" "1" "[ \t]*Kernel: arch/.* is ready" "p"
     else
         log_compare "$TESTDIR" "1" "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" "p"
     fi
-- 
2.7.4

[Fuego] [PATCH 04/10] kernel_build: add some pre_checks

[Daniel Sangorrin]

- The first precheck is about the node being docker. This can
be hard to understand at first, but this is a build test and
it occurs in docker for now.
- The second list of prechecks are regarding the SDK for the
x86_64 builds. This was pointed out by Tim on 22nov2017.
Sorry for the super slow answer.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index d8b3980..b88d07e 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -3,14 +3,28 @@ FUNCTIONAL_KERNEL_BUILD_PER_JOB_BUILD="true"
 function test_pre_check {
     echo "Doing a pre_check"
     # FIXTHIS: if making uImage, check for mkimage
-}
 
-function test_build {
-    # Configuration
-    if [ -z ${FUNCTIONAL_KERNEL_BUILD_ARCH+x} ]; then
+    if [ "$NODE_NAME" != "docker" ]; then
+        abort_job "This test can only run on docker currently."
+    fi
+
+    if [ -z "$FUNCTIONAL_KERNEL_BUILD_ARCH" ]; then
         FUNCTIONAL_KERNEL_BUILD_ARCH="x86_64"
     fi
 
+    if [ "$FUNCTIONAL_KERNEL_BUILD_ARCH" = "x86_64" ]; then
+        is_on_sdk libelf.a LIBELF /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-linux-*/
+        assert_define LIBELF
+        is_on_sdk libssl.a LIBSSL /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-linux-*/
+        assert_define LIBSSL
+        is_on_sdk bison PROGRAM_BISON /usr/bin/
+        assert_define PROGRAM_BISON
+        is_on_sdk flex PROGRAM_FLEX /usr/bin/
+        assert_define PROGRAM_FLEX
+    fi
+}
+
+function test_build {
     if [ -z ${FUNCTIONAL_KERNEL_BUILD_CONFIG+x} ]; then
         FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
     fi
-- 
2.7.4

[Fuego] [PATCH 05/10] kernel_build: remove paranoic shell checks

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index b88d07e..6090af4 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -25,7 +25,7 @@ function test_pre_check {
 }
 
 function test_build {
-    if [ -z ${FUNCTIONAL_KERNEL_BUILD_CONFIG+x} ]; then
+    if [ -z "$FUNCTIONAL_KERNEL_BUILD_CONFIG" ]; then
         FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
     fi
     echo "Configuring kernel with $FUNCTIONAL_KERNEL_BUILD_CONFIG"
@@ -33,11 +33,11 @@ function test_build {
 
     # Building
     echo "Building Kernel"
-    if [ -z ${FUNCTIONAL_KERNEL_BUILD_PARAMS+x} ]; then
+    if [ -z "$FUNCTIONAL_KERNEL_BUILD_PARAMS" ]; then
         FUNCTIONAL_KERNEL_BUILD_PARAMS="-j4 bzImage modules"
     fi
 
-    if [ ! -z ${FUNCTIONAL_KERNEL_BUILD_PLATFORM+x} ]; then
+    if [ ! -z "$FUNCTIONAL_KERNEL_BUILD_PLATFORM" ]; then
           OLD_PLATFORM=$PLATFORM
           PLATFORM=$FUNCTIONAL_KERNEL_BUILD_PLATFORM
           source $FUEGO_RO/toolchains/tools.sh
-- 
2.7.4

[Fuego] [PATCH 06/10] kernel_build: use nproc to guess the number of cores

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index 6090af4..39e692c 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -34,7 +34,7 @@ function test_build {
     # Building
     echo "Building Kernel"
     if [ -z "$FUNCTIONAL_KERNEL_BUILD_PARAMS" ]; then
-        FUNCTIONAL_KERNEL_BUILD_PARAMS="-j4 bzImage modules"
+        FUNCTIONAL_KERNEL_BUILD_PARAMS="-j$(nproc) bzImage modules"
     fi
 
     if [ ! -z "$FUNCTIONAL_KERNEL_BUILD_PLATFORM" ]; then
-- 
2.7.4

[Fuego] [PATCH 07/10] kernel_build: put the test in test_run not test_build

[Daniel Sangorrin]

This is important because we want to test the kernel incrementally
and without having to set rebuild to tru which would clone
the kernel again.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index 39e692c..af1dd20 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -24,7 +24,15 @@ function test_pre_check {
     fi
 }
 
-function test_build {
+function test_run {
+    # make sure we have the latest source code.
+    cd "$WORKSPACE/$JOB_BUILD_DIR"
+    if [ -d ".git" ]; then
+        git pull || echo "WARNING: git pull failed"
+    else
+        echo "WARNING: no git repository, assuming you used a tarball"
+    fi
+
     if [ -z "$FUNCTIONAL_KERNEL_BUILD_CONFIG" ]; then
         FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
     fi
-- 
2.7.4

[Fuego] [PATCH 08/10] kernel_build: simplify testlog generation

[Daniel Sangorrin]

This works because the build occurs always in docker.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index af1dd20..c914c7a 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -54,12 +54,8 @@ function test_run {
 
     make ARCH=$FUNCTIONAL_KERNEL_BUILD_ARCH $FUNCTIONAL_KERNEL_BUILD_PARAMS 2>&1 | tee build.log || true
 
-    # weird bit here - put the build.log on the board... so that
-    put build.log $BOARD_TESTDIR/fuego.$TESTDIR/build.log
-
     # cat on target populates the testlog with build.log
-    report "cat $BOARD_TESTDIR/fuego.$TESTDIR/build.log"
-}
+    report "cat $WORKSPACE/$JOB_BUILD_DIR/build.log"
 
 function test_deploy {
     echo "Deploying kernel"
-- 
2.7.4

[Fuego] [PATCH 09/10] kernel_build: implement kernel deployment

[Daniel Sangorrin]

This does the next step after building the kernel, deploying
it to a folder (eg tftp server). By default, it uses the
LOGDIR.

FIXTHIS: discuss about whether to export modules and dtbs
as well.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index c914c7a..0a04915 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -57,9 +57,13 @@ function test_run {
     # cat on target populates the testlog with build.log
     report "cat $WORKSPACE/$JOB_BUILD_DIR/build.log"
 
-function test_deploy {
-    echo "Deploying kernel"
-    # FIXTHIS: copy to tftp folder for lava etc..
+    if [ -z "$FUNCTIONAL_KERNEL_DEPLOY" ]; then
+        FUNCTIONAL_KERNEL_DEPLOY="$LOGDIR"
+    fi
+
+    # FIXTHIS: output the modules and dtbs as well
+    echo "Deploying kernel to $FUNCTIONAL_KERNEL_DEPLOY"
+    cp $WORKSPACE/$JOB_BUILD_DIR/arch/$FUNCTIONAL_KERNEL_BUILD_ARCH/boot/*Image $FUNCTIONAL_KERNEL_DEPLOY
 }
 
 function test_processing {
-- 
2.7.4

[Fuego] [PATCH 10/10] kernel_build: add test.yaml

[Daniel Sangorrin]

I have added two more parameters:
- host_dependencies: packages that need to be installed for the build.
Please feel free to change its name (eg sdk_dependencies..)
- default: for optional parameters that have a default value

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/test.yaml | 50 ++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
 create mode 100644 engine/tests/Functional.kernel_build/test.yaml

diff --git a/engine/tests/Functional.kernel_build/test.yaml b/engine/tests/Functional.kernel_build/test.yaml
new file mode 100644
index 0000000..d4922c7
--- /dev/null
+++ b/engine/tests/Functional.kernel_build/test.yaml
@@ -0,0 +1,50 @@
+fuego_package_version: 1
+name: Functional.kernel_build
+description: |
+    Build the Linux kernel using various configurations, architectures and
+    versions. You must add the job to the docker node (e.g. ftc add-job
+    -b docker -t Functional.kernel_build -s arm64) and preinstall the
+    SDK/toolchain requirements in docker.
+license: GPL-2.0
+author: Linus Torvalds and thousands of collaborators
+maintainer: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
+version: 4.x
+fuego_release: 1
+type: Functional
+tags: ['linux', 'build']
+gitrepo: https://github.com/torvalds/linux.git
+host_dependencies: bison flex libelf-dev libssl-dev
+params:
+    - platform:
+        description: sdk/toolchain platform. Check <platform>-tools.sh in fuego-ro/toolchains
+        example: aarch64
+        optional: yes
+        default: docker board file's platform (x86_64)
+    - arch:
+        description: computer architecture
+        example: arm64
+        optional: yes
+        default: x86_64
+    - config:
+        description: configuration file/command
+        example: defconfig
+        optional: yes
+        default: defconfig
+    - params:
+        description: build parameters
+        example: "-j8 uImage"
+        optional: yes
+        default: "-j$(nproc) bzImage modules"
+    - deploy:
+        description: folder to deploy the kernel after build
+        example: "/var/lib/tftp/"
+        optional: yes
+        default: $LOGDIR
+    - regex_p:
+        description: regular expression to confirm a build was succesful
+        example: "^  OBJCOPY arch/arm64/boot/Image"
+        optional: yes
+data_files:
+    - fuego_test.sh
+    - spec.json
+    - test.yaml
-- 
2.7.4

Re: [Fuego] Kernel build fixes

[Tim.Bird]

I reviewed these and they look good.  I have a few questions or comments,
but they are mostly applied without comments.

Look for responses to individual patches that I had comments on.

Thanks!
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Tuesday, March 27, 2018 1:53 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] Kernel build fixes
> 
> Hi Tim,
> 
> Here is a bunch of patches for the fuego-core part
> of Functional.kernel_build. I have tested the
> default and arm specs.
> 
> [PATCH 01/10] kernel_build: remove arm64 build test
> [PATCH 02/10] kernel_build: add spec for arm and arm64
> [PATCH 03/10] kernel_build: fix regex
> [PATCH 04/10] kernel_build: add some pre_checks
> [PATCH 05/10] kernel_build: remove paranoic shell checks
> [PATCH 06/10] kernel_build: use nproc to guess the number of cores
> [PATCH 07/10] kernel_build: put the test in test_run not test_build
> [PATCH 08/10] kernel_build: simplify testlog generation
> [PATCH 09/10] kernel_build: implement kernel deployment
> [PATCH 10/10] kernel_build: add test.yaml
> 
> Thanks,
> Daniel
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 02/10] kernel_build: add spec for arm and arm64

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
>
> I have tested arm(hf) but not arm64 because of the dependency
> problem.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/spec.json | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/engine/tests/Functional.kernel_build/spec.json
> b/engine/tests/Functional.kernel_build/spec.json
> index 0b6d20c..df0e748 100644
> --- a/engine/tests/Functional.kernel_build/spec.json
> +++ b/engine/tests/Functional.kernel_build/spec.json
> @@ -4,6 +4,18 @@
>          "default": {
>              "gitrepo": "https://github.com/torvalds/linux.git"
>          },
> +        "arm": {
> +            "gitrepo": "https://github.com/torvalds/linux.git",
> +            "platform": "debian-armhf",
> +            "arch": "arm"
> +        },
> +        "arm64": {
> +            "gitrepo": "https://github.com/torvalds/linux.git",
> +            "platform": "aarch64",
> +            "arch": "arm64",
> +            "params": "-j8 Image dtbs modules",
> +            "regex_p": "^  OBJCOPY arch/arm64/boot/Image"
I was wondering what regex_p was for,  but luckily your last patch
with the test.yaml file including the parameter description made
it clear.  Thanks!
 -- Tim

> +        },
>          "lts-414y": {
>              "gitrepo":
> "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git",
>              "gitref": "linux-4.14.y"
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 04/10] kernel_build: add some pre_checks

[Tim.Bird]

OK - this is a bit different.

So if I understand correctly, you would have jobs like the following:
docker.arm.Functional.kernel_build
and
docker.arm64.Functional.kernel_build

and something like:
beaglebone.arm.Functional.kernel_build
is not allowed.  That seems strange to me.

Why not just always build inside the container, but ignore the
node name?  Do you intend to ultimately support cross-building
from other architectures?

See more questions below.


> -----Original Message-----
> From: Daniel Sangorrin
> 
> - The first precheck is about the node being docker. This can
> be hard to understand at first, but this is a build test and
> it occurs in docker for now.
> - The second list of prechecks are regarding the SDK for the
> x86_64 builds. This was pointed out by Tim on 22nov2017.
> Sorry for the super slow answer.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/fuego_test.sh | 22
> ++++++++++++++++++----
>  1 file changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> b/engine/tests/Functional.kernel_build/fuego_test.sh
> index d8b3980..b88d07e 100755
> --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> @@ -3,14 +3,28 @@ FUNCTIONAL_KERNEL_BUILD_PER_JOB_BUILD="true"
>  function test_pre_check {
>      echo "Doing a pre_check"
>      # FIXTHIS: if making uImage, check for mkimage
> -}
> 
> -function test_build {
> -    # Configuration
> -    if [ -z ${FUNCTIONAL_KERNEL_BUILD_ARCH+x} ]; then
> +    if [ "$NODE_NAME" != "docker" ]; then
> +        abort_job "This test can only run on docker currently."
> +    fi
> +
> +    if [ -z "$FUNCTIONAL_KERNEL_BUILD_ARCH" ]; then
>          FUNCTIONAL_KERNEL_BUILD_ARCH="x86_64"
>      fi
> 
> +    if [ "$FUNCTIONAL_KERNEL_BUILD_ARCH" = "x86_64" ]; then
> +        is_on_sdk libelf.a LIBELF /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> linux-*/
> +        assert_define LIBELF
> +        is_on_sdk libssl.a LIBSSL /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> linux-*/
FUNCTIONAL_KERNEL_BUILD_ARCH is always x86_64, so why use a variable
here?  I think the line reads easier without the variable.

Is /usr/lib/arm-linux-*/libssl.a required for building the ARM kernel?
That seems odd.  Does that code end up in the kernel image or in some
tool?  If it's a tool, are you sure it isn't a host-side tool, that should
be x86_64, and not $FUNCTIONAL_KERNEL)_BUILD_ARCH?

I'm a bit confused by this.

> +        assert_define LIBSSL
> +        is_on_sdk bison PROGRAM_BISON /usr/bin/
> +        assert_define PROGRAM_BISON
> +        is_on_sdk flex PROGRAM_FLEX /usr/bin/
These are obviously generic, and the same for any kernel.

> +        assert_define PROGRAM_FLEX
> +    fi
> +}
> +
> +function test_build {
>      if [ -z ${FUNCTIONAL_KERNEL_BUILD_CONFIG+x} ]; then
>          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
>      fi
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 05/10] kernel_build: remove paranoic shell checks

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/fuego_test.sh | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> b/engine/tests/Functional.kernel_build/fuego_test.sh
> index b88d07e..6090af4 100755
> --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> @@ -25,7 +25,7 @@ function test_pre_check {
>  }
> 
>  function test_build {
> -    if [ -z ${FUNCTIONAL_KERNEL_BUILD_CONFIG+x} ]; then
> +    if [ -z "$FUNCTIONAL_KERNEL_BUILD_CONFIG" ]; then
>          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
>      fi
>      echo "Configuring kernel with $FUNCTIONAL_KERNEL_BUILD_CONFIG"
> @@ -33,11 +33,11 @@ function test_build {
> 
>      # Building
>      echo "Building Kernel"
> -    if [ -z ${FUNCTIONAL_KERNEL_BUILD_PARAMS+x} ]; then
> +    if [ -z "$FUNCTIONAL_KERNEL_BUILD_PARAMS" ]; then
>          FUNCTIONAL_KERNEL_BUILD_PARAMS="-j4 bzImage modules"
>      fi
> 
> -    if [ ! -z ${FUNCTIONAL_KERNEL_BUILD_PLATFORM+x} ]; then
> +    if [ ! -z "$FUNCTIONAL_KERNEL_BUILD_PLATFORM" ]; then
Thanks.  I wasn't a big fan of these old-style shell expansions $(foo+x).
This is easier to read.
 -- Tim

>            OLD_PLATFORM=$PLATFORM
>            PLATFORM=$FUNCTIONAL_KERNEL_BUILD_PLATFORM
>            source $FUEGO_RO/toolchains/tools.sh
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 06/10] kernel_build: use nproc to guess the number of cores

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/fuego_test.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> b/engine/tests/Functional.kernel_build/fuego_test.sh
> index 6090af4..39e692c 100755
> --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> @@ -34,7 +34,7 @@ function test_build {
>      # Building
>      echo "Building Kernel"
>      if [ -z "$FUNCTIONAL_KERNEL_BUILD_PARAMS" ]; then
> -        FUNCTIONAL_KERNEL_BUILD_PARAMS="-j4 bzImage modules"
> +        FUNCTIONAL_KERNEL_BUILD_PARAMS="-j$(nproc) bzImage modules"

I thought nproc was a shell variable, and I was going to ask where it came from,
until I read this closer.

I didn't even know that the utility program 'nproc' existed.  Hmmm.  You learn
something new every day.

Nice enhancement.  Thanks,
 -- Tim



>      fi
> 
>      if [ ! -z "$FUNCTIONAL_KERNEL_BUILD_PLATFORM" ]; then
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 09/10] kernel_build: implement kernel deployment

[Tim.Bird]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Tuesday, March 27, 2018 1:53 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 09/10] kernel_build: implement kernel deployment
> 
> This does the next step after building the kernel, deploying
> it to a folder (eg tftp server). By default, it uses the
> LOGDIR.
> 
> FIXTHIS: discuss about whether to export modules and dtbs
> as well.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/fuego_test.sh | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> b/engine/tests/Functional.kernel_build/fuego_test.sh
> index c914c7a..0a04915 100755
> --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> @@ -57,9 +57,13 @@ function test_run {
>      # cat on target populates the testlog with build.log
>      report "cat $WORKSPACE/$JOB_BUILD_DIR/build.log"
> 
> -function test_deploy {
> -    echo "Deploying kernel"
> -    # FIXTHIS: copy to tftp folder for lava etc..
> +    if [ -z "$FUNCTIONAL_KERNEL_DEPLOY" ]; then
> +        FUNCTIONAL_KERNEL_DEPLOY="$LOGDIR"
> +    fi
> +
> +    # FIXTHIS: output the modules and dtbs as well
> +    echo "Deploying kernel to $FUNCTIONAL_KERNEL_DEPLOY"
> +    cp
> $WORKSPACE/$JOB_BUILD_DIR/arch/$FUNCTIONAL_KERNEL_BUILD_ARCH/
> boot/*Image $FUNCTIONAL_KERNEL_DEPLOY

I reserve the right to come back and adjust this step when we add some
generalized board provisioning in the future.  For now, looks good.
 -- Tim

>  }
> 
>  function test_processing {
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 10/10] kernel_build: add test.yaml

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> I have added two more parameters:
> - host_dependencies: packages that need to be installed for the build.
> Please feel free to change its name (eg sdk_dependencies..)
> - default: for optional parameters that have a default value
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/test.yaml | 50
> ++++++++++++++++++++++++++
>  1 file changed, 50 insertions(+)
>  create mode 100644 engine/tests/Functional.kernel_build/test.yaml
> 
> diff --git a/engine/tests/Functional.kernel_build/test.yaml
> b/engine/tests/Functional.kernel_build/test.yaml
> new file mode 100644
> index 0000000..d4922c7
> --- /dev/null
> +++ b/engine/tests/Functional.kernel_build/test.yaml
> @@ -0,0 +1,50 @@
> +fuego_package_version: 1
> +name: Functional.kernel_build
> +description: |
> +    Build the Linux kernel using various configurations, architectures and
> +    versions. You must add the job to the docker node (e.g. ftc add-job
> +    -b docker -t Functional.kernel_build -s arm64) and preinstall the
> +    SDK/toolchain requirements in docker.
> +license: GPL-2.0
> +author: Linus Torvalds and thousands of collaborators
> +maintainer: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> +version: 4.x
> +fuego_release: 1
> +type: Functional
> +tags: ['linux', 'build']
> +gitrepo: https://github.com/torvalds/linux.git
> +host_dependencies: bison flex libelf-dev libssl-dev
Looks OK.  I think I'd remove bison and flex.   Those are,
IMHO, standard tools like make, ar, git, that should always
be sent in the Fuego host distribution.

host_dependencies is an OK name.

 Thanks,
 -- Tim

> +params:
> +    - platform:
> +        description: sdk/toolchain platform. Check <platform>-tools.sh in
> fuego-ro/toolchains
> +        example: aarch64
> +        optional: yes
> +        default: docker board file's platform (x86_64)
> +    - arch:
> +        description: computer architecture
> +        example: arm64
> +        optional: yes
> +        default: x86_64
> +    - config:
> +        description: configuration file/command
> +        example: defconfig
> +        optional: yes
> +        default: defconfig
> +    - params:
> +        description: build parameters
> +        example: "-j8 uImage"
> +        optional: yes
> +        default: "-j$(nproc) bzImage modules"
> +    - deploy:
> +        description: folder to deploy the kernel after build
> +        example: "/var/lib/tftp/"
> +        optional: yes
> +        default: $LOGDIR
> +    - regex_p:
> +        description: regular expression to confirm a build was succesful
> +        example: "^  OBJCOPY arch/arm64/boot/Image"
> +        optional: yes
> +data_files:
> +    - fuego_test.sh
> +    - spec.json
> +    - test.yaml
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 04/10] kernel_build: add some pre_checks

[Daniel Sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, March 28, 2018 6:42 AM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 04/10] kernel_build: add some pre_checks
> 
> OK - this is a bit different.
> 
> So if I understand correctly, you would have jobs like the following:
> docker.arm.Functional.kernel_build
> and
> docker.arm64.Functional.kernel_build
> 
> and something like:
> beaglebone.arm.Functional.kernel_build
> is not allowed.  That seems strange to me.
> 
> Why not just always build inside the container, but ignore the
> node name?  

The reason is because Fuego will try to ssh into the node name and
run the test there. Why don't want to build the kernel in the node,
we want to build it in docker. I know why you are confused, but please
notice that test_build here only _clones_ the initial source code, it does
not build anything. The real test (building the kernel) is done in "test_run"
which needs to run in docker, not in other nodes.

Previously I put everything in test_build, but that wasn't a good idea:
1) if the first time the build failed, the second time we would have to 
     clone the kernel from scratch again (slow).
2) if the first time the build was succesful, the second time it would skip
     the would phase and would do nothing. An we wouldn't be able to test
     the kernel incrementally either.

> Do you intend to ultimately support cross-building
> from other architectures?

No, that is too much I think.

> See more questions below.
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > - The first precheck is about the node being docker. This can
> > be hard to understand at first, but this is a build test and
> > it occurs in docker for now.
> > - The second list of prechecks are regarding the SDK for the
> > x86_64 builds. This was pointed out by Tim on 22nov2017.
> > Sorry for the super slow answer.
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/tests/Functional.kernel_build/fuego_test.sh | 22
> > ++++++++++++++++++----
> >  1 file changed, 18 insertions(+), 4 deletions(-)
> >
> > diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> > b/engine/tests/Functional.kernel_build/fuego_test.sh
> > index d8b3980..b88d07e 100755
> > --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> > +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> > @@ -3,14 +3,28 @@ FUNCTIONAL_KERNEL_BUILD_PER_JOB_BUILD="true"
> >  function test_pre_check {
> >      echo "Doing a pre_check"
> >      # FIXTHIS: if making uImage, check for mkimage
> > -}
> >
> > -function test_build {
> > -    # Configuration
> > -    if [ -z ${FUNCTIONAL_KERNEL_BUILD_ARCH+x} ]; then
> > +    if [ "$NODE_NAME" != "docker" ]; then
> > +        abort_job "This test can only run on docker currently."
> > +    fi
> > +
> > +    if [ -z "$FUNCTIONAL_KERNEL_BUILD_ARCH" ]; then
> >          FUNCTIONAL_KERNEL_BUILD_ARCH="x86_64"
> >      fi
> >
> > +    if [ "$FUNCTIONAL_KERNEL_BUILD_ARCH" = "x86_64" ]; then
> > +        is_on_sdk libelf.a LIBELF /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> > linux-*/
> > +        assert_define LIBELF
> > +        is_on_sdk libssl.a LIBSSL /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> > linux-*/
> FUNCTIONAL_KERNEL_BUILD_ARCH is always x86_64, so why use a variable
> here?  I think the line reads easier without the variable.

No, ARCH is the variable that goes in the kernel compilation line (eg: make ARCH=arm CROSS_COMPILE=... uImage).
So it is not always x86_64. You can see in the spec.json.
 
> Is /usr/lib/arm-linux-*/libssl.a required for building the ARM kernel?

It is not, that is why the is_on_sdk are only checked for x86_64.

> That seems odd.  Does that code end up in the kernel image or in some
> tool?  If it's a tool, are you sure it isn't a host-side tool, that should
> be x86_64, and not $FUNCTIONAL_KERNEL)_BUILD_ARCH?
> 
> I'm a bit confused by this.
> 
> > +        assert_define LIBSSL
> > +        is_on_sdk bison PROGRAM_BISON /usr/bin/
> > +        assert_define PROGRAM_BISON
> > +        is_on_sdk flex PROGRAM_FLEX /usr/bin/
> These are obviously generic, and the same for any kernel.

I will check if they are actually needed for arm. Here i am only
checking for x86_64.

Thanks,
Daniel

> 
> > +        assert_define PROGRAM_FLEX
> > +    fi
> > +}
> > +
> > +function test_build {
> >      if [ -z ${FUNCTIONAL_KERNEL_BUILD_CONFIG+x} ]; then
> >          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
> >      fi
> > --
> > 2.7.4
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 09/10] kernel_build: implement kernel deployment

[Daniel Sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, March 28, 2018 6:46 AM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 09/10] kernel_build: implement kernel deployment
> 
> 
> 
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> > bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> > Sent: Tuesday, March 27, 2018 1:53 AM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH 09/10] kernel_build: implement kernel deployment
> >
> > This does the next step after building the kernel, deploying
> > it to a folder (eg tftp server). By default, it uses the
> > LOGDIR.
> >
> > FIXTHIS: discuss about whether to export modules and dtbs
> > as well.
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/tests/Functional.kernel_build/fuego_test.sh | 10 +++++++---
> >  1 file changed, 7 insertions(+), 3 deletions(-)
> >
> > diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> > b/engine/tests/Functional.kernel_build/fuego_test.sh
> > index c914c7a..0a04915 100755
> > --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> > +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> > @@ -57,9 +57,13 @@ function test_run {
> >      # cat on target populates the testlog with build.log
> >      report "cat $WORKSPACE/$JOB_BUILD_DIR/build.log"
> >
> > -function test_deploy {
> > -    echo "Deploying kernel"
> > -    # FIXTHIS: copy to tftp folder for lava etc..
> > +    if [ -z "$FUNCTIONAL_KERNEL_DEPLOY" ]; then
> > +        FUNCTIONAL_KERNEL_DEPLOY="$LOGDIR"
> > +    fi
> > +
> > +    # FIXTHIS: output the modules and dtbs as well
> > +    echo "Deploying kernel to $FUNCTIONAL_KERNEL_DEPLOY"
> > +    cp
> > $WORKSPACE/$JOB_BUILD_DIR/arch/$FUNCTIONAL_KERNEL_BUILD_ARCH/
> > boot/*Image $FUNCTIONAL_KERNEL_DEPLOY
> 
> I reserve the right to come back and adjust this step when we add some
> generalized board provisioning in the future.  For now, looks good.

Yes, I am thinking about providing a ktest.pl like approach where we can
specify in the spec the board to install the kernel. Then docker would ssh into
the board, install the kernel/modules and reboot. The current approach
is passive. We just put the file in the tftp folder and we assume that
the board will be powered up and provisioned with the kernel in the tftp folder
(e.g.: using u-boot's tftpboot command).

Thanks,
Daniel

Re: [Fuego] [PATCH 04/10] kernel_build: add some pre_checks

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> 
> > -----Original Message-----
> > From: Tim Bird
> >
> > OK - this is a bit different.
> >
> > So if I understand correctly, you would have jobs like the following:
> > docker.arm.Functional.kernel_build
> > and
> > docker.arm64.Functional.kernel_build
> >
> > and something like:
> > beaglebone.arm.Functional.kernel_build
> > is not allowed.  That seems strange to me.
> >
> > Why not just always build inside the container, but ignore the
> > node name?
> 
> The reason is because Fuego will try to ssh into the node name and
> run the test there.
You can run all kinds of host-side stuff in test_run(), without contacting
the client board.  It runs in the docker container.

I have several tests that only use the target to reflect the testlog back to
the fuego system.  This is a bit awkward, and I've been thinking of adding
support for host-side-only test.  (See Functional.fuego_lint for an example).

There's nothing in Fuego that requires anything to execute on the target board
(with the exception of this weird log bouncing).

Now, there's an issue with Fuego's pre_test phase contacting the target board
to gather information (doing things like ps, free, mount, cat /proc/interrupts, etc.)
(that is, stuff from ov_rootfs_state()).  It would be nice to avoid that, or even checking
if the target is alive, when we're not going to actually communicate with the board
for the test.


> Why don't want to build the kernel in the node,
> we want to build it in docker
Agreed.  It's built in the fuego container, but is not "running" on the
docker virtual test node (at least in my mind).

> I know why you are confused, but please
> notice that test_build here only _clones_ the initial source code, it does
> not build anything. The real test (building the kernel) is done in "test_run"
> which needs to run in docker, not in other nodes.
> 
> Previously I put everything in test_build, but that wasn't a good idea:
> 1) if the first time the build failed, the second time we would have to
>      clone the kernel from scratch again (slow).
> 2) if the first time the build was succesful, the second time it would skip
>      the would phase and would do nothing. An we wouldn't be able to test
>      the kernel incrementally either.

I think putting the actual kernel compilation into test_run is the correct
thing.  I just think we should find a way to avoid doing it in the overall
context of the docker test node.

OK - let me think about these problems.
 
> > Do you intend to ultimately support cross-building
> > from other architectures?
> 
> No, that is too much I think.
OK, agreed.
> 
> > See more questions below.
> >
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin
> > >
> > > - The first precheck is about the node being docker. This can
> > > be hard to understand at first, but this is a build test and
> > > it occurs in docker for now.
> > > - The second list of prechecks are regarding the SDK for the
> > > x86_64 builds. This was pointed out by Tim on 22nov2017.
> > > Sorry for the super slow answer.
> > >
> > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > ---
> > >  engine/tests/Functional.kernel_build/fuego_test.sh | 22
> > > ++++++++++++++++++----
> > >  1 file changed, 18 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> > > b/engine/tests/Functional.kernel_build/fuego_test.sh
> > > index d8b3980..b88d07e 100755
> > > --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> > > +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> > > @@ -3,14 +3,28 @@
> FUNCTIONAL_KERNEL_BUILD_PER_JOB_BUILD="true"
> > >  function test_pre_check {
> > >      echo "Doing a pre_check"
> > >      # FIXTHIS: if making uImage, check for mkimage
> > > -}
> > >
> > > -function test_build {
> > > -    # Configuration
> > > -    if [ -z ${FUNCTIONAL_KERNEL_BUILD_ARCH+x} ]; then
> > > +    if [ "$NODE_NAME" != "docker" ]; then
> > > +        abort_job "This test can only run on docker currently."
> > > +    fi
> > > +
> > > +    if [ -z "$FUNCTIONAL_KERNEL_BUILD_ARCH" ]; then
> > >          FUNCTIONAL_KERNEL_BUILD_ARCH="x86_64"
> > >      fi
> > >
> > > +    if [ "$FUNCTIONAL_KERNEL_BUILD_ARCH" = "x86_64" ]; then
> > > +        is_on_sdk libelf.a LIBELF
> /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> > > linux-*/
> > > +        assert_define LIBELF
> > > +        is_on_sdk libssl.a LIBSSL
> /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> > > linux-*/
> > FUNCTIONAL_KERNEL_BUILD_ARCH is always x86_64, so why use a
> variable
> > here?  I think the line reads easier without the variable.
> 
> No, ARCH is the variable that goes in the kernel compilation line (eg: make
> ARCH=arm CROSS_COMPILE=... uImage).
> So it is not always x86_64. You can see in the spec.json.
It is inside an if statement that requires it to be x86_64.  So, no matter
what's in the spec.json, for these uses of the variable it will always
have the value 'x86_64'.

> 
> > Is /usr/lib/arm-linux-*/libssl.a required for building the ARM kernel?
> 
> It is not, that is why the is_on_sdk are only checked for x86_64.
I'm curious what libelf.a and  libssl.a are actually used for in the kernel
source.  The kernel code itself is self-contained, so I suspect these
are being used to build compilation helper tools or some other host-side
tools, and should always be x86_64.

> 
> > That seems odd.  Does that code end up in the kernel image or in some
> > tool?  If it's a tool, are you sure it isn't a host-side tool, that should
> > be x86_64, and not $FUNCTIONAL_KERNEL)_BUILD_ARCH?
> >
> > I'm a bit confused by this.
> >
> > > +        assert_define LIBSSL
> > > +        is_on_sdk bison PROGRAM_BISON /usr/bin/
> > > +        assert_define PROGRAM_BISON
> > > +        is_on_sdk flex PROGRAM_FLEX /usr/bin/
> > These are obviously generic, and the same for any kernel.
> 
> I will check if they are actually needed for arm. Here i am only
> checking for x86_64.

I'm pretty sure they are needed for any architecture.  There are
lex and yacc files in the kernel source tree.  (bison and flex
are the gnu replacements).
 tlinux:~/work/torvalds/linux$ find . -name "*.y"
./tools/bpf/bpf_exp.y
./tools/perf/util/pmu.y
./tools/perf/util/expr.y
./tools/perf/util/parse-events.y
./scripts/genksyms/parse.y
./scripts/kconfig/zconf.y
./scripts/dtc/dtc-parser.y
./drivers/scsi/aic7xxx/aicasm/aicasm_gram.y
./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y
tlinux:~/work/torvalds/linux$ find . -name "*.l"
./tools/bpf/bpf_exp.l
./tools/perf/util/pmu.l
./tools/perf/util/parse-events.l
./scripts/genksyms/lex.l
./scripts/kconfig/zconf.l
./scripts/dtc/dtc-lexer.l
./drivers/scsi/aic7xxx/aicasm/aicasm_macro_scan.l
./drivers/scsi/aic7xxx/aicasm/aicasm_scan.l

Most of these are for host-side tools (compilation aids, but
some can end up in the kernel on the target.  But the lex
and yacc tools produce parser code that should be architecture
neutral (and is later compiled by the cross-toolchain for
the correct architecture).

 -- Tim

> 
> Thanks,
> Daniel
> 
> >
> > > +        assert_define PROGRAM_FLEX
> > > +    fi
> > > +}
> > > +
> > > +function test_build {
> > >      if [ -z ${FUNCTIONAL_KERNEL_BUILD_CONFIG+x} ]; then
> > >          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
> > >      fi
> > > --
> > > 2.7.4
> > >
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 04/10] kernel_build: add some pre_checks

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, March 28, 2018 10:31 AM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 04/10] kernel_build: add some pre_checks
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > > -----Original Message-----
> > > From: Tim Bird
> > >
> > > OK - this is a bit different.
> > >
> > > So if I understand correctly, you would have jobs like the following:
> > > docker.arm.Functional.kernel_build
> > > and
> > > docker.arm64.Functional.kernel_build
> > >
> > > and something like:
> > > beaglebone.arm.Functional.kernel_build
> > > is not allowed.  That seems strange to me.
> > >
> > > Why not just always build inside the container, but ignore the
> > > node name?
> >
> > The reason is because Fuego will try to ssh into the node name and
> > run the test there.
> You can run all kinds of host-side stuff in test_run(), without contacting
> the client board.  It runs in the docker container.
> 
> I have several tests that only use the target to reflect the testlog back to
> the fuego system.  This is a bit awkward, and I've been thinking of adding
> support for host-side-only test.  (See Functional.fuego_lint for an example).
> 
> There's nothing in Fuego that requires anything to execute on the target board
> (with the exception of this weird log bouncing).
> 
> Now, there's an issue with Fuego's pre_test phase contacting the target board
> to gather information (doing things like ps, free, mount, cat /proc/interrupts, etc.)
> (that is, stuff from ov_rootfs_state()).  It would be nice to avoid that, or even checking
> if the target is alive, when we're not going to actually communicate with the board
> for the test.

Yes, this is what I mean. If we use the node name (e.g. bbb), we would need to have 
the corresponding board ready just for building the kernel even though it has nothing
to do with it.  

> > Why don't want to build the kernel in the node,
> > we want to build it in docker
> Agreed.  It's built in the fuego container, but is not "running" on the
> docker virtual test node (at least in my mind).

I took this abstraction because in the next step I wanted to be able to specify different
build nodes (I will have to make some changes for this to work). In other words, I would like
to have a pool of "build" nodes (e.g. spare PCs, cloud virtual machines) that will do the hard work
of building the kernel for me. Those nodes should have their toolchains ready of course.

> > I know why you are confused, but please
> > notice that test_build here only _clones_ the initial source code, it does
> > not build anything. The real test (building the kernel) is done in "test_run"
> > which needs to run in docker, not in other nodes.
> >
> > Previously I put everything in test_build, but that wasn't a good idea:
> > 1) if the first time the build failed, the second time we would have to
> >      clone the kernel from scratch again (slow).
> > 2) if the first time the build was succesful, the second time it would skip
> >      the would phase and would do nothing. An we wouldn't be able to test
> >      the kernel incrementally either.
> 
> I think putting the actual kernel compilation into test_run is the correct
> thing.  I just think we should find a way to avoid doing it in the overall
> context of the docker test node.
> 
> OK - let me think about these problems.

I see what you mean. But please consider what I wrote above about
"build nodes". This will be needed for yocto builds as well in AGL for example.

> > > Do you intend to ultimately support cross-building
> > > from other architectures?
> >
> > No, that is too much I think.
> OK, agreed.
> >
> > > See more questions below.
> > >
> > >
> > > > -----Original Message-----
> > > > From: Daniel Sangorrin
> > > >
> > > > - The first precheck is about the node being docker. This can
> > > > be hard to understand at first, but this is a build test and
> > > > it occurs in docker for now.
> > > > - The second list of prechecks are regarding the SDK for the
> > > > x86_64 builds. This was pointed out by Tim on 22nov2017.
> > > > Sorry for the super slow answer.
> > > >
> > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > > ---
> > > >  engine/tests/Functional.kernel_build/fuego_test.sh | 22
> > > > ++++++++++++++++++----
> > > >  1 file changed, 18 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> > > > b/engine/tests/Functional.kernel_build/fuego_test.sh
> > > > index d8b3980..b88d07e 100755
> > > > --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> > > > +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> > > > @@ -3,14 +3,28 @@
> > FUNCTIONAL_KERNEL_BUILD_PER_JOB_BUILD="true"
> > > >  function test_pre_check {
> > > >      echo "Doing a pre_check"
> > > >      # FIXTHIS: if making uImage, check for mkimage
> > > > -}
> > > >
> > > > -function test_build {
> > > > -    # Configuration
> > > > -    if [ -z ${FUNCTIONAL_KERNEL_BUILD_ARCH+x} ]; then
> > > > +    if [ "$NODE_NAME" != "docker" ]; then
> > > > +        abort_job "This test can only run on docker currently."
> > > > +    fi
> > > > +
> > > > +    if [ -z "$FUNCTIONAL_KERNEL_BUILD_ARCH" ]; then
> > > >          FUNCTIONAL_KERNEL_BUILD_ARCH="x86_64"
> > > >      fi
> > > >
> > > > +    if [ "$FUNCTIONAL_KERNEL_BUILD_ARCH" = "x86_64" ]; then
> > > > +        is_on_sdk libelf.a LIBELF
> > /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> > > > linux-*/
> > > > +        assert_define LIBELF
> > > > +        is_on_sdk libssl.a LIBSSL
> > /usr/lib/$FUNCTIONAL_KERNEL_BUILD_ARCH-
> > > > linux-*/
> > > FUNCTIONAL_KERNEL_BUILD_ARCH is always x86_64, so why use a
> > variable
> > > here?  I think the line reads easier without the variable.
> >
> > No, ARCH is the variable that goes in the kernel compilation line (eg: make
> > ARCH=arm CROSS_COMPILE=... uImage).
> > So it is not always x86_64. You can see in the spec.json.
> It is inside an if statement that requires it to be x86_64.  So, no matter
> what's in the spec.json, for these uses of the variable it will always
> have the value 'x86_64'.

Oh, sorry. I understand what you mean now. I was using it just in case we need
to support the same checks for more architectures (eg aarch64) that I haven't
tested yet.

> > > Is /usr/lib/arm-linux-*/libssl.a required for building the ARM kernel?
> >
> > It is not, that is why the is_on_sdk are only checked for x86_64.
> I'm curious what libelf.a and  libssl.a are actually used for in the kernel
> source.  The kernel code itself is self-contained, so I suspect these
> are being used to build compilation helper tools or some other host-side
> tools, and should always be x86_64.

I think ssl was needed for module signing. libelf is something new for me. It seems
it is needed for perf when you don't cross-compile, and maybe other cases like CONFIG_UNWINDER_ORC.

> 
> >
> > > That seems odd.  Does that code end up in the kernel image or in some
> > > tool?  If it's a tool, are you sure it isn't a host-side tool, that should
> > > be x86_64, and not $FUNCTIONAL_KERNEL)_BUILD_ARCH?
> > >
> > > I'm a bit confused by this.
> > >
> > > > +        assert_define LIBSSL
> > > > +        is_on_sdk bison PROGRAM_BISON /usr/bin/
> > > > +        assert_define PROGRAM_BISON
> > > > +        is_on_sdk flex PROGRAM_FLEX /usr/bin/
> > > These are obviously generic, and the same for any kernel.
> >
> > I will check if they are actually needed for arm. Here i am only
> > checking for x86_64.
> 
> I'm pretty sure they are needed for any architecture.  There are
> lex and yacc files in the kernel source tree.  (bison and flex
> are the gnu replacements).
>  tlinux:~/work/torvalds/linux$ find . -name "*.y"
> ./tools/bpf/bpf_exp.y
> ./tools/perf/util/pmu.y
> ./tools/perf/util/expr.y
> ./tools/perf/util/parse-events.y
> ./scripts/genksyms/parse.y
> ./scripts/kconfig/zconf.y
> ./scripts/dtc/dtc-parser.y
> ./drivers/scsi/aic7xxx/aicasm/aicasm_gram.y
> ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_gram.y
> tlinux:~/work/torvalds/linux$ find . -name "*.l"
> ./tools/bpf/bpf_exp.l
> ./tools/perf/util/pmu.l
> ./tools/perf/util/parse-events.l
> ./scripts/genksyms/lex.l
> ./scripts/kconfig/zconf.l
> ./scripts/dtc/dtc-lexer.l
> ./drivers/scsi/aic7xxx/aicasm/aicasm_macro_scan.l
> ./drivers/scsi/aic7xxx/aicasm/aicasm_scan.l
> 
> Most of these are for host-side tools (compilation aids, but
> some can end up in the kernel on the target.  But the lex
> and yacc tools produce parser code that should be architecture
> neutral (and is later compiled by the cross-toolchain for
> the correct architecture).

Ok, then I can put those checks outside the if.

Thanks,
Daniel

> 
>  -- Tim
> 
> >
> > Thanks,
> > Daniel
> >
> > >
> > > > +        assert_define PROGRAM_FLEX
> > > > +    fi
> > > > +}
> > > > +
> > > > +function test_build {
> > > >      if [ -z ${FUNCTIONAL_KERNEL_BUILD_CONFIG+x} ]; then
> > > >          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
> > > >      fi
> > > > --
> > > > 2.7.4
> > > >
> > > >
> > > > _______________________________________________
> > > > Fuego mailing list
> > > > Fuego@lists.linuxfoundation.org
> > > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> >
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego