[LOCAL-CI v2 0/3] Add support for local automated testing

* [LOCAL-CI v2 0/3] Add support for local automated testing
@ 2022-01-04 18:00 Glenn Washburn
  2022-01-04 18:00 ` [LOCAL-CI v2 1/3] scripts: Add general scripts to aid " Glenn Washburn
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Glenn Washburn @ 2022-01-04 18:00 UTC (permalink / raw)
  To: Daniel Kiper, grub-devel; +Cc: Glenn Washburn

Updates since v1:
* update multipipe to return error code from children
* fix bootstrap run cache file not written to correct location
* fix run_test and run_build to properly return test and build error codes
* fix functional test post-processing to recognize failures in any functional
  test, not just ones whose name ends in "test"
* fix issue where tests which were scripts that didn't use @BUILD_SHEBANG@ as
  shell were being run with @BUILD_SHEBANG@ shell, and thus weren't running
  those tests properly (or at all)
* other minor improvements

This patch series aims to make automated testing of all supported targets easy.
Not all targets are actually tested for a variety of reasons, but this series
covers a significant portion of the target space and should make it easy to
add more targets as we figure out how do run their tests. All supported targets
(that I know of) are build tested.

The main work horse is the scripts/local-tester.sh script which ties everything
together. It is very configurable, but defaults to building and testing as much
as it can. To start it just run "./scripts/local-tester.sh" from the root of the
repository. By default a directory named 'grub-tests' in the current working
directory is created and everything is put in there. See the beginning of
./scripts/ci/function.sh for available environment variables that can be used
to configure how you want it to run.

The file ./scripts/ci/function.sh has a bunch of functions most of which are
for stages in the automated testing. It is intended to be sourced by config
files for other CI systems so that they can reuse this code.

One might ask, why not just have all CI systems use local-tester.sh and put
everything in there? The issue is that currently local-tester.sh does not do
things in parallel (the make subprocess can be made more parallel with the
JOBS environment variable). So in many CI systems, one could have all targets
building and testing at the same time, which local-tester.sh only does this
serially. This makes local-tester.sh a lot slower than it needs to be. Also
other CI systems allow the CI pipeline to be broken into many stages, each of
which could be run on a different machine, with the ability to cache the
output of certain stages.

These scripts have been written and tested on debian based systems, specifically
the reference system, Debian 11. Some of the stages are debian or debian-
derivative specific, such as the package install stage which uses apt and dpkg.
Most of the stages, I believe, are fairly distro agnostic and the ones that
aren't should be able to be adapted for a specific distro fairly easily.

Also, this patch series is meant to be used on top of the grub-shell patch
series already submitted to the list. It will run without that series, but
some of the features may not work or work as well. Noteably, the QEMU tests
for targets i386-efi, arm-efi and arm64-efi will fail.

Of particular note, there are some knobs that can provide a lot debugging
output and save the intermediate files of failed tests for later analysis.

local-tester.sh will try to download and install all packages it needs to
function. Obviously, this will not work when not running as a privileged user.
A further patch series is intended, which will add support for running the
system successfully completely as an unprivileged user. If local-tester.sh is
run as an unprivileged user, it will skip running of privileged commands, like
the package installer. This means it can continue past the package install
phase, but it assumes that the needed packages are already installed.

Glenn

Glenn Washburn (3):
  scripts: Add general scripts to aid automated testing
  scripts: Add functions for CI stages and default variables to
    functions.sh
  scripts: Add local-tester.sh script to run local CI tests

 scripts/ci/build.sh           |  67 +++
 scripts/ci/functions.local.sh |  37 ++
 scripts/ci/functions.sh       | 954 ++++++++++++++++++++++++++++++++++
 scripts/ci/make-images.sh     |  86 +++
 scripts/ci/process-tests.sh   | 111 ++++
 scripts/ci/test.sh            | 121 +++++
 scripts/local-tester.sh       |  39 ++
 7 files changed, 1415 insertions(+)
 create mode 100755 scripts/ci/build.sh
 create mode 100644 scripts/ci/functions.local.sh
 create mode 100644 scripts/ci/functions.sh
 create mode 100755 scripts/ci/make-images.sh
 create mode 100755 scripts/ci/process-tests.sh
 create mode 100755 scripts/ci/test.sh
 create mode 100755 scripts/local-tester.sh

Interdiff against v1:

diff --git a/scripts/ci/functions.sh b/scripts/ci/functions.sh
index f94f90dc1..734b36d3f 100644
--- a/scripts/ci/functions.sh
+++ b/scripts/ci/functions.sh
@@ -350,7 +350,8 @@ EXPECTED_FAILURES=${EXPECTED_FAILURES:-"
       ### Usually this test passes, but occasionally the virtual machine is
       ### too slow and the time delta threshold is exceeded. So allow failures.
       #grub_cmd_sleep
-      ### This fails on the volume label check
+      ### This can fail on the volume label check because newer Debian versions
+      ### hfsprogs has a bug where mkfs.hfs does not write the volume label.
       hfs_test
       ### This fails because grub-fstest ls -la is not outputting file time
       ### on files, but it is on directories. In linux there are file times.
@@ -377,7 +378,6 @@ EXPECTED_FAILURES=${EXPECTED_FAILURES:-"
 
 EXPECTED_FUNCTIONAL_FAILURES=${EXPECTED_FUNCTIONAL_FAILURES:-"
       ### These have not worked for a very long time
-      videotest_checksum
       gfxterm_menu
       cmdline_cat_test
       ### Sometimes the machine the test are running on are slow causing
@@ -476,9 +476,17 @@ helper_process_redirects() {
 # that stderr is piped into the next command in the pipe as fd 3.
 multipipe() {
   local ADELIM=$'\x01'
-  local LAST_PID IFS_OLD CMD
+  local ERROR_ANY RET_PID RET_PID_NUM IFS_OLD CMD
   local PIPEFILEOUT PIPEFILEERR PIPEFILEOUTPREV PIPEFILEERRPREV
 
+  while [ "$#" -gt 0 ]; do
+    case "$1" in
+      -e) ERROR_ANY=1; shift;;
+      -r) RET_PID_NUM="$2"; shift 2;;
+      *) break;;
+    esac
+  done
+
   # For each command from last to first, start command with input and output
   # as file descriptors to named pipes.
   local i=0 max=$(echo "$*" | sed 's/|/\n/g' | wc -l) PID=
@@ -492,9 +500,10 @@ multipipe() {
       PIPEFILEERR=$(mktemp -u "${TMPDIR:-/tmp}"/multipipe.XXXXXXXXXX.fifo)
       mkfifo $PIPEFILEERR
       helper_process_redirects "$@" <$PIPEFILEOUT 3<$PIPEFILEERR &
-      LAST_PID=$!
+      PID=$!
     elif [ "$i" -eq "$((max-1))" ]; then
       helper_process_redirects "$@" >$PIPEFILEOUT 2>$PIPEFILEERR &
+      PID=$!
       rm -f $PIPEFILEOUT $PIPEFILEERR
     else
       PIPEFILEOUTPREV=$PIPEFILEOUT
@@ -517,13 +526,27 @@ multipipe() {
         # previous command in the pipeline.
         exec >$PIPEFILEOUTPREV 2>$PIPEFILEERRPREV
         helper_process_redirects "$@" <$PIPEFILEOUT 3<$PIPEFILEERR &
+        exit $!
       )
+      PID=$?
       rm -f $PIPEFILEOUTPREV $PIPEFILEERRPREV
     fi
+
+    if [ "$ERROR_ANY" = "1" ]; then
+      RET_PID="$RET_PID $PID"
+    elif [ -z "$RET_PID_NUM" ] || [ "$RET_PID_NUM" -eq "$((max-$i))" ]; then
+      RET_PID=$PID
+    elif [ "$RET_PID_NUM" -eq "$((max-$i))" ]; then
+      RET_PID=$PID
+    fi
+
     i=$((i+1))
     IFS=$IFSOLD
   done < <(IFS=$ADELIM; { echo -n "$*" | sed "s/${ADELIM}|${ADELIM}/\n/g"; echo; } | tac)
-  wait $LAST_PID
+
+  for PID in $RET_PID; do
+    wait -f $PID || return $?
+  done
 }
 
 multipipe_test() {
@@ -561,12 +584,8 @@ setup_packages() {
     echo "No apt binary found, skipping package installation..."
     return
   fi
-  if [ "$EUID" = 0 ]; then
-    :
-  elif which apt-user >/dev/null 2>&1; then
-    apt() { ROOT="rootfs.ovl" apt-user "$@" ||:; }
-  else
-    echo "Not root and apt-user not found, continuing assuming that all needed packages are installed..."
+  if [ "$EUID" != 0 ]; then
+    echo "Not root, continuing assuming that all needed packages are installed..."
     return
   fi
   apt $APT_OPTS update -yqq &&
@@ -584,6 +603,7 @@ setup_repo() {
     while [ ! -d "${GIT_REPO_DEFAULT}/.git" ]; do
       GIT_REPO_DEFAULT="${GIT_REPO_DEFAULT%/*}"
     done
+    GIT_REPO_DEFAULT="file://${GIT_REPO_DEFAULT}"
   fi
 
   if [ -d "$SRCDIR" ]; then
@@ -714,7 +734,7 @@ setup_bootstrap() {
   # We cache the bootstrap, so if the bootstrap canary exists, don't run the bootstrap
   if [ ! -f "$SRCDIR"/.bootstrap.finished ]; then
     ( cd "$SRCDIR" && ./bootstrap ) &&
-      touch .bootstrap.finished || RET=1
+      touch "$SRCDIR"/.bootstrap.finished || RET=1
   fi
   end_log -n "bootstrap"
   return ${RET:-0}
@@ -780,6 +800,7 @@ setup_env() {
 }
 
 run_build() {
+  local RET=0
   if [ "x${DISABLE_ALL_BUILDS}" = "xy" ]; then
     echo "All builds are disabled"
     return
@@ -803,7 +824,7 @@ run_build() {
 #!$SHELL
 if [ "x\${SHELL_TRACE}" = "xy" ]; then
   set -x
-  if [ -n "\${SHELL_OPTS##* -x*}" ]; then
+  if [ -z "\$SHELL_OPTS" ] || [ -n "\${SHELL_OPTS##* -x*}" ]; then
     SHELL_OPTS="\$SHELL_OPTS -x"
   fi
 fi
@@ -815,15 +836,17 @@ EOF
   export CONFIGURE_OPTS="${CONFIGURE_OPTS} CONFIG_SHELL=${BUILDDIR}/shell-wrapper.sh"
 
   if type multipipe >/dev/null; then
-    multipipe "${CI_SCRIPTS_DIR}/build.sh" \
-      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/build.log"
+    multipipe -e "${CI_SCRIPTS_DIR}/build.sh" \
+      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/build.log" || RET=$?
   else
-    { "${CI_SCRIPTS_DIR}/build.sh" |
-      tee >(exec cat >&3) >"${BUILDDIR}/build.log" 2>&1; } 3>&1
+    { { "${CI_SCRIPTS_DIR}/build.sh" || RET=$?; } |
+      tee >(exec cat >&3) >"${BUILDDIR}/build.log" 2>&1; } 3>&1 || :
   fi
+  return $RET
 }
 
 run_test() {
+  local RET=0
   if [ "x${DISABLE_ALL_TESTS}" = "xy" ]; then
     echo "All tests are disabled"
     return
@@ -831,12 +854,13 @@ run_test() {
 
   export TESTTMPDIR="${TMPDIR:-/tmp}/grub-test-$TARGET-${CI_COMMIT_SHORT_SHA}"
   if type multipipe >/dev/null; then
-    multipipe "${CI_SCRIPTS_DIR}/test.sh" \
-      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/test.log" || :
+    multipipe -e "${CI_SCRIPTS_DIR}/test.sh" \
+      '|' multitee 0:1,5 3:2,5 "5>${BUILDDIR}/test.log" || RET=$?
   else
-    { "${CI_SCRIPTS_DIR}/test.sh" |
-        tee >(exec cat >&3) >"${BUILDDIR}/test.log" 2>&1; } 3>&1 || :
+    { { "${CI_SCRIPTS_DIR}/test.sh" || RET=$?; } |
+      tee >(exec cat >&3) >"${BUILDDIR}/test.log" 2>&1; } 3>&1 || :
   fi
+  return $RET
 }
 
 post_processing() {
diff --git a/scripts/ci/make-images.sh b/scripts/ci/make-images.sh
index 3c5fd64f8..d156af46f 100755
--- a/scripts/ci/make-images.sh
+++ b/scripts/ci/make-images.sh
@@ -74,7 +74,7 @@ for tfmt in $tformats; do
   [ "${_RET:-0}" -ne 0 ] && RET=$_RET
 
   if [ "$RET" -ne 0 ]; then
-    echo -e "${TXT_RED}"'Failed to build image for target format ${tfmt}:' "${TESTNAME}$TXT_CLEAR"
+    echo -e "${TXT_RED}Failed to build image for target format ${tfmt}: ${TESTNAME}$TXT_CLEAR"
     echo -e -n "$TXT_LOG_COLOR"
     echo "Last ${NUM_FAILED_LOG_LINES} lines of grub-mkimage verbose log"
     tail -n "${NUM_FAILED_LOG_LINES}" "${BUILDDIR}/grub-mkimage-${tfmt}.log"
diff --git a/scripts/ci/process-tests.sh b/scripts/ci/process-tests.sh
index bdd6c652d..9bf763502 100755
--- a/scripts/ci/process-tests.sh
+++ b/scripts/ci/process-tests.sh
@@ -66,7 +66,7 @@ if [ -f ${BUILDDIR}/test-suite.log ]; then
     # If any unexpected failures in the functional tests, count a failure in
     # grub_func_test as a true failure.
     elif [ "$TESTNAME" = "grub_func_test" ]; then
-      grep -E 'test: FAIL' ${BUILDDIR}/${TESTNAME}.log | sort -u |
+      grep -E ': FAIL' ${BUILDDIR}/${TESTNAME}.log | sort -u |
       while read FTESTNAME STATUS; do
         FTESTNAME=${FTESTNAME%:*}
         if echo "${EXPECTED_FUNCTIONAL_FAILURES}" | grep -qE "^\s*(${FTESTNAME}|${TARGET}:${FTESTNAME})$"; then
diff --git a/scripts/ci/test.sh b/scripts/ci/test.sh
index f70905462..833912587 100755
--- a/scripts/ci/test.sh
+++ b/scripts/ci/test.sh
@@ -66,18 +66,19 @@ mkdir -p "\$TMPDIR"
 
 # if not a shell script, run normally
 if [ "\$(head -c2 \$1)" = "#!" ]; then
-  TEST_SHELL="$(head -n1 "${BUILDDIR}/grub-shell" | tail -c+3 | tr -d ' ')"
+  BUILD_SHEBANG="\$(head -n1 "${BUILDDIR}/grub-shell" | tail -c+3 | sed -E 's|^\\s*||')"
+  TEST_SHELL=\$(head -n1 "\$1" | tail -c+3 | sed -E 's|^\\s*||')
   # Only turn on tracing if the shell is the one used by grub-shell
-  if head -n1 \$1 | grep -q \$TEST_SHELL; then
+  if test "\${TEST_SHELL}" = "\${BUILD_SHEBANG}"; then
     if [ '(' "\${TEST_VERBOSITY:-0}" -gt 0 -o "x\${SHELL_TRACE}" = "xy" ')' \\
-        -a -n "\${TEST_SHELL##*-x*}" ]; then
-      TEST_SHELL="\$TEST_SHELL -x"
+        -a '(' -z "\$SHELL_OPTS" -o -n "\${SHELL_OPTS##*-x*}" ')' ]; then
+      SHELL_OPTS="\$SHELL_OPTS -x"
     fi
   fi
-fi;
+fi
 
 TSTART=\$(date +%s)
-eval \$(echo -n "\$STRACE" | sed "s/xxx/\$TESTNAME/g") \$TEST_SHELL "\$@" || RET=\$?
+eval \$(echo -n "\$STRACE" | sed "s/xxx/\$TESTNAME/g") "\$TEST_SHELL" "\$@" || RET=\$?
 TEND=\$(date +%s)
 echo "Test duration in seconds: \$((TEND - TSTART))"
 
-- 
2.27.0



^ permalink raw reply related	[flat|nested] 4+ messages in thread