[PATCH v3 00/10] Travis fixes

[PATCH v3 00/10] Travis fixes

[Alexander Graf]

This patch set collects a few fixes for Travis CI since the initial
commit was applied:

  - catch up with the tree
  - fix targets that need a board specified
  - make mkimage loop more robust
  - add QEMU tests for ARM and AArch64 EFI targets
  - add ARM Thumb mode tests
  - verify that on demand module and config loading from TFTP works
  - disable MIPS/IA64 tests
  - add arm_thumb test

That way we should hopefully catch even more problems going forward.

v1 -> v2:

  - add ARM Thumb mode tests
  - verify that on demand module and config loading from TFTP works
  - use local gnulib (saves ~3m git checkout time)

v2 -> v3:

  - add arm_thumb target
  - disable MIPS/IA64
  - Use 19.03 release firmware files
  - add comments to why we still use the system gnulib

Alexander Graf (10):
  travis: Run bootstrap instead of autogen.sh
  travis: Fix sparc64 test
  travis: Fix mips QEMU target
  travis: Fix arm coreboot test and make loop more robus
  arm coreboot: Use common directory path
  travis: Add smoke tests for arm and aarch64
  travis: Add ARM thumb target to tests
  travis: Test module loading from tftp as well
  travis: Disable MIPS target
  travis: Disable IA64 target

 .travis.yml    | 67 +++++++++++++++++++++++++++++++++++++++++++++-------------
 util/mkimage.c |  4 ++--
 2 files changed, 54 insertions(+), 17 deletions(-)

-- 
2.16.4

[PATCH v3 01/10] travis: Run bootstrap instead of autogen.sh

[Alexander Graf]

Commit 35b909062e7b3 ("gnulib: Upgrade Gnulib and switch to bootstrap tool")
changed the build flow from running ./autogen.sh to running ./bootstrap
but missed to update .travis.yml. Adapt it accordingly.

Fixes: 35b909062e7b3 ("gnulib: Upgrade Gnulib and switch to bootstrap tool")
Signed-off-by: Alexander Graf <agraf@csgraf.de>

---

v1 -> v2:

  - use local gnulib copy (saves ~3m git checkout time)

v2 -> v3:

  - add comments to why we still use the system gnulib
---
 .travis.yml | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 81de20fa3..de8ff057a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -18,6 +18,7 @@ addons:
     - python
     - qemu-system
     - unifont
+    - gnulib
 
 env:
   global:
@@ -35,7 +36,13 @@ before_script:
 script:
   # Comments must be outside the command strings below, or the Travis parser
   # will get confused.
-  - ./autogen.sh
+
+  # The Ubuntu gnulib does not compile with gcc8, so only use it with the system compiler.
+  # For targets using the system compiler though (x86 currently), this trick saves a few
+  # minutes on every compile run.
+  - if [ ! "$CROSS_TARGETS" ]; then rm -rf gnulib; ln -s /usr/share/gnulib; fi
+  # Initialize build system
+  - ./bootstrap
 
   # Build all selected GRUB targets.
   - for target in $GRUB_TARGETS; do
-- 
2.16.4

[PATCH v3 02/10] travis: Fix sparc64 test

[Alexander Graf]

For sparc64, we were not selecting the correct mkimage output type. Just
pick aout at random. Also, make sure the rest of the variable logic can
deal with the 3rd element.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>
---
 .travis.yml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index de8ff057a..1af489630 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -47,7 +47,8 @@ script:
   # Build all selected GRUB targets.
   - for target in $GRUB_TARGETS; do
       plat=${target#*-};
-      arch=${target%-*};
+      plat=${plat%-*};
+      arch=${target%%-*};
       [ "$arch" = "arm64" ] && arch=aarch64-linux;
       [ "$arch" = "arm" ] && arch=arm-linux-gnueabi;
       [ "$arch" = "ia64" ] && arch=ia64-linux;
@@ -87,7 +88,7 @@ matrix:
         - CROSS_TARGETS="powerpc64-linux"
     - name: "sparc64"
       env:
-        - GRUB_TARGETS="sparc64-ieee1275"
+        - GRUB_TARGETS="sparc64-ieee1275-aout"
         - CROSS_TARGETS="sparc64-linux"
     - name: "ia64"
       env:
-- 
2.16.4

[PATCH v3 03/10] travis: Fix mips QEMU target

[Alexander Graf]

The MIPS QEMU targets can be built as either elf binary or flash image.
Build one each for BE/LE and make sure we pass the correct one into
mkimage.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>
---
 .travis.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 1af489630..d7563958e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -96,7 +96,7 @@ matrix:
         - CROSS_TARGETS="ia64-linux"
     - name: "mips"
       env:
-        - GRUB_TARGETS="mips-arc mipsel-arc mipsel-qemu_mips mips-qemu_mips"
+        - GRUB_TARGETS="mips-arc mipsel-arc mipsel-qemu_mips-elf mips-qemu_mips-flash"
         - CROSS_TARGETS="mips64-linux"
     - name: "arm"
       env:
-- 
2.16.4

[PATCH v3 04/10] travis: Fix arm coreboot test and make loop more robus

[Alexander Graf]

We missed the fact that the coreboot target did not succeed on mkimage.
Properly abort the loop if we see a failure and fix the coreboot target
to also indicate the board target.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>
---
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index d7563958e..7945efa14 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -68,7 +68,7 @@ script:
   - echo -e "insmod echo\\ninsmod reboot\\necho hello world\\nreboot" > grub.cfg
 
   # Assemble images and possibly run them.
-  - for target in $GRUB_TARGETS; do grub-mkimage -c grub.cfg -p / -O $target -o grub-$target echo reboot normal; done
+  - ( for target in $GRUB_TARGETS; do grub-mkimage -c grub.cfg -p / -O $target -o grub-$target echo reboot normal || exit; done )
 
   # Run images we know how to run.
   - if [[ "$GRUB_TARGETS" == *"x86_64-efi"* ]]; then qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-x86_64-efi | tee grub.log && grep "hello world" grub.log; fi
@@ -100,7 +100,7 @@ matrix:
         - CROSS_TARGETS="mips64-linux"
     - name: "arm"
       env:
-        - GRUB_TARGETS="arm-coreboot arm-efi arm-uboot"
+        - GRUB_TARGETS="arm-coreboot-vexpress arm-efi arm-uboot"
         - CROSS_TARGETS="arm-linux-gnueabi"
     - name: "arm64"
       env:
-- 
2.16.4

[PATCH v3 05/10] arm coreboot: Use common directory path

[Alexander Graf]

The ARM coreboot target supports multiple boards, but they are all built
using the same object path. The only difference in target boards is done
in mkimage to determine the load address.

Currently, mkimage is looking at a board specific path
(/usr/lib/grub/arm-coreboot-vexpress) for modules while it installs it
at a non-specific path (/usr/lib/grub/arm-coreboot).

So let's sync the two up again and tell mkimage to look for coreboot
modules at a board agnostic path, syncing it up with all other targets.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>
---
 util/mkimage.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/util/mkimage.c b/util/mkimage.c
index 37d6249f1..f1a9c9c86 100644
--- a/util/mkimage.c
+++ b/util/mkimage.c
@@ -553,7 +553,7 @@ static const struct grub_install_image_target_desc image_targets[] =
     },
     /* For coreboot versions that don't support self-relocating images. */
     {
-      .dirname = "arm-coreboot-vexpress",
+      .dirname = "arm-coreboot",
       .names = { "arm-coreboot-vexpress", NULL },
       .voidp_sizeof = 4,
       .bigendian = 0,
@@ -572,7 +572,7 @@ static const struct grub_install_image_target_desc image_targets[] =
       .link_addr = 0x62000000,
     },
     {
-      .dirname = "arm-coreboot-veyron",
+      .dirname = "arm-coreboot",
       .names = { "arm-coreboot-veyron", NULL },
       .voidp_sizeof = 4,
       .bigendian = 0,
-- 
2.16.4

[PATCH v3 06/10] travis: Add smoke tests for arm and aarch64

[Alexander Graf]

We've had an arm regression in grub recently where grub would not even
be able to boot up anymore. So let's include arm and aarch64 in our
simple "hello world" smoke tests as well.

For OVMF on ARM to work, we need a newer version of QEMU, add a PPA
dependency for it.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>

---

v2 -> v3:

  - Use 19.03 release firmware files
---
 .travis.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 7945efa14..d6b827960 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,6 +11,8 @@ language: c
 
 addons:
   apt:
+    sources:
+    - sourceline: 'ppa:jacob/virtualisation'
     packages:
     - libsdl1.2-dev
     - lzop
@@ -32,6 +34,8 @@ before_script:
   - for i in $CROSS_TARGETS; do
         ( cd /tmp/cross; wget -t 3 -O - https://mirrors.kernel.org/pub/tools/crosstool/files/bin/x86_64/8.1.0/x86_64-gcc-8.1.0-nolibc-$i.tar.xz | tar xJ );
     done
+  - if [[ "$GRUB_TARGETS" == *"arm64-efi"* ]]; then wget http://releases.linaro.org/reference-platform/enterprise/firmware/open-source/19.03/release/qemu-aarch64/QEMU_EFI.fd -O QEMU_EFI.aarch64.fd; fi
+  - if [[ "$GRUB_TARGETS" == *"arm-efi"* ]]; then wget http://releases.linaro.org/reference-platform/enterprise/firmware/open-source/19.03/release/qemu-arm/QEMU_EFI.fd -O QEMU_EFI.arm.fd; fi
 
 script:
   # Comments must be outside the command strings below, or the Travis parser
@@ -71,7 +75,9 @@ script:
   - ( for target in $GRUB_TARGETS; do grub-mkimage -c grub.cfg -p / -O $target -o grub-$target echo reboot normal || exit; done )
 
   # Run images we know how to run.
-  - if [[ "$GRUB_TARGETS" == *"x86_64-efi"* ]]; then qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-x86_64-efi | tee grub.log && grep "hello world" grub.log; fi
+  - if [[ "$GRUB_TARGETS" == *"x86_64-efi"* ]]; then qemu-system-x86_64                          -bios /usr/share/ovmf/OVMF.fd -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-x86_64-efi | tee grub.log && grep "hello world" grub.log; fi
+  - if [[ "$GRUB_TARGETS" == *"arm64-efi"* ]];  then qemu-system-aarch64 -M virt -cpu cortex-a57 -bios QEMU_EFI.aarch64.fd     -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-arm64-efi  | tee grub.log && grep "hello world" grub.log; fi
+  - if [[ "$GRUB_TARGETS" == *"arm-efi"* ]];    then qemu-system-arm     -M virt -cpu cortex-a15 -bios QEMU_EFI.arm.fd         -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-arm-efi    | tee grub.log && grep "hello world" grub.log; fi
 
 matrix:
   include:
-- 
2.16.4

[PATCH v3 07/10] travis: Add ARM thumb target to tests

[Alexander Graf]

We hit an error case which only got triggered on ARM Thumb code. So
let's make sure we test that combination as well.

Due to popular request from Leif, let's also ensure that this actually
tests thumb in armv7-a mode.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
---
 .travis.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index d6b827960..170af5312 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -108,6 +108,11 @@ matrix:
       env:
         - GRUB_TARGETS="arm-coreboot-vexpress arm-efi arm-uboot"
         - CROSS_TARGETS="arm-linux-gnueabi"
+    - name: "arm_thumb"
+      env:
+        - GRUB_TARGETS="arm-coreboot-vexpress arm-efi arm-uboot"
+        - CROSS_TARGETS="arm-linux-gnueabi"
+        - TARGET_CFLAGS="-mthumb -march=armv7-a"
     - name: "arm64"
       env:
         - GRUB_TARGETS="arm64-efi"
-- 
2.16.4

[PATCH v3 08/10] travis: Test module loading from tftp as well

[Alexander Graf]

The travis test today only uses modules that are delivered with the
grub.efi binary. Let's drop echo and reboot and see if grub can load
them dynamically.

For this, we need to ensure that all modules required to load additional
modules are included in the grub binary. Some of these are target
specific (like efinet), so only include them conditionally.

Furthermore, the iPXE option rom included in Ubuntu Xenial seems to be
broken, so we need to ensure that we only use the OVMF built-in network
driver and not pass in an option rom.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>

---

v1 -> v2:

  - new patch
---
 .travis.yml | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 170af5312..5867efea5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -69,15 +69,27 @@ script:
     done
 
   # Our test canary.
-  - echo -e "insmod echo\\ninsmod reboot\\necho hello world\\nreboot" > grub.cfg
+  - echo -e "insmod echo\\ninsmod reboot\\necho hello world\\nreboot" > /tmp/grub/lib/grub/grub.cfg
 
   # Assemble images and possibly run them.
-  - ( for target in $GRUB_TARGETS; do grub-mkimage -c grub.cfg -p / -O $target -o grub-$target echo reboot normal || exit; done )
+  - ( for target in $GRUB_TARGETS; do
+        pushd /tmp/grub/lib/grub/;
+        if [ -f $target/efinet.mod ]; then
+          EXTMODULES="efinet";
+        else
+          EXTMODULES="";
+        fi;
+        grub-mkimage -p / -O $target -o grub-$target normal tftp $EXTMODULES || exit;
+        popd;
+      done )
 
   # Run images we know how to run.
-  - if [[ "$GRUB_TARGETS" == *"x86_64-efi"* ]]; then qemu-system-x86_64                          -bios /usr/share/ovmf/OVMF.fd -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-x86_64-efi | tee grub.log && grep "hello world" grub.log; fi
-  - if [[ "$GRUB_TARGETS" == *"arm64-efi"* ]];  then qemu-system-aarch64 -M virt -cpu cortex-a57 -bios QEMU_EFI.aarch64.fd     -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-arm64-efi  | tee grub.log && grep "hello world" grub.log; fi
-  - if [[ "$GRUB_TARGETS" == *"arm-efi"* ]];    then qemu-system-arm     -M virt -cpu cortex-a15 -bios QEMU_EFI.arm.fd         -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-arm-efi    | tee grub.log && grep "hello world" grub.log; fi
+  - if [[ "$GRUB_TARGETS" == *"x86_64-efi"* ]]; then qemu-system-x86_64                          -bios /usr/share/ovmf/OVMF.fd
+                         -m 512 -no-reboot -nographic -net none -netdev user,id=nd,tftp=/tmp/grub/lib/grub/,bootfile=grub-x86_64-efi -device virtio-net-pci,netdev=nd,romfile="" | tee grub.log && grep "hello world" grub.log; fi
+  - if [[ "$GRUB_TARGETS" == *"arm64-efi"* ]];  then qemu-system-aarch64 -M virt -cpu cortex-a57 -bios QEMU_EFI.aarch64.fd
+                         -m 512 -no-reboot -nographic -net none -netdev user,id=nd,tftp=/tmp/grub/lib/grub/,bootfile=grub-arm64-efi  -device virtio-net-pci,netdev=nd,romfile="" | tee grub.log && grep "hello world" grub.log; fi
+  - if [[ "$GRUB_TARGETS" == *"arm-efi"* ]];    then qemu-system-arm     -M virt -cpu cortex-a15 -bios QEMU_EFI.arm.fd
+                         -m 512 -no-reboot -nographic -net none -netdev user,id=nd,tftp=/tmp/grub/lib/grub/,bootfile=grub-arm-efi    -device virtio-net-pci,netdev=nd,romfile="" | tee grub.log && grep "hello world" grub.log; fi
 
 matrix:
   include:
-- 
2.16.4

[PATCH v3 09/10] travis: Disable MIPS target

[Alexander Graf]

The MIPS target currently does not compile due to the following error
message:

  configure: error: could not force big-endian

If anyone cares enough about MIPS to fix it up, be my guest and revert
this commit afterwards. For now, we really need to move forward with
a fully successful travis run to ensure that we catch regressions early
on.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
---
 .travis.yml | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 5867efea5..b63a992aa 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -112,10 +112,13 @@ matrix:
       env:
         - GRUB_TARGETS="ia64-efi"
         - CROSS_TARGETS="ia64-linux"
-    - name: "mips"
-      env:
-        - GRUB_TARGETS="mips-arc mipsel-arc mipsel-qemu_mips-elf mips-qemu_mips-flash"
-        - CROSS_TARGETS="mips64-linux"
+    # MIPS fails with the following error currently. If anyone cares about MIPS
+    # testing, please reenable it after resolving the problem:
+    #    configure: error: could not force big-endian
+    #- name: "mips"
+    #  env:
+    #    - GRUB_TARGETS="mips-arc mipsel-arc mipsel-qemu_mips-elf mips-qemu_mips-flash"
+    #    - CROSS_TARGETS="mips64-linux"
     - name: "arm"
       env:
         - GRUB_TARGETS="arm-coreboot-vexpress arm-efi arm-uboot"
-- 
2.16.4

[PATCH v3 10/10] travis: Disable IA64 target

[Alexander Graf]

The IA64 target currently does not compile due to the following error
message:

  grub-mkimage: error: undefined symbol grub_arch_sync_dma_caches.

If anyone cares enough about IA64 to fix it up, be my guest and revert
this commit afterwards. For now, we really need to move forward with
a fully successful travis run to ensure that we catch regressions early
on.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
---
 .travis.yml | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index b63a992aa..5367ca15b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -108,10 +108,13 @@ matrix:
       env:
         - GRUB_TARGETS="sparc64-ieee1275-aout"
         - CROSS_TARGETS="sparc64-linux"
-    - name: "ia64"
-      env:
-        - GRUB_TARGETS="ia64-efi"
-        - CROSS_TARGETS="ia64-linux"
+    # IA fails with the following error currently. If anyone cares about IA64
+    # testing, please reenable it after resolving the problem:
+    #    grub-mkimage: error: undefined symbol grub_arch_sync_dma_caches.
+    #- name: "ia64"
+    #  env:
+    #    - GRUB_TARGETS="ia64-efi"
+    #    - CROSS_TARGETS="ia64-linux"
     # MIPS fails with the following error currently. If anyone cares about MIPS
     # testing, please reenable it after resolving the problem:
     #    configure: error: could not force big-endian
-- 
2.16.4

Re: [PATCH v3 10/10] travis: Disable IA64 target

[John Paul Adrian Glaubitz]

On 6/4/19 12:27 PM, Alexander Graf wrote:
> The IA64 target currently does not compile due to the following error
> message:
> 
>   grub-mkimage: error: undefined symbol grub_arch_sync_dma_caches.
> 
> If anyone cares enough about IA64 to fix it up, be my guest and revert
> this commit afterwards. For now, we really need to move forward with
> a fully successful travis run to ensure that we catch regressions early
> on.

This can't be. GRUB builds just fine on ia64, I just tested that yesterday.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 09/10] travis: Disable MIPS target

[John Paul Adrian Glaubitz]

On 6/4/19 12:27 PM, Alexander Graf wrote:
> The MIPS target currently does not compile due to the following error
> message:
> 
>   configure: error: could not force big-endian
> 
> If anyone cares enough about MIPS to fix it up, be my guest and revert
> this commit afterwards. For now, we really need to move forward with
> a fully successful travis run to ensure that we catch regressions early
> on.
Same here, builds fine on Debian unstable. Just tested yesterday.

In fact, all supported targets build fine.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 10/10] travis: Disable IA64 target

[Alexander Graf]

On 04.06.19 13:35, John Paul Adrian Glaubitz wrote:
> On 6/4/19 12:27 PM, Alexander Graf wrote:
>> The IA64 target currently does not compile due to the following error
>> message:
>>
>>   grub-mkimage: error: undefined symbol grub_arch_sync_dma_caches.
>>
>> If anyone cares enough about IA64 to fix it up, be my guest and revert
>> this commit afterwards. For now, we really need to move forward with
>> a fully successful travis run to ensure that we catch regressions early
>> on.
> This can't be. GRUB builds just fine on ia64, I just tested that yesterday.


Travis doesn't lie :). Maybe it's something about the toolchain used?

https://travis-ci.org/agraf/grub/jobs/541099309


Alex

Re: [PATCH v3 09/10] travis: Disable MIPS target

[Alexander Graf]

On 04.06.19 13:36, John Paul Adrian Glaubitz wrote:
> On 6/4/19 12:27 PM, Alexander Graf wrote:
>> The MIPS target currently does not compile due to the following error
>> message:
>>
>>   configure: error: could not force big-endian
>>
>> If anyone cares enough about MIPS to fix it up, be my guest and revert
>> this commit afterwards. For now, we really need to move forward with
>> a fully successful travis run to ensure that we catch regressions early
>> on.
> Same here, builds fine on Debian unstable. Just tested yesterday.
>
> In fact, all supported targets build fine.


Could you please just fix up the travis tests to succeed? I think the
MIPS problem is definitely toolchain related. Maybe you can somehow
convince that travis ubuntu container to use official Debian cross
compilers instead of the kernel.org ones that we use right now?


Alex

Re: [PATCH v3 10/10] travis: Disable IA64 target

[Alexander Graf]

On 04.06.19 13:35, John Paul Adrian Glaubitz wrote:
> On 6/4/19 12:27 PM, Alexander Graf wrote:
>> The IA64 target currently does not compile due to the following error
>> message:
>>
>>   grub-mkimage: error: undefined symbol grub_arch_sync_dma_caches.
>>
>> If anyone cares enough about IA64 to fix it up, be my guest and revert
>> this commit afterwards. For now, we really need to move forward with
>> a fully successful travis run to ensure that we catch regressions early
>> on.
> This can't be. GRUB builds just fine on ia64, I just tested that yesterday.


Actually, the IA64 error is not a build error but a runtime error.
Mkimage fails.

My main motivation with this patch set is to get us into green travis
state. If that means we only cover 80% ground today, so be it. It's
still much better than the 0% we have today.

Going forward, I would *really* like to see a policy put into place that
mandates every upstream merge to succeed in Travis. Any commit that
breaks the CI should simply not land in master. That way we can
guarantee that base functionality is always available.

That also means that once you have time to enable MIPS and IA64, we have
a good regression test in place to ensure that they won't break going
forward from that point on :).


Alex

Re: [PATCH v3 10/10] travis: Disable IA64 target

[John Paul Adrian Glaubitz]

On 6/4/19 1:46 PM, Alexander Graf wrote:
> Travis doesn't lie :). Maybe it's something about the toolchain used?
> 
> https://travis-ci.org/agraf/grub/jobs/541099309

I don't lie either :P. Toolchain and kernel are recent:

root@titanium:~# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/ia64-linux-gnu/8/lto-wrapper
Target: ia64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 8.3.0-10~ia64.1' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=ia64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libssp --disable-libitm --disable-libsanitizer --enable-plugin --with-system-zlib --disable-libphobos --enable-objc-gc=auto --enable-multiarch --disable-werror --with-system-libunwind --enable-checking=release --build=ia64-linux-gnu --host=ia64-linux-gnu --target=ia64-linux-gnu
Thread model: posix
gcc version 8.3.0 (Debian 8.3.0-10~ia64.1) 
root@titanium:~# ld --version
GNU ld (GNU Binutils for Debian) 2.31.1
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
root@titanium:~# uname -a
Linux titanium 4.19.0-5-mckinley #1 SMP Debian 4.19.37-3 (2019-05-18) ia64 GNU/Linux
root@titanium:~#

This is built natively on ia64. I'll try on one of SUSE's own ia64 boxes now.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 06/10] travis: Add smoke tests for arm and aarch64

[Leif Lindholm]

On Tue, Jun 04, 2019 at 12:27:30PM +0200, Alexander Graf wrote:
> We've had an arm regression in grub recently where grub would not even
> be able to boot up anymore. So let's include arm and aarch64 in our
> simple "hello world" smoke tests as well.
> 
> For OVMF on ARM to work, we need a newer version of QEMU, add a PPA
> dependency for it.

On the whole, this looks good. *But* the default arm target here is now
armv5t, which is a very different beast from armv7-a (enabled in a
separate patch).

I'm not saying we need to do a full
- armv5t
- armv6
- armv7-a (A32/T32)
- armv7-a (A32 only)
matrix from day 1, but I think it's worth being explicit about which
instruction set we are tasting (also in case the default changes with
future toolchain releases, or we decide to switch where we get our
toolchains from).

So could you add a CFLAGS=-march=armv5t to the arm test?

/
    Leif

> Signed-off-by: Alexander Graf <agraf@csgraf.de>
> Reviewed-by: Daniel Kiper <daniel.kiper@oracle.com>
> 
> ---
> 
> v2 -> v3:
> 
>   - Use 19.03 release firmware files
> ---
>  .travis.yml | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 7945efa14..d6b827960 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -11,6 +11,8 @@ language: c
>  
>  addons:
>    apt:
> +    sources:
> +    - sourceline: 'ppa:jacob/virtualisation'
>      packages:
>      - libsdl1.2-dev
>      - lzop
> @@ -32,6 +34,8 @@ before_script:
>    - for i in $CROSS_TARGETS; do
>          ( cd /tmp/cross; wget -t 3 -O - https://mirrors.kernel.org/pub/tools/crosstool/files/bin/x86_64/8.1.0/x86_64-gcc-8.1.0-nolibc-$i.tar.xz | tar xJ );
>      done
> +  - if [[ "$GRUB_TARGETS" == *"arm64-efi"* ]]; then wget http://releases.linaro.org/reference-platform/enterprise/firmware/open-source/19.03/release/qemu-aarch64/QEMU_EFI.fd -O QEMU_EFI.aarch64.fd; fi
> +  - if [[ "$GRUB_TARGETS" == *"arm-efi"* ]]; then wget http://releases.linaro.org/reference-platform/enterprise/firmware/open-source/19.03/release/qemu-arm/QEMU_EFI.fd -O QEMU_EFI.arm.fd; fi
>  
>  script:
>    # Comments must be outside the command strings below, or the Travis parser
> @@ -71,7 +75,9 @@ script:
>    - ( for target in $GRUB_TARGETS; do grub-mkimage -c grub.cfg -p / -O $target -o grub-$target echo reboot normal || exit; done )
>  
>    # Run images we know how to run.
> -  - if [[ "$GRUB_TARGETS" == *"x86_64-efi"* ]]; then qemu-system-x86_64 -bios /usr/share/ovmf/OVMF.fd -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-x86_64-efi | tee grub.log && grep "hello world" grub.log; fi
> +  - if [[ "$GRUB_TARGETS" == *"x86_64-efi"* ]]; then qemu-system-x86_64                          -bios /usr/share/ovmf/OVMF.fd -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-x86_64-efi | tee grub.log && grep "hello world" grub.log; fi
> +  - if [[ "$GRUB_TARGETS" == *"arm64-efi"* ]];  then qemu-system-aarch64 -M virt -cpu cortex-a57 -bios QEMU_EFI.aarch64.fd     -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-arm64-efi  | tee grub.log && grep "hello world" grub.log; fi
> +  - if [[ "$GRUB_TARGETS" == *"arm-efi"* ]];    then qemu-system-arm     -M virt -cpu cortex-a15 -bios QEMU_EFI.arm.fd         -m 512 -no-reboot -nographic -net nic -net user,tftp=.,bootfile=grub-arm-efi    | tee grub.log && grep "hello world" grub.log; fi
>  
>  matrix:
>    include:
> -- 
> 2.16.4
> 

Re: [PATCH v3 09/10] travis: Disable MIPS target

[John Paul Adrian Glaubitz]

On 6/4/19 1:47 PM, Alexander Graf wrote:
> Could you please just fix up the travis tests to succeed? I think the
> MIPS problem is definitely toolchain related. Maybe you can somehow
> convince that travis ubuntu container to use official Debian cross
> compilers instead of the kernel.org ones that we use right now?

Yes, Travis should use the cross-compiler packages from Debian. We have
cross-compilers for all targets in Debian except for ia64 as it's a bit
more complicated to build a cross-compiler there due to the fact that
gcc needs libunwind there.

I will look into this, at least for all other targets, the cross-compilers
can easily be installed with "apt install gcc-$GCCVER-$ARCH-linux-gnu",
so if Travis uses Docker here, it will be trivial to fix.

The cross-compilers from kernel.org are bare-bone and can't be used to build
anything but the kernel and a few other utilities due to missing std libraries.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 09/10] travis: Disable MIPS target

[Daniel Kiper]

On Tue, Jun 04, 2019 at 01:47:52PM +0200, Alexander Graf wrote:
>
> On 04.06.19 13:36, John Paul Adrian Glaubitz wrote:
> > On 6/4/19 12:27 PM, Alexander Graf wrote:
> >> The MIPS target currently does not compile due to the following error
> >> message:
> >>
> >>   configure: error: could not force big-endian
> >>
> >> If anyone cares enough about MIPS to fix it up, be my guest and revert
> >> this commit afterwards. For now, we really need to move forward with
> >> a fully successful travis run to ensure that we catch regressions early
> >> on.
> > Same here, builds fine on Debian unstable. Just tested yesterday.
> >
> > In fact, all supported targets build fine.
>
>
> Could you please just fix up the travis tests to succeed? I think the
> MIPS problem is definitely toolchain related. Maybe you can somehow
> convince that travis ubuntu container to use official Debian cross
> compilers instead of the kernel.org ones that we use right now?

Could you install gcc-mipsel-linux-gnu package (it is Debian name but
I expect that Ubuntu will have something similar) and use this compiler
for mipsel? This is not the latest and greatest but at least we will
have some test builds for MIPS.

Daniel

Re: [PATCH v3 09/10] travis: Disable MIPS target

[Alexander Graf]

On 04.06.19 14:01, John Paul Adrian Glaubitz wrote:
> On 6/4/19 1:47 PM, Alexander Graf wrote:
>> Could you please just fix up the travis tests to succeed? I think the
>> MIPS problem is definitely toolchain related. Maybe you can somehow
>> convince that travis ubuntu container to use official Debian cross
>> compilers instead of the kernel.org ones that we use right now?
> Yes, Travis should use the cross-compiler packages from Debian. We have
> cross-compilers for all targets in Debian except for ia64 as it's a bit
> more complicated to build a cross-compiler there due to the fact that
> gcc needs libunwind there.
>
> I will look into this, at least for all other targets, the cross-compilers
> can easily be installed with "apt install gcc-$GCCVER-$ARCH-linux-gnu",
> so if Travis uses Docker here, it will be trivial to fix.


Yes, it does. I did try back when I first added the cross compiler
support, but for some reason did not manage to make use of the built-in
cross compilers. Maybe the versions in that Ubuntu version were just too
old? I really don't remember.

I'll be more than happy to see you make them work though :).


>
> The cross-compilers from kernel.org are bare-bone and can't be used to build
> anything but the kernel and a few other utilities due to missing std libraries.


They seem to work surprisingly well for grub, given that we're even
doing execution tests against a few of the resulting binaries ;).

Alex

Re: [PATCH v3 09/10] travis: Disable MIPS target

[John Paul Adrian Glaubitz]

On 6/4/19 2:07 PM, Alexander Graf wrote:
> Yes, it does. I did try back when I first added the cross compiler
> support, but for some reason did not manage to make use of the built-in
> cross compilers. Maybe the versions in that Ubuntu version were just too
> old? I really don't remember.

Those compilers all come from Debian, so it might be a better idea to use
a Debian container anyway. The maintainer of those compilers maintains them
in Debian, they just get imported into Ubuntu at some point.

> I'll be more than happy to see you make them work though :).

I'll give it a try. Native builds are 100% fine in any case.

>> The cross-compilers from kernel.org are bare-bone and can't be used to build
>> anything but the kernel and a few other utilities due to missing std libraries.
> 
> 
> They seem to work surprisingly well for grub, given that we're even
> doing execution tests against a few of the resulting binaries ;).

Interesting.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 10/10] travis: Disable IA64 target

[Leif Lindholm]

On Tue, Jun 04, 2019 at 12:27:34PM +0200, Alexander Graf wrote:
> The IA64 target currently does not compile due to the following error
> message:
> 
>   grub-mkimage: error: undefined symbol grub_arch_sync_dma_caches.
>
> If anyone cares enough about IA64 to fix it up, be my guest and revert
> this commit afterwards. For now, we really need to move forward with
> a fully successful travis run to ensure that we catch regressions early
> on.

I am reasonably convinced this was simply neglected when d8901e3ba1
("cache: Fix compilation for ppc, sparc and arm64") was pushed.
(And I don't believe ia64 has been buildable since.)

Adding (__ia64__) to that conditional makes the mkimage step work.
Unless someone convinces me differently, I'll submit that as a patch.

/
    Leif

> Signed-off-by: Alexander Graf <agraf@csgraf.de>
> ---
>  .travis.yml | 11 +++++++----
>  1 file changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index b63a992aa..5367ca15b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -108,10 +108,13 @@ matrix:
>        env:
>          - GRUB_TARGETS="sparc64-ieee1275-aout"
>          - CROSS_TARGETS="sparc64-linux"
> -    - name: "ia64"
> -      env:
> -        - GRUB_TARGETS="ia64-efi"
> -        - CROSS_TARGETS="ia64-linux"
> +    # IA fails with the following error currently. If anyone cares about IA64
> +    # testing, please reenable it after resolving the problem:
> +    #    grub-mkimage: error: undefined symbol grub_arch_sync_dma_caches.
> +    #- name: "ia64"
> +    #  env:
> +    #    - GRUB_TARGETS="ia64-efi"
> +    #    - CROSS_TARGETS="ia64-linux"
>      # MIPS fails with the following error currently. If anyone cares about MIPS
>      # testing, please reenable it after resolving the problem:
>      #    configure: error: could not force big-endian
> -- 
> 2.16.4
> 

Re: [PATCH v3 10/10] travis: Disable IA64 target

[John Paul Adrian Glaubitz]

On 6/4/19 3:34 PM, Leif Lindholm wrote:
> I am reasonably convinced this was simply neglected when d8901e3ba1
> ("cache: Fix compilation for ppc, sparc and arm64") was pushed.
> (And I don't believe ia64 has been buildable since.)

*cross*-buildable. It's build absolutely fine naively.

Daniel explicitly asked me to test build GRUB on all supported architectures
and I did that natively on all those targets:

> http://lists.gnu.org/archive/html/grub-devel/2019-06/msg00005.html

> Adding (__ia64__) to that conditional makes the mkimage step work.
> Unless someone convinces me differently, I'll submit that as a patch.
I would have to test that on an actual ia64 machine.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 10/10] travis: Disable IA64 target

[Alexander Graf]

On 04.06.19 15:37, John Paul Adrian Glaubitz wrote:
> On 6/4/19 3:34 PM, Leif Lindholm wrote:
>> I am reasonably convinced this was simply neglected when d8901e3ba1
>> ("cache: Fix compilation for ppc, sparc and arm64") was pushed.
>> (And I don't believe ia64 has been buildable since.)
> *cross*-buildable. It's build absolutely fine naively.
>
> Daniel explicitly asked me to test build GRUB on all supported architectures
> and I did that natively on all those targets:
>
>> http://lists.gnu.org/archive/html/grub-devel/2019-06/msg00005.html


Did you run mkimage on all targets as part of these tests?

Alex

Re: [PATCH v3 10/10] travis: Disable IA64 target

[Leif Lindholm]

On Tue, Jun 04, 2019 at 03:37:39PM +0200, John Paul Adrian Glaubitz wrote:
> On 6/4/19 3:34 PM, Leif Lindholm wrote:
> > I am reasonably convinced this was simply neglected when d8901e3ba1
> > ("cache: Fix compilation for ppc, sparc and arm64") was pushed.
> > (And I don't believe ia64 has been buildable since.)
> 
> *cross*-buildable. It's build absolutely fine naively.

GRUB's configure script accepts (and handles) the --target= option,
which means it considers itself to be a tolchain - with grub-mkimage as
the linker.

(And hence I have a bad habit of including "running grub-mkimage to
actually resolve symbols" when I talk about building grub.)

Yes, this greatly complicates testing.

/
   Leif

Re: [PATCH v3 09/10] travis: Disable MIPS target

[Leif Lindholm]

On Tue, Jun 04, 2019 at 12:27:33PM +0200, Alexander Graf wrote:
> The MIPS target currently does not compile due to the following error
> message:
> 
>   configure: error: could not force big-endian
> 
> If anyone cares enough about MIPS to fix it up, be my guest and revert
> this commit afterwards. For now, we really need to move forward with
> a fully successful travis run to ensure that we catch regressions early
> on.

Having looked into this, I could possibly stitch something together.

As Alex and I found in parallel, the fault mentioned above is caused
by configure failing to identify the target (ending up with
mips-unknown-elf), and it then falling back to use the host cc for the
tests.

The way to work around that is to fully override the target
specification (--target=mips-linux, --target=mips64-linux). But a
further complication is that mips and mips64 use different
toolchains. And --target=mips*el-linux fails in the original way, so
we'd also need to override the toolchain name.

And then there are the different platforms: yeeloong, fuloong,
loongson, arc, qemu_mips.

So in order to enable CI for mips, we would need to:
- add the mips-linux toolchain from the same location as the others
- split up 64-bit and 32-bit targets into different jobs
- fix overriding toolchain name to support -el flavours with this
  prebuilt toolchain. (Or revamp toolchain support in some other way.)
and the killer
- figure out which cpu/platform combos actually make sense

But until someone steps up to clean up and look after mips, from my
point of view:
Reviewed-by: Leif Lindholm <leif.lindholm@linaro.org>

> Signed-off-by: Alexander Graf <agraf@csgraf.de>
> ---
>  .travis.yml | 11 +++++++----
>  1 file changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 5867efea5..b63a992aa 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -112,10 +112,13 @@ matrix:
>        env:
>          - GRUB_TARGETS="ia64-efi"
>          - CROSS_TARGETS="ia64-linux"
> -    - name: "mips"
> -      env:
> -        - GRUB_TARGETS="mips-arc mipsel-arc mipsel-qemu_mips-elf mips-qemu_mips-flash"
> -        - CROSS_TARGETS="mips64-linux"
> +    # MIPS fails with the following error currently. If anyone cares about MIPS
> +    # testing, please reenable it after resolving the problem:
> +    #    configure: error: could not force big-endian
> +    #- name: "mips"
> +    #  env:
> +    #    - GRUB_TARGETS="mips-arc mipsel-arc mipsel-qemu_mips-elf mips-qemu_mips-flash"
> +    #    - CROSS_TARGETS="mips64-linux"
>      - name: "arm"
>        env:
>          - GRUB_TARGETS="arm-coreboot-vexpress arm-efi arm-uboot"
> -- 
> 2.16.4
> 

Re: [PATCH v3 00/10] Travis fixes

[Vladimir 'phcoder' Serbinenko]

LGTM for the whole series. I do care about mips and ia64 but I think
it's fine to disable them for now and iterate from there

On Tue, Jun 4, 2019 at 12:30 PM Alexander Graf <agraf@csgraf.de> wrote:
>
> This patch set collects a few fixes for Travis CI since the initial
> commit was applied:
>
>   - catch up with the tree
>   - fix targets that need a board specified
>   - make mkimage loop more robust
>   - add QEMU tests for ARM and AArch64 EFI targets
>   - add ARM Thumb mode tests
>   - verify that on demand module and config loading from TFTP works
>   - disable MIPS/IA64 tests
>   - add arm_thumb test
>
> That way we should hopefully catch even more problems going forward.
>
> v1 -> v2:
>
>   - add ARM Thumb mode tests
>   - verify that on demand module and config loading from TFTP works
>   - use local gnulib (saves ~3m git checkout time)
>
> v2 -> v3:
>
>   - add arm_thumb target
>   - disable MIPS/IA64
>   - Use 19.03 release firmware files
>   - add comments to why we still use the system gnulib
>
> Alexander Graf (10):
>   travis: Run bootstrap instead of autogen.sh
>   travis: Fix sparc64 test
>   travis: Fix mips QEMU target
>   travis: Fix arm coreboot test and make loop more robus
>   arm coreboot: Use common directory path
>   travis: Add smoke tests for arm and aarch64
>   travis: Add ARM thumb target to tests
>   travis: Test module loading from tftp as well
>   travis: Disable MIPS target
>   travis: Disable IA64 target
>
>  .travis.yml    | 67 +++++++++++++++++++++++++++++++++++++++++++++-------------
>  util/mkimage.c |  4 ++--
>  2 files changed, 54 insertions(+), 17 deletions(-)
>
> --
> 2.16.4
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel



-- 
Regards
Vladimir 'phcoder' Serbinenko

Re: [PATCH v3 00/10] Travis fixes

[John Paul Adrian Glaubitz]

On 7/18/19 6:58 PM, Vladimir 'phcoder' Serbinenko wrote:
> LGTM for the whole series. I do care about mips and ia64 but I think
> it's fine to disable them for now and iterate from there
Unless I am missing something, ia64 should be fine as the issue in question
has been fixed. So, it's only mips that needs to be disabled due to the toolchain
being used not being able to build all mips targets.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913