[Qemu-devel] [PATCH 0/5] travis-ci: Build EDK2 roms

[Qemu-devel] [PATCH 0/5] travis-ci: Build EDK2 roms

[Philippe Mathieu-Daudé]

Hi,

This series add a job to Travis CI to build the last EDK2 stable
release roms within QEMU.
This job is only triggered if a commit matches 'EDK2' or during
QEMU releases (when tags are pushed).

Job output: https://travis-ci.org/philmd/qemu/jobs/504460932
Duration: 33 min 51 sec (output 5080 lines)

Based on Laszlo's series "bundle edk2 platform firmware with QEMU":
https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02846.html

Please review,

Phil.

Based-on: 20190309004826.9027-1-lersek@redhat.com

Philippe Mathieu-Daudé (5):
  roms/edk2: Avoid bashism in Makefile 'shell' function
  roms/edk2: Avoid bashism in script
  roms/edk2: Pass extra arguments to the build script via
    EDK2_BUILD_OPTIONS
  NOTFORMERGE roms/edk2: Use arm-linux-gnueabihf-gcc on Debian based
    distribs
  .travis.yml: Build and install EDK2 roms

 .travis.yml        | 21 +++++++++++++++++++++
 roms/Makefile.edk2 |  8 +++++++-
 roms/edk2-funcs.sh |  9 +++++++--
 3 files changed, 35 insertions(+), 3 deletions(-)

-- 
2.20.1

[Qemu-devel] [PATCH 1/5] roms/edk2: Avoid bashism in Makefile 'shell' function

[Philippe Mathieu-Daudé]

The shell builtin 'source' command is a bash extension.
Use the '.' portable command.

This fix building when /bin/sh is dash (Ubuntu 16):

  $ make -C roms efi
  [...]
  Fd File Name:QEMU_EFI
  (/source/qemu/roms/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd)
  Fd File Name:QEMU_VARS
  (/source/qemu/roms/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/QEMU_VARS.fd)
  GUID cross reference file can be found at
  /source/qemu/roms/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Guid.xref
  - Done -
  Build end time: 16:33:29, Mar.09 2019
  Build total time: 00:03:35
  cp edk2/Build/ArmVirtQemu-AARCH64/DEBUG_/FV/QEMU_EFI.fd \
  	../pc-bios/edk2-aarch64-code.fd
  cp: cannot stat 'edk2/Build/ArmVirtQemu-AARCH64/DEBUG_/FV/QEMU_EFI.fd':
  No such file or directory
  Makefile.edk2:45: recipe for target '../pc-bios/edk2-aarch64-code.fd' failed
  make[1]: *** [../pc-bios/edk2-aarch64-code.fd] Error 1
  make[1]: Leaving directory '/source/qemu/roms'
  Makefile:148: recipe for target 'efi' failed
  make: *** [efi] Error 2
  make: Leaving directory '/source/qemu/roms'

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/Makefile.edk2 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/roms/Makefile.edk2 b/roms/Makefile.edk2
index ad6fff044c..6c5e9b9eea 100644
--- a/roms/Makefile.edk2
+++ b/roms/Makefile.edk2
@@ -11,7 +11,7 @@
 # THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT
 # WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
 
-toolchain = $(shell source ./edk2-funcs.sh && qemu_edk2_get_toolchain $(1))
+toolchain = $(shell . ./edk2-funcs.sh && qemu_edk2_get_toolchain $(1))
 
 licenses := \
 	edk2/License.txt \
-- 
2.20.1

[Qemu-devel] [PATCH 2/5] roms/edk2: Avoid bashism in script

[Philippe Mathieu-Daudé]

Use the POSIX '=' to evaluate equality.

This fixes building the roms with a dash shell:

  $ make -C roms efi
  [...]
  cp edk2/Build/ArmVirtQemu-ARM/DEBUG_GCC5/FV/QEMU_EFI.fd \
        ../pc-bios/edk2-arm-code.fd
  truncate --size=64M ../pc-bios/edk2-arm-code.fd
  /bin/sh: 111: [: i686: unexpected operator
  /bin/sh: 112: [: i686: unexpected operator
  /bin/sh: 116: [: i686: unexpected operator

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/edk2-funcs.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh
index 7fc62f074c..402fc126fb 100644
--- a/roms/edk2-funcs.sh
+++ b/roms/edk2-funcs.sh
@@ -108,8 +108,8 @@ qemu_edk2_get_cross_prefix()
 
   host_arch=$(uname -m)
 
-  if [ "$gcc_arch" == "$host_arch" ] ||
-     ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then
+  if [ "$gcc_arch" = "$host_arch" ] ||
+     ( [ "$gcc_arch" = i686 ] && [ "$host_arch" = x86_64 ] ); then
     # no cross-compiler needed
     :
   else
-- 
2.20.1

[Qemu-devel] [PATCH 3/5] roms/edk2: Pass extra arguments to the build script via EDK2_BUILD_OPTIONS

[Philippe Mathieu-Daudé]

The current EDK2 options produce a very verbose output.
When building the roms on Continuous Integration systems (such
Travis CI) the system limit is quickly reached.

Add an variable to optionally pass arguments to the edk2-build.sh
script.

Example of use:

  $ EDK2_BUILD_OPTIONS="--quiet --silent" make -C roms efi
  ...

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/Makefile.edk2 | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/roms/Makefile.edk2 b/roms/Makefile.edk2
index 6c5e9b9eea..70f84b20df 100644
--- a/roms/Makefile.edk2
+++ b/roms/Makefile.edk2
@@ -44,6 +44,7 @@ submodules:
 ../pc-bios/edk2-aarch64-code.fd: submodules
 	+./edk2-build.sh \
 		aarch64 \
+		$(EDK2_BUILD_OPTIONS) \
 		--arch=AARCH64 \
 		--platform=ArmVirtPkg/ArmVirtQemu.dsc \
 		-D NETWORK_IP6_ENABLE \
@@ -55,6 +56,7 @@ submodules:
 ../pc-bios/edk2-arm-code.fd: submodules
 	+./edk2-build.sh \
 		arm \
+		$(EDK2_BUILD_OPTIONS) \
 		--arch=ARM \
 		--platform=ArmVirtPkg/ArmVirtQemu.dsc \
 		-D NETWORK_IP6_ENABLE \
@@ -66,6 +68,7 @@ submodules:
 ../pc-bios/edk2-i386-code.fd: submodules
 	+./edk2-build.sh \
 		i386 \
+		$(EDK2_BUILD_OPTIONS) \
 		--arch=IA32 \
 		--platform=OvmfPkg/OvmfPkgIa32.dsc \
 		-D NETWORK_IP6_ENABLE \
@@ -78,6 +81,7 @@ submodules:
 ../pc-bios/edk2-i386-secure-code.fd: submodules
 	+./edk2-build.sh \
 		i386 \
+		$(EDK2_BUILD_OPTIONS) \
 		--arch=IA32 \
 		--platform=OvmfPkg/OvmfPkgIa32.dsc \
 		-D NETWORK_IP6_ENABLE \
@@ -92,6 +96,7 @@ submodules:
 ../pc-bios/edk2-x86_64-code.fd: submodules
 	+./edk2-build.sh \
 		x86_64 \
+		$(EDK2_BUILD_OPTIONS) \
 		--arch=X64 \
 		--platform=OvmfPkg/OvmfPkgX64.dsc \
 		-D NETWORK_IP6_ENABLE \
@@ -104,6 +109,7 @@ submodules:
 ../pc-bios/edk2-x86_64-secure-code.fd: submodules
 	+./edk2-build.sh \
 		x86_64 \
+		$(EDK2_BUILD_OPTIONS) \
 		--arch=IA32 \
 		--arch=X64 \
 		--platform=OvmfPkg/OvmfPkgIa32X64.dsc \
-- 
2.20.1

[Qemu-devel] [PATCH 4/5] NOTFORMERGE roms/edk2: Use arm-linux-gnueabihf-gcc on Debian based distribs

[Philippe Mathieu-Daudé]

Do not enforce GCCx_ARM_PREFIX=arm-linux-gnu- on Debian based
distributions.
This is a kludge not meant for merging!

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/edk2-funcs.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh
index 402fc126fb..2f1fa2af1a 100644
--- a/roms/edk2-funcs.sh
+++ b/roms/edk2-funcs.sh
@@ -113,6 +113,11 @@ qemu_edk2_get_cross_prefix()
     # no cross-compiler needed
     :
   else
+    if [ -f /etc/lsb-release ] && [ "$gcc_arch" = arm ]; then
+      # XXX kludge for Debian-based distribs
+      printf 'arm-linux-gnueabihf-\n'
+      return 0
+    fi
     printf '%s-linux-gnu-\n' "$gcc_arch"
   fi
 }
-- 
2.20.1

[Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms

[Philippe Mathieu-Daudé]

Add a job to build and install the EDK2 platform firmware binaries.

This job is only triggered if the last commit matches the EDK2
name (case insensitive), or when tag are created (such releases
or release candidates).

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .travis.yml | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index e942175dd3..628cc52c99 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -258,3 +258,24 @@ matrix:
     - env:
         - CONFIG="--disable-system"
         - TEST_CMD="make -j3 check-tcg V=1"
+
+
+    # EDK2 roms builds
+    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
+      env:
+        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
+        - EDK2_BUILD_OPTIONS="--quiet --silent"
+      script:
+        - git submodule update --init roms/edk2
+        - make -j3
+        - make -C roms efi -j2
+        - make install
+      addons:
+        apt:
+          packages:
+            - dos2unix
+            - gcc-aarch64-linux-gnu
+            - gcc-arm-linux-gnueabihf
+            - iasl
+            - nasm
+            - uuid-dev
-- 
2.20.1

Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms

[Philippe Mathieu-Daudé]

Hi Laszlo,

On 3/12/19 5:29 PM, Laszlo Ersek wrote:
> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>> Add a job to build and install the EDK2 platform firmware binaries.
>>
>> This job is only triggered if the last commit matches the EDK2
>> name (case insensitive), or when tag are created (such releases
>> or release candidates).
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  .travis.yml | 21 +++++++++++++++++++++
>>  1 file changed, 21 insertions(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index e942175dd3..628cc52c99 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -258,3 +258,24 @@ matrix:
>>      - env:
>>          - CONFIG="--disable-system"
>>          - TEST_CMD="make -j3 check-tcg V=1"
>> +
>> +
>> +    # EDK2 roms builds
>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>> +      env:
>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>> +      script:
>> +        - git submodule update --init roms/edk2
>> +        - make -j3
>> +        - make -C roms efi -j2
>> +        - make install
>> +      addons:
>> +        apt:
>> +          packages:
>> +            - dos2unix
>> +            - gcc-aarch64-linux-gnu
>> +            - gcc-arm-linux-gnueabihf
>> +            - iasl
>> +            - nasm
>> +            - uuid-dev
>>
> 
> Regardless of what problem we're trying to address with "--quiet
> --silent", those options are wrong. You certainly want detailed build
> logs for the case a CI job fails (at build time or at runtime).

On Travis we get:

The job exceeded the maximum log length, and has been terminated.
(https://travis-ci.org/philmd/qemu/jobs/545329905)

So I moved to GitLab, but we still get:

"Job's log exceeded limit of 4194304 bytes."
(https://gitlab.com/philmd/qemu/-/jobs/230772314)

Regarding the options to pass to edk2-build.sh,

$ build --help
  -j LOGFILE, --log=LOGFILE
                   Put log in specified file as well as on console.
  -s, --silent     Make use of silent mode of (n)make.
  -q, --quiet      Disable all messages except FATAL ERRORS.
  -v, --verbose    Turn on verbose output with informational messages
                   printed, including library instances selected, final
                   dependency expression, and warning messages, etc.

'--log' duplicate the output, and I don't want to reduce the log
details, so I understand I should use:

  ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log

Is that correct? But then I'd need to modify Makefile.edk2 to redirect
stdout.

> The reason why I only include DEBUG firmware builds in the edk2 bundling
> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
> and as such they are 100% unsupportable in my book. Bugs in software are
> the norm, not the exception, so we should allow (even force) the user
> (and remote systems) to provide as much information as they can.

Sure, we have the same book here ;)

Regards,

Phil.

Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms

[Laszlo Ersek]

On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
> Hi Laszlo,
> 
> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>
>>> This job is only triggered if the last commit matches the EDK2
>>> name (case insensitive), or when tag are created (such releases
>>> or release candidates).
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> ---
>>>  .travis.yml | 21 +++++++++++++++++++++
>>>  1 file changed, 21 insertions(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index e942175dd3..628cc52c99 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -258,3 +258,24 @@ matrix:
>>>      - env:
>>>          - CONFIG="--disable-system"
>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>> +
>>> +
>>> +    # EDK2 roms builds
>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>> +      env:
>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>> +      script:
>>> +        - git submodule update --init roms/edk2
>>> +        - make -j3
>>> +        - make -C roms efi -j2
>>> +        - make install
>>> +      addons:
>>> +        apt:
>>> +          packages:
>>> +            - dos2unix
>>> +            - gcc-aarch64-linux-gnu
>>> +            - gcc-arm-linux-gnueabihf
>>> +            - iasl
>>> +            - nasm
>>> +            - uuid-dev
>>>
>>
>> Regardless of what problem we're trying to address with "--quiet
>> --silent", those options are wrong. You certainly want detailed build
>> logs for the case a CI job fails (at build time or at runtime).
> 
> On Travis we get:
> 
> The job exceeded the maximum log length, and has been terminated.
> (https://travis-ci.org/philmd/qemu/jobs/545329905)
> 
> So I moved to GitLab, but we still get:
> 
> "Job's log exceeded limit of 4194304 bytes."
> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
> 
> Regarding the options to pass to edk2-build.sh,
> 
> $ build --help
>   -j LOGFILE, --log=LOGFILE
>                    Put log in specified file as well as on console.
>   -s, --silent     Make use of silent mode of (n)make.
>   -q, --quiet      Disable all messages except FATAL ERRORS.
>   -v, --verbose    Turn on verbose output with informational messages
>                    printed, including library instances selected, final
>                    dependency expression, and warning messages, etc.
> 
> '--log' duplicate the output, and I don't want to reduce the log
> details, so I understand I should use:
> 
>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
> 
> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
> stdout.

Would it be possible to invoke the outermost make like this?

  make -C roms -j2 efi >make.roms.efi.log 2>&1 \
  || ( tail -c 2M make.roms.efi.log; false )

Thanks
Laszlo

> 
>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>> and as such they are 100% unsupportable in my book. Bugs in software are
>> the norm, not the exception, so we should allow (even force) the user
>> (and remote systems) to provide as much information as they can.
> 
> Sure, we have the same book here ;)
> 
> Regards,
> 
> Phil.
> 

Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms

[Laszlo Ersek]

On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
> Hi Laszlo,
> 
> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>
>>> This job is only triggered if the last commit matches the EDK2
>>> name (case insensitive), or when tag are created (such releases
>>> or release candidates).
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>> ---
>>>  .travis.yml | 21 +++++++++++++++++++++
>>>  1 file changed, 21 insertions(+)
>>>
>>> diff --git a/.travis.yml b/.travis.yml
>>> index e942175dd3..628cc52c99 100644
>>> --- a/.travis.yml
>>> +++ b/.travis.yml
>>> @@ -258,3 +258,24 @@ matrix:
>>>      - env:
>>>          - CONFIG="--disable-system"
>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>> +
>>> +
>>> +    # EDK2 roms builds
>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>> +      env:
>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>> +      script:
>>> +        - git submodule update --init roms/edk2
>>> +        - make -j3
>>> +        - make -C roms efi -j2
>>> +        - make install
>>> +      addons:
>>> +        apt:
>>> +          packages:
>>> +            - dos2unix
>>> +            - gcc-aarch64-linux-gnu
>>> +            - gcc-arm-linux-gnueabihf
>>> +            - iasl
>>> +            - nasm
>>> +            - uuid-dev
>>>
>>
>> Regardless of what problem we're trying to address with "--quiet
>> --silent", those options are wrong. You certainly want detailed build
>> logs for the case a CI job fails (at build time or at runtime).
> 
> On Travis we get:
> 
> The job exceeded the maximum log length, and has been terminated.
> (https://travis-ci.org/philmd/qemu/jobs/545329905)
> 
> So I moved to GitLab, but we still get:
> 
> "Job's log exceeded limit of 4194304 bytes."
> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
> 
> Regarding the options to pass to edk2-build.sh,
> 
> $ build --help
>   -j LOGFILE, --log=LOGFILE
>                    Put log in specified file as well as on console.
>   -s, --silent     Make use of silent mode of (n)make.
>   -q, --quiet      Disable all messages except FATAL ERRORS.
>   -v, --verbose    Turn on verbose output with informational messages
>                    printed, including library instances selected, final
>                    dependency expression, and warning messages, etc.
> 
> '--log' duplicate the output, and I don't want to reduce the log
> details, so I understand I should use:
> 
>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
> 
> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
> stdout.

Would it be possible to invoke the outermost make like this?

  make -C roms -j2 efi >make.roms.efi.log 2>&1 \
  || ( tail -c 2M make.roms.efi.log; false )

Thanks
Laszlo

> 
>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>> and as such they are 100% unsupportable in my book. Bugs in software are
>> the norm, not the exception, so we should allow (even force) the user
>> (and remote systems) to provide as much information as they can.
> 
> Sure, we have the same book here ;)
> 
> Regards,
> 
> Phil.
> 

Re: [Qemu-devel] [PATCH 5/5] .travis.yml: Build and install EDK2 roms

[Alex Bennée]

Laszlo Ersek <lersek@redhat.com> writes:

> On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
>> Hi Laszlo,
>>
>> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>>
>>>> This job is only triggered if the last commit matches the EDK2
>>>> name (case insensitive), or when tag are created (such releases
>>>> or release candidates).
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>> ---
>>>>  .travis.yml | 21 +++++++++++++++++++++
>>>>  1 file changed, 21 insertions(+)
>>>>
>>>> diff --git a/.travis.yml b/.travis.yml
>>>> index e942175dd3..628cc52c99 100644
>>>> --- a/.travis.yml
>>>> +++ b/.travis.yml
>>>> @@ -258,3 +258,24 @@ matrix:
>>>>      - env:
>>>>          - CONFIG="--disable-system"
>>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>>> +
>>>> +
>>>> +    # EDK2 roms builds
>>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>>> +      env:
>>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>>> +      script:
>>>> +        - git submodule update --init roms/edk2
>>>> +        - make -j3
>>>> +        - make -C roms efi -j2
>>>> +        - make install
>>>> +      addons:
>>>> +        apt:
>>>> +          packages:
>>>> +            - dos2unix
>>>> +            - gcc-aarch64-linux-gnu
>>>> +            - gcc-arm-linux-gnueabihf
>>>> +            - iasl
>>>> +            - nasm
>>>> +            - uuid-dev
>>>>
>>>
>>> Regardless of what problem we're trying to address with "--quiet
>>> --silent", those options are wrong. You certainly want detailed build
>>> logs for the case a CI job fails (at build time or at runtime).
>>
>> On Travis we get:
>>
>> The job exceeded the maximum log length, and has been terminated.
>> (https://travis-ci.org/philmd/qemu/jobs/545329905)
>>
>> So I moved to GitLab, but we still get:
>>
>> "Job's log exceeded limit of 4194304 bytes."
>> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
>>
>> Regarding the options to pass to edk2-build.sh,
>>
>> $ build --help
>>   -j LOGFILE, --log=LOGFILE
>>                    Put log in specified file as well as on console.
>>   -s, --silent     Make use of silent mode of (n)make.
>>   -q, --quiet      Disable all messages except FATAL ERRORS.
>>   -v, --verbose    Turn on verbose output with informational messages
>>                    printed, including library instances selected, final
>>                    dependency expression, and warning messages, etc.
>>
>> '--log' duplicate the output, and I don't want to reduce the log
>> details, so I understand I should use:
>>
>>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
>>
>> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
>> stdout.
>
> Would it be possible to invoke the outermost make like this?
>
>   make -C roms -j2 efi >make.roms.efi.log 2>&1 \
>   || ( tail -c 2M make.roms.efi.log; false )

Or something like:

     after_failure:
       - tail -c 2M make.roms.efi.log

as Cleber suggested for his acceptance tests. Generally we want to try
and make the builds less noisy and only echo relevant details when it
fails. However we've tended to turn up the noise so we can debug stuff
and that is just painful to browser on the Travis website.


>
> Thanks
> Laszlo
>
>>
>>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>>> and as such they are 100% unsupportable in my book. Bugs in software are
>>> the norm, not the exception, so we should allow (even force) the user
>>> (and remote systems) to provide as much information as they can.
>>
>> Sure, we have the same book here ;)
>>
>> Regards,
>>
>> Phil.
>>


--
Alex Bennée

Re: [PATCH 5/5] .travis.yml: Build and install EDK2 roms

[Philippe Mathieu-Daudé]

On 6/14/19 8:44 PM, Alex Bennée wrote:
> Laszlo Ersek <lersek@redhat.com> writes:
>> On 06/13/19 18:59, Philippe Mathieu-Daudé wrote:
>>> Hi Laszlo,
>>>
>>> On 3/12/19 5:29 PM, Laszlo Ersek wrote:
>>>> On 03/11/19 01:30, Philippe Mathieu-Daudé wrote:
>>>>> Add a job to build and install the EDK2 platform firmware binaries.
>>>>>
>>>>> This job is only triggered if the last commit matches the EDK2
>>>>> name (case insensitive), or when tag are created (such releases
>>>>> or release candidates).
>>>>>
>>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>>> ---
>>>>>  .travis.yml | 21 +++++++++++++++++++++
>>>>>  1 file changed, 21 insertions(+)
>>>>>
>>>>> diff --git a/.travis.yml b/.travis.yml
>>>>> index e942175dd3..628cc52c99 100644
>>>>> --- a/.travis.yml
>>>>> +++ b/.travis.yml
>>>>> @@ -258,3 +258,24 @@ matrix:
>>>>>      - env:
>>>>>          - CONFIG="--disable-system"
>>>>>          - TEST_CMD="make -j3 check-tcg V=1"
>>>>> +
>>>>> +
>>>>> +    # EDK2 roms builds
>>>>> +    - if: tag IS present OR commit_message =~ /(edk2|EDK2)/
>>>>> +      env:
>>>>> +        - CONFIG="--disable-system --disable-user --prefix=$PWD/dist"
>>>>> +        - EDK2_BUILD_OPTIONS="--quiet --silent"
>>>>> +      script:
>>>>> +        - git submodule update --init roms/edk2
>>>>> +        - make -j3
>>>>> +        - make -C roms efi -j2
>>>>> +        - make install
>>>>> +      addons:
>>>>> +        apt:
>>>>> +          packages:
>>>>> +            - dos2unix
>>>>> +            - gcc-aarch64-linux-gnu
>>>>> +            - gcc-arm-linux-gnueabihf
>>>>> +            - iasl
>>>>> +            - nasm
>>>>> +            - uuid-dev
>>>>>
>>>>
>>>> Regardless of what problem we're trying to address with "--quiet
>>>> --silent", those options are wrong. You certainly want detailed build
>>>> logs for the case a CI job fails (at build time or at runtime).
>>>
>>> On Travis we get:
>>>
>>> The job exceeded the maximum log length, and has been terminated.
>>> (https://travis-ci.org/philmd/qemu/jobs/545329905)
>>>
>>> So I moved to GitLab, but we still get:
>>>
>>> "Job's log exceeded limit of 4194304 bytes."
>>> (https://gitlab.com/philmd/qemu/-/jobs/230772314)
>>>
>>> Regarding the options to pass to edk2-build.sh,
>>>
>>> $ build --help
>>>   -j LOGFILE, --log=LOGFILE
>>>                    Put log in specified file as well as on console.
>>>   -s, --silent     Make use of silent mode of (n)make.
>>>   -q, --quiet      Disable all messages except FATAL ERRORS.
>>>   -v, --verbose    Turn on verbose output with informational messages
>>>                    printed, including library instances selected, final
>>>                    dependency expression, and warning messages, etc.
>>>
>>> '--log' duplicate the output, and I don't want to reduce the log
>>> details, so I understand I should use:
>>>
>>>   ./edk2-build.sh [...] --log=build.log >/dev/null || cat build.log
>>>
>>> Is that correct? But then I'd need to modify Makefile.edk2 to redirect
>>> stdout.
>>
>> Would it be possible to invoke the outermost make like this?
>>
>>   make -C roms -j2 efi >make.roms.efi.log 2>&1 \
>>   || ( tail -c 2M make.roms.efi.log; false )

The build gets killed:

  No output has been received in the last 10m0s, this potentially
  indicates a stalled build or something wrong with the build itself.
  Check the details on how to adjust your build configuration on:

https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received

  The build has been terminated

> Or something like:
> 
>      after_failure:
>        - tail -c 2M make.roms.efi.log
> 
> as Cleber suggested for his acceptance tests. Generally we want to try
> and make the builds less noisy and only echo relevant details when it
> fails. However we've tended to turn up the noise so we can debug stuff
> and that is just painful to browser on the Travis website.

I'll try that.

>>>> The reason why I only include DEBUG firmware builds in the edk2 bundling
>>>> series is similar -- RELEASE builds lack DEBUG messages and ASSERT()s,
>>>> and as such they are 100% unsupportable in my book. Bugs in software are
>>>> the norm, not the exception, so we should allow (even force) the user
>>>> (and remote systems) to provide as much information as they can.
>>>
>>> Sure, we have the same book here ;)
>>>
>>> Regards,
>>>
>>> Phil.
>>>
> --
> Alex Bennée
>