[PULL 0/5] edk2-next patches for 2019-10-07

[PULL 0/5] edk2-next patches for 2019-10-07

[Philippe Mathieu-Daudé]

The following changes since commit 9e5319ca52a5b9e84d55ad9c36e2c0b317a122bb:

  Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2019-10-04 18:32:34 +0100)

are available in the Git repository at:

  https://gitlab.com/philmd/qemu.git tags/edk2-next-20191007

for you to fetch changes up to 037973bb0d2b1a3c8618ccf41caa4da3666588c3:

  edk2 build scripts: work around TianoCore#1607 without forcing Python 2 (2019-10-07 15:14:15 +0200)

----------------------------------------------------------------
Improve scripts relying on the EDK2 submodule,
drop Python2 dependency in EDK2 build scripts.

----------------------------------------------------------------

Laszlo Ersek (2):
  edk2 build scripts: honor external BaseTools flags with
    uefi-test-tools
  edk2 build scripts: work around TianoCore#1607 without forcing Python
    2

Michael Roth (2):
  make-release: pull in edk2 submodules so we can build it from tarballs
  roms/Makefile.edk2: don't pull in submodules when building from
    tarball

Philippe Mathieu-Daudé (1):
  roms: Add a 'make help' target alias

 roms/Makefile                  |  3 ++-
 roms/Makefile.edk2             |  7 ++++++-
 roms/edk2-build.sh             |  4 ++--
 roms/edk2-funcs.sh             | 17 +++++++++++++++++
 scripts/make-release           |  8 ++++++++
 tests/uefi-test-tools/Makefile |  5 ++++-
 tests/uefi-test-tools/build.sh |  6 ++++--
 7 files changed, 43 insertions(+), 7 deletions(-)

-- 
2.21.0

[PULL 1/5] make-release: pull in edk2 submodules so we can build it from tarballs

[Philippe Mathieu-Daudé]

From: Michael Roth <mdroth@linux.vnet.ibm.com>

The `make efi` target added by 536d2173 is built from the roms/edk2
submodule, which in turn relies on additional submodules nested under
roms/edk2.

The make-release script currently only pulls in top-level submodules,
so these nested submodules are missing in the resulting tarball.

We could try to address this situation more generally by recursively
pulling in all submodules, but this doesn't necessarily ensure the
end-result will build properly (this case also required other changes).

Additionally, due to the nature of submodules, we may not always have
control over how these sorts of things are dealt with, so for now we
continue to handle it on a case-by-case in the make-release script.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Bruce Rogers <brogers@suse.com>
Cc: qemu-stable@nongnu.org # v4.1.0
Reported-by: Bruce Rogers <brogers@suse.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Message-Id: <20190912231202.12327-2-mdroth@linux.vnet.ibm.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 scripts/make-release | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/scripts/make-release b/scripts/make-release
index b4af9c9e52..a2a8cda33c 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -20,6 +20,14 @@ git checkout "v${version}"
 git submodule update --init
 (cd roms/seabios && git describe --tags --long --dirty > .version)
 (cd roms/skiboot && ./make_version.sh > .version)
+# Fetch edk2 submodule's submodules, since it won't have access to them via
+# the tarball later.
+#
+# A more uniform way to handle this sort of situation would be nice, but we
+# don't necessarily have much control over how a submodule handles its
+# submodule dependencies, so we continue to handle these on a case-by-case
+# basis for now.
+(cd roms/edk2 && git submodule update --init)
 popd
 tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
-- 
2.21.0

[PULL 2/5] roms/Makefile.edk2: don't pull in submodules when building from tarball

[Philippe Mathieu-Daudé]

From: Michael Roth <mdroth@linux.vnet.ibm.com>

Currently the `make efi` target pulls submodules nested under the
roms/edk2 submodule as dependencies. However, when we attempt to build
from a tarball this fails since we are no longer in a git tree.

A preceding patch will pre-populate these submodules in the tarball,
so assume this build dependency is only needed when building from a
git tree.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Bruce Rogers <brogers@suse.com>
Cc: qemu-stable@nongnu.org # v4.1.0
Reported-by: Bruce Rogers <brogers@suse.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Message-Id: <20190912231202.12327-3-mdroth@linux.vnet.ibm.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/Makefile.edk2 | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/roms/Makefile.edk2 b/roms/Makefile.edk2
index c2f2ff59d5..33a074d3a4 100644
--- a/roms/Makefile.edk2
+++ b/roms/Makefile.edk2
@@ -46,8 +46,13 @@ all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
 # files.
 .INTERMEDIATE: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd)
 
+# Fetch edk2 submodule's submodules. If it is not in a git tree, assume
+# we're building from a tarball and that they've already been fetched by
+# make-release/tarball scripts.
 submodules:
-	cd edk2 && git submodule update --init --force
+	if test -d edk2/.git; then \
+		cd edk2 && git submodule update --init --force; \
+	fi
 
 # See notes on the ".NOTPARALLEL" target and the "+" indicator in
 # "tests/uefi-test-tools/Makefile".
-- 
2.21.0

[PULL 3/5] roms: Add a 'make help' target alias

[Philippe Mathieu-Daudé]

Various C projects provide a 'make help' target. Our root directory
does so. The roms/ directory lacks a such rule, but already displays
a help output when the default target is called.
Add a 'help' target aliased to the default one, to avoid:

  $ make -C roms help
  make: *** No rule to make target 'help'.  Stop.

Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20190920171159.18633-1-philmd@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/roms/Makefile b/roms/Makefile
index 6cf07d3b44..3ffd13cc7e 100644
--- a/roms/Makefile
+++ b/roms/Makefile
@@ -51,7 +51,7 @@ SEABIOS_EXTRAVERSION="-prebuilt.qemu.org"
 #
 EDK2_EFIROM = edk2/BaseTools/Source/C/bin/EfiRom
 
-default:
+default help:
 	@echo "nothing is build by default"
 	@echo "available build targets:"
 	@echo "  bios               -- update bios.bin (seabios)"
-- 
2.21.0

[PULL 4/5] edk2 build scripts: honor external BaseTools flags with uefi-test-tools

[Philippe Mathieu-Daudé]

From: Laszlo Ersek <lersek@redhat.com>

Unify the recipe for "build-edk2-tools" in
"tests/uefi-test-tools/Makefile" with the recipe for "edk2-basetools" in
"roms/Makefile".

Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20190920083808.21399-2-lersek@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 tests/uefi-test-tools/Makefile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile
index 1d78bc14d5..7e0177d733 100644
--- a/tests/uefi-test-tools/Makefile
+++ b/tests/uefi-test-tools/Makefile
@@ -99,7 +99,9 @@ Build/bios-tables-test.%.efi: build-edk2-tools
 	+./build.sh $(edk2_dir) BiosTablesTest $* $@
 
 build-edk2-tools:
-	$(MAKE) -C $(edk2_dir)/BaseTools
+	$(MAKE) -C $(edk2_dir)/BaseTools \
+		EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \
+		EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'
 
 clean:
 	rm -rf Build Conf log
-- 
2.21.0

[PULL 5/5] edk2 build scripts: work around TianoCore#1607 without forcing Python 2

[Philippe Mathieu-Daudé]

From: Laszlo Ersek <lersek@redhat.com>

It turns out that forcing python2 for running the edk2 "build" utility is
neither necessary nor sufficient.

Forcing python2 is not sufficient for two reasons:

- QEMU is moving away from python2, with python2 nearing EOL,

- according to my most recent testing, the lacking dependency information
  in the makefiles that are generated by edk2's "build" utility can cause
  parallel build failures even when "build" is executed by python2.

And forcing python2 is not necessary because we can still return to the
original idea of filtering out jobserver-related options from MAKEFLAGS.
So do that.

While at it, cut short edk2's auto-detection of the python3.* minor
version, by setting PYTHON_COMMAND to "python3" (which we expect to be
available wherever we intend to build edk2).

With this patch, the guest UEFI binaries that are used as part of the BIOS
tables test, and the OVMF and ArmVirtQemu platform firmwares, will be
built strictly in a single job, regardless of an outermost "-jN" make
option. Alas, there appears to be no reliable way to build edk2 in an
(outer make, inner make) environment, with a jobserver enabled.

Cc: Eduardo Habkost <ehabkost@redhat.com>
Cc: John Snow <jsnow@redhat.com>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Reported-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20190920083808.21399-3-lersek@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/Makefile                  |  1 +
 roms/edk2-build.sh             |  4 ++--
 roms/edk2-funcs.sh             | 17 +++++++++++++++++
 tests/uefi-test-tools/Makefile |  1 +
 tests/uefi-test-tools/build.sh |  6 ++++--
 5 files changed, 25 insertions(+), 4 deletions(-)

diff --git a/roms/Makefile b/roms/Makefile
index 3ffd13cc7e..da4efa47a9 100644
--- a/roms/Makefile
+++ b/roms/Makefile
@@ -141,6 +141,7 @@ build-efi-roms: build-pxe-roms
 #
 edk2-basetools:
 	$(MAKE) -C edk2/BaseTools \
+		PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \
 		EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \
 		EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'
 
diff --git a/roms/edk2-build.sh b/roms/edk2-build.sh
index 4f46f8a6a2..d5391c7637 100755
--- a/roms/edk2-build.sh
+++ b/roms/edk2-build.sh
@@ -27,8 +27,7 @@ shift $num_args
 
 cd edk2
 
-# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607>.
-export PYTHON_COMMAND=python2
+export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3}
 
 # Source "edksetup.sh" carefully.
 set +e +u +C
@@ -43,6 +42,7 @@ fi
 # any), for the edk2 "build" utility.
 source ../edk2-funcs.sh
 edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target")
+MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS")
 edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS")
 qemu_edk2_set_cross_env "$emulation_target"
 
diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh
index a9fae7ee89..3f4485b201 100644
--- a/roms/edk2-funcs.sh
+++ b/roms/edk2-funcs.sh
@@ -251,3 +251,20 @@ qemu_edk2_get_thread_count()
     printf '1\n'
   fi
 }
+
+
+# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607> by
+# filtering jobserver-related flags out of MAKEFLAGS. Print the result to the
+# standard output.
+#
+# Parameters:
+#   $1: the value of the MAKEFLAGS variable
+qemu_edk2_quirk_tianocore_1607()
+{
+  local makeflags="$1"
+
+  printf %s "$makeflags" \
+  | LC_ALL=C sed --regexp-extended \
+      --expression='s/--jobserver-(auth|fds)=[0-9]+,[0-9]+//' \
+      --expression='s/-j([0-9]+)?//'
+}
diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile
index 7e0177d733..1dcddcdbba 100644
--- a/tests/uefi-test-tools/Makefile
+++ b/tests/uefi-test-tools/Makefile
@@ -100,6 +100,7 @@ Build/bios-tables-test.%.efi: build-edk2-tools
 
 build-edk2-tools:
 	$(MAKE) -C $(edk2_dir)/BaseTools \
+		PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \
 		EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \
 		EXTRA_LDFLAGS='$(EDK2_BASETOOLS_LDFLAGS)'
 
diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh
index 8aa7935c43..3b78f30840 100755
--- a/tests/uefi-test-tools/build.sh
+++ b/tests/uefi-test-tools/build.sh
@@ -29,8 +29,7 @@ export PACKAGES_PATH=$(realpath -- "$edk2_dir")
 export WORKSPACE=$PWD
 mkdir -p Conf
 
-# Work around <https://bugzilla.tianocore.org/show_bug.cgi?id=1607>.
-export PYTHON_COMMAND=python2
+export PYTHON_COMMAND=${EDK2_PYTHON_COMMAND:-python3}
 
 # Source "edksetup.sh" carefully.
 set +e +u +C
@@ -46,12 +45,15 @@ fi
 source "$edk2_dir/../edk2-funcs.sh"
 edk2_arch=$(qemu_edk2_get_arch "$emulation_target")
 edk2_toolchain=$(qemu_edk2_get_toolchain "$emulation_target")
+MAKEFLAGS=$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS")
+edk2_thread_count=$(qemu_edk2_get_thread_count "$MAKEFLAGS")
 qemu_edk2_set_cross_env "$emulation_target"
 
 # Build the UEFI binary
 mkdir -p log
 build \
   --arch="$edk2_arch" \
+  -n "$edk2_thread_count" \
   --buildtarget=DEBUG \
   --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \
   --tagname="$edk2_toolchain" \
-- 
2.21.0

Re: [PULL 0/5] edk2-next patches for 2019-10-07

[Peter Maydell]

On Mon, 7 Oct 2019 at 14:37, Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>
> The following changes since commit 9e5319ca52a5b9e84d55ad9c36e2c0b317a122bb:
>
>   Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging (2019-10-04 18:32:34 +0100)
>
> are available in the Git repository at:
>
>   https://gitlab.com/philmd/qemu.git tags/edk2-next-20191007
>
> for you to fetch changes up to 037973bb0d2b1a3c8618ccf41caa4da3666588c3:
>
>   edk2 build scripts: work around TianoCore#1607 without forcing Python 2 (2019-10-07 15:14:15 +0200)
>
> ----------------------------------------------------------------
> Improve scripts relying on the EDK2 submodule,
> drop Python2 dependency in EDK2 build scripts.
>
> ----------------------------------------------------------------


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/4.2
for any user-visible changes.

-- PMM