From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50803ECE58C for ; Mon, 7 Oct 2019 13:43:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 26760206BB for ; Mon, 7 Oct 2019 13:43:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 26760206BB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:44970 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iHTIr-0003uH-3D for qemu-devel@archiver.kernel.org; Mon, 07 Oct 2019 09:43:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:60178) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iHTBF-0003yd-QZ for qemu-devel@nongnu.org; Mon, 07 Oct 2019 09:36:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iHTBE-0007Qv-Co for qemu-devel@nongnu.org; Mon, 07 Oct 2019 09:36:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55630) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iHTBE-0007Qa-4h for qemu-devel@nongnu.org; Mon, 07 Oct 2019 09:36:04 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 597DC116BB23 for ; Mon, 7 Oct 2019 13:36:03 +0000 (UTC) Received: from x1w.redhat.com (unknown [10.40.206.44]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A724C60127; Mon, 7 Oct 2019 13:35:58 +0000 (UTC) From: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= To: qemu-devel@nongnu.org Subject: [PULL 5/5] edk2 build scripts: work around TianoCore#1607 without forcing Python 2 Date: Mon, 7 Oct 2019 15:35:40 +0200 Message-Id: <20191007133540.30623-6-philmd@redhat.com> In-Reply-To: <20191007133540.30623-1-philmd@redhat.com> References: <20191007133540.30623-1-philmd@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.65]); Mon, 07 Oct 2019 13:36:03 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Laszlo Ersek , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Laszlo Ersek It turns out that forcing python2 for running the edk2 "build" utility is neither necessary nor sufficient. Forcing python2 is not sufficient for two reasons: - QEMU is moving away from python2, with python2 nearing EOL, - according to my most recent testing, the lacking dependency information in the makefiles that are generated by edk2's "build" utility can cause parallel build failures even when "build" is executed by python2. And forcing python2 is not necessary because we can still return to the original idea of filtering out jobserver-related options from MAKEFLAGS. So do that. While at it, cut short edk2's auto-detection of the python3.* minor version, by setting PYTHON_COMMAND to "python3" (which we expect to be available wherever we intend to build edk2). With this patch, the guest UEFI binaries that are used as part of the BIO= S tables test, and the OVMF and ArmVirtQemu platform firmwares, will be built strictly in a single job, regardless of an outermost "-jN" make option. Alas, there appears to be no reliable way to build edk2 in an (outer make, inner make) environment, with a jobserver enabled. Cc: Eduardo Habkost Cc: John Snow Cc: Philippe Mathieu-Daud=C3=A9 Reported-by: John Snow Reviewed-by: Philippe Mathieu-Daud=C3=A9 Tested-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: John Snow Signed-off-by: Laszlo Ersek Message-Id: <20190920083808.21399-3-lersek@redhat.com> Signed-off-by: Philippe Mathieu-Daud=C3=A9 --- roms/Makefile | 1 + roms/edk2-build.sh | 4 ++-- roms/edk2-funcs.sh | 17 +++++++++++++++++ tests/uefi-test-tools/Makefile | 1 + tests/uefi-test-tools/build.sh | 6 ++++-- 5 files changed, 25 insertions(+), 4 deletions(-) diff --git a/roms/Makefile b/roms/Makefile index 3ffd13cc7e..da4efa47a9 100644 --- a/roms/Makefile +++ b/roms/Makefile @@ -141,6 +141,7 @@ build-efi-roms: build-pxe-roms # edk2-basetools: $(MAKE) -C edk2/BaseTools \ + PYTHON_COMMAND=3D$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS=3D'$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS=3D'$(EDK2_BASETOOLS_LDFLAGS)' =20 diff --git a/roms/edk2-build.sh b/roms/edk2-build.sh index 4f46f8a6a2..d5391c7637 100755 --- a/roms/edk2-build.sh +++ b/roms/edk2-build.sh @@ -27,8 +27,7 @@ shift $num_args =20 cd edk2 =20 -# Work around . -export PYTHON_COMMAND=3Dpython2 +export PYTHON_COMMAND=3D${EDK2_PYTHON_COMMAND:-python3} =20 # Source "edksetup.sh" carefully. set +e +u +C @@ -43,6 +42,7 @@ fi # any), for the edk2 "build" utility. source ../edk2-funcs.sh edk2_toolchain=3D$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=3D$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") edk2_thread_count=3D$(qemu_edk2_get_thread_count "$MAKEFLAGS") qemu_edk2_set_cross_env "$emulation_target" =20 diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh index a9fae7ee89..3f4485b201 100644 --- a/roms/edk2-funcs.sh +++ b/roms/edk2-funcs.sh @@ -251,3 +251,20 @@ qemu_edk2_get_thread_count() printf '1\n' fi } + + +# Work around by +# filtering jobserver-related flags out of MAKEFLAGS. Print the result t= o the +# standard output. +# +# Parameters: +# $1: the value of the MAKEFLAGS variable +qemu_edk2_quirk_tianocore_1607() +{ + local makeflags=3D"$1" + + printf %s "$makeflags" \ + | LC_ALL=3DC sed --regexp-extended \ + --expression=3D's/--jobserver-(auth|fds)=3D[0-9]+,[0-9]+//' \ + --expression=3D's/-j([0-9]+)?//' +} diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makef= ile index 7e0177d733..1dcddcdbba 100644 --- a/tests/uefi-test-tools/Makefile +++ b/tests/uefi-test-tools/Makefile @@ -100,6 +100,7 @@ Build/bios-tables-test.%.efi: build-edk2-tools =20 build-edk2-tools: $(MAKE) -C $(edk2_dir)/BaseTools \ + PYTHON_COMMAND=3D$${EDK2_PYTHON_COMMAND:-python3} \ EXTRA_OPTFLAGS=3D'$(EDK2_BASETOOLS_OPTFLAGS)' \ EXTRA_LDFLAGS=3D'$(EDK2_BASETOOLS_LDFLAGS)' =20 diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build= .sh index 8aa7935c43..3b78f30840 100755 --- a/tests/uefi-test-tools/build.sh +++ b/tests/uefi-test-tools/build.sh @@ -29,8 +29,7 @@ export PACKAGES_PATH=3D$(realpath -- "$edk2_dir") export WORKSPACE=3D$PWD mkdir -p Conf =20 -# Work around . -export PYTHON_COMMAND=3Dpython2 +export PYTHON_COMMAND=3D${EDK2_PYTHON_COMMAND:-python3} =20 # Source "edksetup.sh" carefully. set +e +u +C @@ -46,12 +45,15 @@ fi source "$edk2_dir/../edk2-funcs.sh" edk2_arch=3D$(qemu_edk2_get_arch "$emulation_target") edk2_toolchain=3D$(qemu_edk2_get_toolchain "$emulation_target") +MAKEFLAGS=3D$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") +edk2_thread_count=3D$(qemu_edk2_get_thread_count "$MAKEFLAGS") qemu_edk2_set_cross_env "$emulation_target" =20 # Build the UEFI binary mkdir -p log build \ --arch=3D"$edk2_arch" \ + -n "$edk2_thread_count" \ --buildtarget=3DDEBUG \ --platform=3DUefiTestToolsPkg/UefiTestToolsPkg.dsc \ --tagname=3D"$edk2_toolchain" \ --=20 2.21.0