From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC64FC3A5A6 for ; Thu, 19 Sep 2019 18:56:20 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B303C20640 for ; Thu, 19 Sep 2019 18:56:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B303C20640 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47886 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iB1bH-0005EZ-Pm for qemu-devel@archiver.kernel.org; Thu, 19 Sep 2019 14:56:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36950) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iB1Zv-0004DS-RM for qemu-devel@nongnu.org; Thu, 19 Sep 2019 14:54:57 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iB1Zt-0006x2-SE for qemu-devel@nongnu.org; Thu, 19 Sep 2019 14:54:55 -0400 Received: from mx1.redhat.com ([209.132.183.28]:59312) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iB1Zt-0006wZ-KG for qemu-devel@nongnu.org; Thu, 19 Sep 2019 14:54:53 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 58A2787648; Thu, 19 Sep 2019 18:54:52 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-121-45.rdu2.redhat.com [10.10.121.45]) by smtp.corp.redhat.com (Postfix) with ESMTP id 11A9C60606; Thu, 19 Sep 2019 18:54:45 +0000 (UTC) Subject: Re: [PATCH] edk2 build scripts: work around TianoCore#1607 without forcing Python 2 To: =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= , qemu devel list References: <20190918171141.15957-1-lersek@redhat.com> From: Laszlo Ersek Message-ID: <6f100c98-cc55-b570-93eb-4db807810a06@redhat.com> Date: Thu, 19 Sep 2019 20:54:45 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Thu, 19 Sep 2019 18:54:52 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?UTF-8?Q?Alex_Benn=c3=a9e?= , John Snow , Eduardo Habkost Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On 09/19/19 15:41, Philippe Mathieu-Daud=C3=A9 wrote: > Hi Laszlo, >=20 > On 9/18/19 7:11 PM, Laszlo Ersek wrote: >> It turns out that forcing python2 for running the edk2 "build" utility= is >> neither necessary nor sufficient. >> >> Forcing python2 is not sufficient for two reasons: >> >> - QEMU is moving away from python2, with python2 nearing EOL, >> >> - according to my most recent testing, the lacking dependency informat= ion >> in the makefiles that are generated by edk2's "build" utility can ca= use >> parallel build failures even when "build" is executed by python2. >> >> And forcing python2 is not necessary because we can still return to th= e >> original idea of filtering out jobserver-related options from MAKEFLAG= S. >> So do that. >> >> With this patch, the guest UEFI binaries that are used as part of the = BIOS >> tables test, and the OVMF and ArmVirtQemu platform firmwares, will be >> built strictly in a single job, regardless of an outermost "-jN" make >> option. Alas, there appears to be no reliable way to build edk2 in an >> (outer make, inner make) environment, with a jobserver enabled. >> >> Cc: Eduardo Habkost >> Cc: John Snow >> Cc: Philippe Mathieu-Daud=C3=A9 >> Reported-by: John Snow >> Signed-off-by: Laszlo Ersek >> --- >> >> Notes: >> - Tested on RHEL7 (where the outer "make" sets the old-style >> "--jobserver-fds" flag) and on Fedora 29 (where the outer "make"= sets >> the new-style "--jobserver-auth" flag). >> =20 >> - I've rebuilt all the edk2 binaries with this patch applied. Ever= ything >> works fine. However, if you test this patch, you might notice th= at git >> reports all the build products as modified. That's because when = using >> the python3 code in edk2 BaseTools, the generated makefiles diff= er >> greatly from the ones generated when running in python2 mode (e.= g. due >> to different random seeds in python hashes / dictionaries). As a >> result, parts of the firmware volumes / firmware filesystems cou= ld >> appear in a different order than before. This is harmless, and d= oesn't >> necessitate checking in the rebuilt binaries. >> >> roms/edk2-build.sh | 4 +--- >> roms/edk2-funcs.sh | 17 +++++++++++++++++ >> tests/uefi-test-tools/build.sh | 6 +++--- >> 3 files changed, 21 insertions(+), 6 deletions(-) >> >> diff --git a/roms/edk2-build.sh b/roms/edk2-build.sh >> index 4f46f8a6a217..8161c55ef507 100755 >> --- a/roms/edk2-build.sh >> +++ b/roms/edk2-build.sh >> @@ -27,9 +27,6 @@ shift $num_args >> =20 >> cd edk2 >> =20 >> -# Work around = . >> -export PYTHON_COMMAND=3Dpython2 >> - >> # Source "edksetup.sh" carefully. >> set +e +u +C >> source ./edksetup.sh >> @@ -43,6 +40,7 @@ fi >> # any), for the edk2 "build" utility. >> source ../edk2-funcs.sh >> edk2_toolchain=3D$(qemu_edk2_get_toolchain "$emulation_target") >> +MAKEFLAGS=3D$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") >> edk2_thread_count=3D$(qemu_edk2_get_thread_count "$MAKEFLAGS") >> qemu_edk2_set_cross_env "$emulation_target" >> =20 >> diff --git a/roms/edk2-funcs.sh b/roms/edk2-funcs.sh >> index a9fae7ee891b..3f4485b201f1 100644 >> --- a/roms/edk2-funcs.sh >> +++ b/roms/edk2-funcs.sh >> @@ -251,3 +251,20 @@ qemu_edk2_get_thread_count() >> printf '1\n' >> fi >> } >> + >> + >> +# Work around = by >> +# filtering jobserver-related flags out of MAKEFLAGS. Print the resul= t to the >> +# standard output. >> +# >> +# Parameters: >> +# $1: the value of the MAKEFLAGS variable >> +qemu_edk2_quirk_tianocore_1607() >> +{ >> + local makeflags=3D"$1" >> + >> + printf %s "$makeflags" \ >> + | LC_ALL=3DC sed --regexp-extended \ >> + --expression=3D's/--jobserver-(auth|fds)=3D[0-9]+,[0-9]+//' \ >> + --expression=3D's/-j([0-9]+)?//' >> +} >> diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/bu= ild.sh >> index 8aa7935c43bb..eba7964a163b 100755 >> --- a/tests/uefi-test-tools/build.sh >> +++ b/tests/uefi-test-tools/build.sh >> @@ -29,9 +29,6 @@ export PACKAGES_PATH=3D$(realpath -- "$edk2_dir") >> export WORKSPACE=3D$PWD >> mkdir -p Conf >> =20 >> -# Work around = . >> -export PYTHON_COMMAND=3Dpython2 >> - >> # Source "edksetup.sh" carefully. >> set +e +u +C >> source "$PACKAGES_PATH/edksetup.sh" >> @@ -46,12 +43,15 @@ fi >> source "$edk2_dir/../edk2-funcs.sh" >> edk2_arch=3D$(qemu_edk2_get_arch "$emulation_target") >> edk2_toolchain=3D$(qemu_edk2_get_toolchain "$emulation_target") >> +MAKEFLAGS=3D$(qemu_edk2_quirk_tianocore_1607 "$MAKEFLAGS") >> +edk2_thread_count=3D$(qemu_edk2_get_thread_count "$MAKEFLAGS") >> qemu_edk2_set_cross_env "$emulation_target" >> =20 >> # Build the UEFI binary >> mkdir -p log >> build \ >> --arch=3D"$edk2_arch" \ >> + -n "$edk2_thread_count" \ >> --buildtarget=3DDEBUG \ >> --platform=3DUefiTestToolsPkg/UefiTestToolsPkg.dsc \ >> --tagname=3D"$edk2_toolchain" \ >> >=20 > Very clear explanation, thanks. > Reviewed-by: Philippe Mathieu-Daud=C3=A9 Thanks! >=20 > Hmm this failed on Ubuntu Xenial which is the image we use for Travis-C= I: >=20 > $ lsb_release -a > Distributor ID: Ubuntu > Description: Ubuntu 16.04.6 LTS > Release: 16.04 > Codename: xenial >=20 > make -f Makefile.edk2 > make[1]: Entering directory '/home/phil/qemu/roms' > if test -d edk2/.git; then \ > cd edk2 && git submodule update --init --force; \ > fi > ./edk2-build.sh \ > aarch64 \ > --arch=3DAARCH64 \ > --platform=3DArmVirtPkg/ArmVirtQemu.dsc \ > -D NETWORK_IP6_ENABLE \ > -D NETWORK_HTTP_BOOT_ENABLE > WORKSPACE: /home/phil/qemu/roms/edk2 > EDK_TOOLS_PATH: /home/phil/qemu/roms/edk2/BaseTools > CONF_PATH: /home/phil/qemu/roms/edk2/Conf > Copying $EDK_TOOLS_PATH/Conf/build_rule.template > to /home/phil/qemu/roms/edk2/Conf/build_rule.txt > Copying $EDK_TOOLS_PATH/Conf/tools_def.template > to /home/phil/qemu/roms/edk2/Conf/tools_def.txt > Copying $EDK_TOOLS_PATH/Conf/target.template > to /home/phil/qemu/roms/edk2/Conf/target.txt > pyenv: python3.7: command not found > The `python3.7' command exists in these Python versions: > 3.7 > 3.7.1 ... I don't have the slightest idea what this error message means. Edk2 contains no reference to "pyenv". >=20 > Makefile.edk2:62: recipe for target '../pc-bios/edk2-aarch64-code.fd' f= ailed > make[1]: *** [../pc-bios/edk2-aarch64-code.fd] Error 127 > make[1]: Leaving directory '/home/phil/qemu/roms' > Makefile:168: recipe for target 'efi' failed > make: *** [efi] Error 2 > make: Leaving directory '/home/phil/qemu/roms' > The command "make -C roms efi -j2" exited with 2. >=20 > The local Python3 version is: >=20 > $ apt-cache show python3-minimal > Package: python3-minimal > Version: 3.5.1-3 >=20 > Any idea which script is choosing python3.7? >=20 It's the SetupPython() function in "edksetup.sh". If there is a universal pathname (or just filename) that refers to python3 on all build hosts where "make efi" is expected to run, I can assign that to PYTHON_COMMAND. Otherwise, I'm out of ideas. Thanks Laszlo