[PATCH v2 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[PATCH v2 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[Philippe Mathieu-Daudé]

cmocka website SSL certificate expired, making CI pipelines
fail [*]. However EDK2 images built to test QEMU don't need
cmocka, nor oniguruma. Avoid cloning them.

Note: scripts/make-release is neither covered in MAINTAINERS
      nor in our CI.

[*] https://gitlab.com/rth7680/qemu/-/jobs/1685387520
fatal: unable to access 'https://git.cryptomilk.org/projects/cmocka.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
fatal: clone of 'https://git.cryptomilk.org/projects/cmocka.git' into submodule path 'UnitTestFrameworkPkg/Library/CmockaLib/cmocka' failed

Since v1:
- Fixed typo (thuth)
- Added missing '--' shell separator

Philippe Mathieu-Daudé (2):
  roms/edk2: Only init brotli submodule to build BaseTools
  roms/edk2: Only initialize required submodules

 .gitlab-ci.d/edk2.yml | 6 +++++-
 roms/Makefile         | 3 ++-
 roms/Makefile.edk2    | 7 ++++++-
 scripts/make-release  | 7 ++++++-
 4 files changed, 19 insertions(+), 4 deletions(-)

-- 
2.31.1

[PATCH v2 1/2] roms/edk2: Only init brotli submodule to build BaseTools

[Philippe Mathieu-Daudé]

Since EDK2 BaseTools only require the brotli submodule,
we don't need to initialize other submodules to build it.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/roms/Makefile b/roms/Makefile
index eeb5970348c..b967b53bb76 100644
--- a/roms/Makefile
+++ b/roms/Makefile
@@ -143,7 +143,8 @@ build-efi-roms: build-pxe-roms
 #    efirom
 #
 edk2-basetools:
-	cd edk2/BaseTools && git submodule update --init --force
+	cd edk2/BaseTools && git submodule update --init --force \
+		Source/C/BrotliCompress/brotli
 	$(MAKE) -C edk2/BaseTools \
 		PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \
 		EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \
-- 
2.31.1

[PATCH v2 2/2] roms/edk2: Only initialize required submodules

[Philippe Mathieu-Daudé]

The EDK2 firmware images built to test QEMU do not require
the following submodules:

 - MdeModulePkg/Universal/RegularExpressionDxe/oniguruma
 - UnitTestFrameworkPkg/Library/CmockaLib/cmocka

The only submodules required are:

 - ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
 - BaseTools/Source/C/BrotliCompress/brotli
 - CryptoPkg/Library/OpensslLib/openssl
 - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli

Adapt the buildsys machinery to only initialize the required
submodules.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
Cc: Michael Roth <michael.roth@amd.com>
---
 .gitlab-ci.d/edk2.yml | 6 +++++-
 roms/Makefile.edk2    | 7 ++++++-
 scripts/make-release  | 7 ++++++-
 3 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
index 62497ba47fb..13d0f8b019f 100644
--- a/.gitlab-ci.d/edk2.yml
+++ b/.gitlab-ci.d/edk2.yml
@@ -50,7 +50,11 @@ build-edk2:
    GIT_DEPTH: 3
  script: # Clone the required submodules and build EDK2
  - git submodule update --init roms/edk2
- - git -C roms/edk2 submodule update --init
+ - git -C roms/edk2 submodule update --init --
+     ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
+     BaseTools/Source/C/BrotliCompress/brotli
+     CryptoPkg/Library/OpensslLib/openssl
+     MdeModulePkg/Library/BrotliCustomDecompressLib/brotli
  - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
  - echo "=== Using ${JOBS} simultaneous jobs ==="
  - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
diff --git a/roms/Makefile.edk2 b/roms/Makefile.edk2
index a8ed3255757..fdae0b511f5 100644
--- a/roms/Makefile.edk2
+++ b/roms/Makefile.edk2
@@ -51,7 +51,12 @@ all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
 # make-release/tarball scripts.
 submodules:
 	if test -d edk2/.git; then \
-		cd edk2 && git submodule update --init --force; \
+		cd edk2 && git submodule update --init --force -- \
+			ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
+			BaseTools/Source/C/BrotliCompress/brotli \
+			CryptoPkg/Library/OpensslLib/openssl \
+			MdeModulePkg/Library/BrotliCustomDecompressLib/brotli \
+		; \
 	fi
 
 # See notes on the ".NOTPARALLEL" target and the "+" indicator in
diff --git a/scripts/make-release b/scripts/make-release
index a2a8cda33c0..05b14ecc95e 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -27,7 +27,12 @@ git submodule update --init
 # don't necessarily have much control over how a submodule handles its
 # submodule dependencies, so we continue to handle these on a case-by-case
 # basis for now.
-(cd roms/edk2 && git submodule update --init)
+(cd roms/edk2 && \
+    git submodule update --init -- \
+        ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
+        BaseTools/Source/C/BrotliCompress/brotli \
+        CryptoPkg/Library/OpensslLib/openssl \
+        MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
 popd
 tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
-- 
2.31.1

Re: [PATCH v2 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[Gerd Hoffmann]

On Mon, Oct 18, 2021 at 12:58:14PM +0200, Philippe Mathieu-Daudé wrote:
> cmocka website SSL certificate expired, making CI pipelines
> fail [*]. However EDK2 images built to test QEMU don't need
> cmocka, nor oniguruma. Avoid cloning them.
> 
> Note: scripts/make-release is neither covered in MAINTAINERS
>       nor in our CI.
> 
> [*] https://gitlab.com/rth7680/qemu/-/jobs/1685387520
> fatal: unable to access 'https://git.cryptomilk.org/projects/cmocka.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
> fatal: clone of 'https://git.cryptomilk.org/projects/cmocka.git' into submodule path 'UnitTestFrameworkPkg/Library/CmockaLib/cmocka' failed
> 
> Since v1:
> - Fixed typo (thuth)
> - Added missing '--' shell separator

Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>

Re: [PATCH v2 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[Richard Henderson]

On 10/18/21 3:58 AM, Philippe Mathieu-Daudé wrote:
> cmocka website SSL certificate expired, making CI pipelines
> fail [*]. However EDK2 images built to test QEMU don't need
> cmocka, nor oniguruma. Avoid cloning them.
> 
> Note: scripts/make-release is neither covered in MAINTAINERS
>        nor in our CI.
> 
> [*] https://gitlab.com/rth7680/qemu/-/jobs/1685387520
> fatal: unable to access 'https://git.cryptomilk.org/projects/cmocka.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
> fatal: clone of 'https://git.cryptomilk.org/projects/cmocka.git' into submodule path 'UnitTestFrameworkPkg/Library/CmockaLib/cmocka' failed
> 
> Since v1:
> - Fixed typo (thuth)
> - Added missing '--' shell separator
> 
> Philippe Mathieu-Daudé (2):
>    roms/edk2: Only init brotli submodule to build BaseTools
>    roms/edk2: Only initialize required submodules

Thanks a lot.

Tested-by: Richard Henderson <richard.henderson@linaro.org>

I'll queue this to target-arm.next.


r~

Re: [PATCH v2 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[Philippe Mathieu-Daudé]

On 10/18/21 20:10, Richard Henderson wrote:
> On 10/18/21 3:58 AM, Philippe Mathieu-Daudé wrote:
>> cmocka website SSL certificate expired, making CI pipelines
>> fail [*]. However EDK2 images built to test QEMU don't need
>> cmocka, nor oniguruma. Avoid cloning them.
>>
>> Note: scripts/make-release is neither covered in MAINTAINERS
>>        nor in our CI.
>>
>> [*] https://gitlab.com/rth7680/qemu/-/jobs/1685387520
>> fatal: unable to access
>> 'https://git.cryptomilk.org/projects/cmocka.git/': server certificate
>> verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt
>> CRLfile: none
>> fatal: clone of 'https://git.cryptomilk.org/projects/cmocka.git' into
>> submodule path 'UnitTestFrameworkPkg/Library/CmockaLib/cmocka' failed
>>
>> Since v1:
>> - Fixed typo (thuth)
>> - Added missing '--' shell separator
>>
>> Philippe Mathieu-Daudé (2):
>>    roms/edk2: Only init brotli submodule to build BaseTools
>>    roms/edk2: Only initialize required submodules
> 
> Thanks a lot.
> 
> Tested-by: Richard Henderson <richard.henderson@linaro.org>
> 
> I'll queue this to target-arm.next.

If you don't have anything in your queue I can send a pullreq
tomorrow, otherwise thanks for taking care of it.

Re: [PATCH v2 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[Richard Henderson]

On 10/18/21 11:53 AM, Philippe Mathieu-Daudé wrote:
> If you don't have anything in your queue I can send a pullreq
> tomorrow, otherwise thanks for taking care of it.

I do have a few other patches already in the queue.
I'll take care of it.


r~