[PATCH 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[PATCH 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[Philippe Mathieu-Daudé]

cmocka website SSL certificate expired, making CI pipelines
fail [*]. However EDK2 images built to test QEMU don't need
cmocka. Avoid cloning it.

[*] https://gitlab.com/rth7680/qemu/-/jobs/1685387520
fatal: unable to access 'https://git.cryptomilk.org/projects/cmocka.git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
fatal: clone of 'https://git.cryptomilk.org/projects/cmocka.git' into submodule path 'UnitTestFrameworkPkg/Library/CmockaLib/cmocka' failed

Philippe Mathieu-Daudé (2):
  roms/edk2: Only init brotli submodule to build BaseTools
  roms/edk2: Only initialize required submodules

 .gitlab-ci.d/edk2.yml | 4 ++++
 roms/Makefile         | 3 ++-
 roms/Makefile.edk2    | 7 ++++++-
 scripts/make-release  | 7 ++++++-
 4 files changed, 18 insertions(+), 3 deletions(-)

-- 
2.31.1

[PATCH 1/2] roms/edk2: Only init brotli submodule to build BaseTools

[Philippe Mathieu-Daudé]

Since EDK2 BaseTools only require the brotli submodule,
we don't need to initialize other submodules at to build it.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 roms/Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/roms/Makefile b/roms/Makefile
index eeb5970348c..b967b53bb76 100644
--- a/roms/Makefile
+++ b/roms/Makefile
@@ -143,7 +143,8 @@ build-efi-roms: build-pxe-roms
 #    efirom
 #
 edk2-basetools:
-	cd edk2/BaseTools && git submodule update --init --force
+	cd edk2/BaseTools && git submodule update --init --force \
+		Source/C/BrotliCompress/brotli
 	$(MAKE) -C edk2/BaseTools \
 		PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \
 		EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \
-- 
2.31.1

[PATCH 2/2] roms/edk2: Only initialize required submodules

[Philippe Mathieu-Daudé]

The EDK2 firmware images built to test QEMU do not require
the following submodules:

 - MdeModulePkg/Universal/RegularExpressionDxe/oniguruma
 - UnitTestFrameworkPkg/Library/CmockaLib/cmocka

The only submodules required are:

 - ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
 - BaseTools/Source/C/BrotliCompress/brotli
 - CryptoPkg/Library/OpensslLib/openssl
 - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli

Adapt the buildsys machinery to only initialize the required
submodules.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
---
 .gitlab-ci.d/edk2.yml | 4 ++++
 roms/Makefile.edk2    | 7 ++++++-
 scripts/make-release  | 7 ++++++-
 3 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.d/edk2.yml b/.gitlab-ci.d/edk2.yml
index 62497ba47fb..2f12939d744 100644
--- a/.gitlab-ci.d/edk2.yml
+++ b/.gitlab-ci.d/edk2.yml
@@ -51,6 +51,10 @@ build-edk2:
  script: # Clone the required submodules and build EDK2
  - git submodule update --init roms/edk2
  - git -C roms/edk2 submodule update --init
+     ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
+     BaseTools/Source/C/BrotliCompress/brotli
+     CryptoPkg/Library/OpensslLib/openssl
+     MdeModulePkg/Library/BrotliCustomDecompressLib/brotli
  - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
  - echo "=== Using ${JOBS} simultaneous jobs ==="
  - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a edk2-stderr.log >&2
diff --git a/roms/Makefile.edk2 b/roms/Makefile.edk2
index a8ed3255757..fdae0b511f5 100644
--- a/roms/Makefile.edk2
+++ b/roms/Makefile.edk2
@@ -51,7 +51,12 @@ all: $(foreach flashdev,$(flashdevs),../pc-bios/edk2-$(flashdev).fd.bz2) \
 # make-release/tarball scripts.
 submodules:
 	if test -d edk2/.git; then \
-		cd edk2 && git submodule update --init --force; \
+		cd edk2 && git submodule update --init --force -- \
+			ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
+			BaseTools/Source/C/BrotliCompress/brotli \
+			CryptoPkg/Library/OpensslLib/openssl \
+			MdeModulePkg/Library/BrotliCustomDecompressLib/brotli \
+		; \
 	fi
 
 # See notes on the ".NOTPARALLEL" target and the "+" indicator in
diff --git a/scripts/make-release b/scripts/make-release
index a2a8cda33c0..a9ff933a4ca 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -27,7 +27,12 @@ git submodule update --init
 # don't necessarily have much control over how a submodule handles its
 # submodule dependencies, so we continue to handle these on a case-by-case
 # basis for now.
-(cd roms/edk2 && git submodule update --init)
+(cd roms/edk2 && \
+    git submodule update --init \
+        ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
+        BaseTools/Source/C/BrotliCompress/brotli \
+        CryptoPkg/Library/OpensslLib/openssl \
+        MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
 popd
 tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
-- 
2.31.1

Re: [PATCH 0/2] roms/edk2: Avoid cloning unused cmocka submodule

[Philippe Mathieu-Daudé]

On 10/18/21 11:23, Philippe Mathieu-Daudé wrote:
> cmocka website SSL certificate expired, making CI pipelines
> fail [*]. However EDK2 images built to test QEMU don't need
> cmocka. Avoid cloning it.

Oh I forgot to mention we don't need oniguruma either, so
this series also avoid cloning it.

> Philippe Mathieu-Daudé (2):
>   roms/edk2: Only init brotli submodule to build BaseTools
>   roms/edk2: Only initialize required submodules

Re: [PATCH 1/2] roms/edk2: Only init brotli submodule to build BaseTools

[Thomas Huth]

On 18/10/2021 11.23, Philippe Mathieu-Daudé wrote:
> Since EDK2 BaseTools only require the brotli submodule,
> we don't need to initialize other submodules at to build it.

s/at to/to/

> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>   roms/Makefile | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/roms/Makefile b/roms/Makefile
> index eeb5970348c..b967b53bb76 100644
> --- a/roms/Makefile
> +++ b/roms/Makefile
> @@ -143,7 +143,8 @@ build-efi-roms: build-pxe-roms
>   #    efirom
>   #
>   edk2-basetools:
> -	cd edk2/BaseTools && git submodule update --init --force
> +	cd edk2/BaseTools && git submodule update --init --force \
> +		Source/C/BrotliCompress/brotli
>   	$(MAKE) -C edk2/BaseTools \
>   		PYTHON_COMMAND=$${EDK2_PYTHON_COMMAND:-python3} \
>   		EXTRA_OPTFLAGS='$(EDK2_BASETOOLS_OPTFLAGS)' \

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 2/2] roms/edk2: Only initialize required submodules

[Philippe Mathieu-Daudé]

On 10/18/21 11:23, Philippe Mathieu-Daudé wrote:
> The EDK2 firmware images built to test QEMU do not require
> the following submodules:
> 
>  - MdeModulePkg/Universal/RegularExpressionDxe/oniguruma
>  - UnitTestFrameworkPkg/Library/CmockaLib/cmocka
> 
> The only submodules required are:
> 
>  - ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
>  - BaseTools/Source/C/BrotliCompress/brotli
>  - CryptoPkg/Library/OpensslLib/openssl
>  - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli
> 
> Adapt the buildsys machinery to only initialize the required
> submodules.
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> ---
>  .gitlab-ci.d/edk2.yml | 4 ++++
>  roms/Makefile.edk2    | 7 ++++++-
>  scripts/make-release  | 7 ++++++-
>  3 files changed, 16 insertions(+), 2 deletions(-)

>  # See notes on the ".NOTPARALLEL" target and the "+" indicator in
> diff --git a/scripts/make-release b/scripts/make-release
> index a2a8cda33c0..a9ff933a4ca 100755
> --- a/scripts/make-release
> +++ b/scripts/make-release
> @@ -27,7 +27,12 @@ git submodule update --init
>  # don't necessarily have much control over how a submodule handles its
>  # submodule dependencies, so we continue to handle these on a case-by-case
>  # basis for now.
> -(cd roms/edk2 && git submodule update --init)
> +(cd roms/edk2 && \
> +    git submodule update --init \

Hmm I forgot the '--' separator here.

> +        ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
> +        BaseTools/Source/C/BrotliCompress/brotli \
> +        CryptoPkg/Library/OpensslLib/openssl \
> +        MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
>  popd
>  tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
>  rm -rf ${destination}
> 

Re: [PATCH 2/2] roms/edk2: Only initialize required submodules

[Philippe Mathieu-Daudé]

Cc'ing Michael, since the make-release script is not covered
in any MAINTAINERS section.

On 10/18/21 11:31, Philippe Mathieu-Daudé wrote:
> On 10/18/21 11:23, Philippe Mathieu-Daudé wrote:
>> The EDK2 firmware images built to test QEMU do not require
>> the following submodules:
>>
>>  - MdeModulePkg/Universal/RegularExpressionDxe/oniguruma
>>  - UnitTestFrameworkPkg/Library/CmockaLib/cmocka
>>
>> The only submodules required are:
>>
>>  - ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
>>  - BaseTools/Source/C/BrotliCompress/brotli
>>  - CryptoPkg/Library/OpensslLib/openssl
>>  - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli
>>
>> Adapt the buildsys machinery to only initialize the required
>> submodules.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> ---
>>  .gitlab-ci.d/edk2.yml | 4 ++++
>>  roms/Makefile.edk2    | 7 ++++++-
>>  scripts/make-release  | 7 ++++++-
>>  3 files changed, 16 insertions(+), 2 deletions(-)
> 
>>  # See notes on the ".NOTPARALLEL" target and the "+" indicator in
>> diff --git a/scripts/make-release b/scripts/make-release
>> index a2a8cda33c0..a9ff933a4ca 100755
>> --- a/scripts/make-release
>> +++ b/scripts/make-release
>> @@ -27,7 +27,12 @@ git submodule update --init
>>  # don't necessarily have much control over how a submodule handles its
>>  # submodule dependencies, so we continue to handle these on a case-by-case
>>  # basis for now.
>> -(cd roms/edk2 && git submodule update --init)
>> +(cd roms/edk2 && \
>> +    git submodule update --init \
> 
> Hmm I forgot the '--' separator here.

BTW Michael since commit 9d03f5abede ("travis.yml: Remove the
"Release tarball" job") we don't test make-release anymore in
our CI, you might want to add some GitLab equivalent job.

Regards,

Phil.