[PATCH 0/3] scripts/make-release: Decrease the size of the release tarballs

[PATCH 0/3] scripts/make-release: Decrease the size of the release tarballs

[Thomas Huth]

Our release tarballs are huge - qemu-7.0.0.tar.xz has a size of 119 MiB.
If you look at the contents, more than half of the size is used for the
edk2 sources that we ship along to provide the sources for the firmware
binaries, too. This feels very wrong, why do we urge users to download
such huge tarballs while 99.9% of them never will rebuilt the firmware
sources? We were also struggeling a bit in the past already with server
load and costs, so we should really try to decrease the size of our
release tarballs to a saner level.

Fortunately, edk2 has a permissive BSD license, so we are not forced
to distribute the sources for this. Thus instead of packaging the whole
edk2 source tree in our tarballs, let's just do the bare minimum and
provide the license information and a pointer to where the users can
download the edk2 sources instead. This decreases the size of our tarballs
already to the half of the original size.

Some few additional MiBs can be saved by omitting the sources of the
skiboot firmware, which has a permissive license, too (see second patch).
The final patch is rather cosmetics only - it drops some additional
.yml and .git files from the tarball that are of no use for the normal
user without the corresponding git repository.

Thomas Huth (3):
  scripts/make-release: Do not include the edk2 sources in the tarball
    anymore
  scripts/make-release: Do not include the skiboot sources in the
    tarball anymore
  scripts/make-release: Remove CI yaml and more git files from the
    tarball

 scripts/make-release | 46 ++++++++++++++++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 8 deletions(-)

-- 
2.31.1

[PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Thomas Huth]

The edk2 sources are bigger than the sources of QEMU - so they double the
size of our release tarballs if we include them. Fortunately, edk2 has a
permissive license, so there is no need for us to do this as long as we
continue to distribute the edk2 license information in our release tarball.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 scripts/make-release | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/scripts/make-release b/scripts/make-release
index 05b14ecc95..f79ed7a2ae 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -20,19 +20,34 @@ git checkout "v${version}"
 git submodule update --init
 (cd roms/seabios && git describe --tags --long --dirty > .version)
 (cd roms/skiboot && ./make_version.sh > .version)
-# Fetch edk2 submodule's submodules, since it won't have access to them via
-# the tarball later.
+
+# Fetch edk2 submodule's submodules to get their license information.
+# Edk2 is distributed under permissive licenses, so we don't have to
+# include the full source tree here (which is bigger than QEMU's sources)
+# as long as we provide the license information in the tarball.
 #
 # A more uniform way to handle this sort of situation would be nice, but we
 # don't necessarily have much control over how a submodule handles its
 # submodule dependencies, so we continue to handle these on a case-by-case
 # basis for now.
-(cd roms/edk2 && \
-    git submodule update --init -- \
+cd roms/edk2
+git submodule update --init --depth 1 -- \
         ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
         BaseTools/Source/C/BrotliCompress/brotli \
-        CryptoPkg/Library/OpensslLib/openssl \
-        MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
+        CryptoPkg/Library/OpensslLib/openssl
+mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \
+   License-softfloat.txt
+mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt
+mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt
+edk2rev=$(git rev-parse --short=12 HEAD)
+rm -r $(ls | grep -v License) .[a-z]*
+cat > sources.txt <<EOF
+The edk2 sources can be downloaded from:
+
+https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
+EOF
+cd ../..
+
 popd
 tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
-- 
2.31.1

[PATCH 2/3] scripts/make-release: Do not include the skiboot sources in the tarball anymore

[Thomas Huth]

The skiboot sources are licensed under the Apache license, so we don't
have to include them in our tarball as long as we continue to distribute
the skiboot license information in our release tarball.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 scripts/make-release | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/scripts/make-release b/scripts/make-release
index f79ed7a2ae..176304f30b 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -19,7 +19,6 @@ pushd ${destination}
 git checkout "v${version}"
 git submodule update --init
 (cd roms/seabios && git describe --tags --long --dirty > .version)
-(cd roms/skiboot && ./make_version.sh > .version)
 
 # Fetch edk2 submodule's submodules to get their license information.
 # Edk2 is distributed under permissive licenses, so we don't have to
@@ -48,6 +47,20 @@ https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
 EOF
 cd ../..
 
+# Same for skiboot - it has a permissive Apache license, so we don't
+# need to ship the sources for this.
+cd roms/skiboot
+skibootrev=$(git rev-parse --short=12 HEAD)
+head -n 38 libstb/tss2/ibmtpm20tss/utils/tss.c > LICENSE-tss.txt
+head -n 50 libfdt/fdt.c > LICENSE-libfdt.txt
+rm -r $(ls | grep -v LICEN) .[a-z]*
+cat > sources.txt <<EOF
+The skiboot sources can be downloaded from:
+
+https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz
+EOF
+cd ../..
+
 popd
 tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
-- 
2.31.1

[PATCH 3/3] scripts/make-release: Remove CI yaml and more git files from the tarball

[Thomas Huth]

These files are of no use in a normal tarball and thus should not
be included here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 scripts/make-release | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/make-release b/scripts/make-release
index 176304f30b..61c0fd0bfb 100755
--- a/scripts/make-release
+++ b/scripts/make-release
@@ -61,6 +61,8 @@ https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz
 EOF
 cd ../..
 
+rm .*.yml
 popd
-tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
+
+tar --exclude=".git*" -cjf ${destination}.tar.bz2 ${destination}
 rm -rf ${destination}
-- 
2.31.1

Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Stefan Hajnoczi]

On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote:
>
> The edk2 sources are bigger than the sources of QEMU - so they double the
> size of our release tarballs if we include them. Fortunately, edk2 has a
> permissive license, so there is no need for us to do this as long as we
> continue to distribute the edk2 license information in our release tarball.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  scripts/make-release | 27 +++++++++++++++++++++------
>  1 file changed, 21 insertions(+), 6 deletions(-)
>
> diff --git a/scripts/make-release b/scripts/make-release
> index 05b14ecc95..f79ed7a2ae 100755
> --- a/scripts/make-release
> +++ b/scripts/make-release
> @@ -20,19 +20,34 @@ git checkout "v${version}"
>  git submodule update --init
>  (cd roms/seabios && git describe --tags --long --dirty > .version)
>  (cd roms/skiboot && ./make_version.sh > .version)
> -# Fetch edk2 submodule's submodules, since it won't have access to them via
> -# the tarball later.
> +
> +# Fetch edk2 submodule's submodules to get their license information.
> +# Edk2 is distributed under permissive licenses, so we don't have to
> +# include the full source tree here (which is bigger than QEMU's sources)
> +# as long as we provide the license information in the tarball.
>  #
>  # A more uniform way to handle this sort of situation would be nice, but we
>  # don't necessarily have much control over how a submodule handles its
>  # submodule dependencies, so we continue to handle these on a case-by-case
>  # basis for now.
> -(cd roms/edk2 && \
> -    git submodule update --init -- \
> +cd roms/edk2
> +git submodule update --init --depth 1 -- \
>          ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
>          BaseTools/Source/C/BrotliCompress/brotli \
> -        CryptoPkg/Library/OpensslLib/openssl \
> -        MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
> +        CryptoPkg/Library/OpensslLib/openssl
> +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \
> +   License-softfloat.txt
> +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt
> +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt
> +edk2rev=$(git rev-parse --short=12 HEAD)
> +rm -r $(ls | grep -v License) .[a-z]*
> +cat > sources.txt <<EOF
> +The edk2 sources can be downloaded from:
> +
> +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz

Please use QEMU's edk2 mirror:
https://gitlab.com/qemu-project/edk2

QEMU mirrors all dependencies so that even if upstream projects go
offline we can still rebuild all of QEMU from source.

Thanks,
Stefan

Re: [PATCH 3/3] scripts/make-release: Remove CI yaml and more git files from the tarball

[Stefan Hajnoczi]

On Mon, 4 Jul 2022 at 07:50, Thomas Huth <thuth@redhat.com> wrote:
>
> These files are of no use in a normal tarball and thus should not
> be included here.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  scripts/make-release | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/make-release b/scripts/make-release
> index 176304f30b..61c0fd0bfb 100755
> --- a/scripts/make-release
> +++ b/scripts/make-release
> @@ -61,6 +61,8 @@ https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz
>  EOF
>  cd ../..
>
> +rm .*.yml
>  popd
> -tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
> +
> +tar --exclude=".git*" -cjf ${destination}.tar.bz2 ${destination}

If the excludes become more elaborate in the future we can use tar
--exclude-from=FILE and keep a separate file with all the patterns
instead of using both rm and --exclude= patterns.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

Re: [PATCH 0/3] scripts/make-release: Decrease the size of the release tarballs

[Stefan Hajnoczi]

Thanks for doing this! I haven't reviewed the licensing impact.

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Thomas Huth]

On 04/07/2022 09.26, Stefan Hajnoczi wrote:
> On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote:
>>
>> The edk2 sources are bigger than the sources of QEMU - so they double the
>> size of our release tarballs if we include them. Fortunately, edk2 has a
>> permissive license, so there is no need for us to do this as long as we
>> continue to distribute the edk2 license information in our release tarball.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   scripts/make-release | 27 +++++++++++++++++++++------
>>   1 file changed, 21 insertions(+), 6 deletions(-)
>>
>> diff --git a/scripts/make-release b/scripts/make-release
>> index 05b14ecc95..f79ed7a2ae 100755
>> --- a/scripts/make-release
>> +++ b/scripts/make-release
>> @@ -20,19 +20,34 @@ git checkout "v${version}"
>>   git submodule update --init
>>   (cd roms/seabios && git describe --tags --long --dirty > .version)
>>   (cd roms/skiboot && ./make_version.sh > .version)
>> -# Fetch edk2 submodule's submodules, since it won't have access to them via
>> -# the tarball later.
>> +
>> +# Fetch edk2 submodule's submodules to get their license information.
>> +# Edk2 is distributed under permissive licenses, so we don't have to
>> +# include the full source tree here (which is bigger than QEMU's sources)
>> +# as long as we provide the license information in the tarball.
>>   #
>>   # A more uniform way to handle this sort of situation would be nice, but we
>>   # don't necessarily have much control over how a submodule handles its
>>   # submodule dependencies, so we continue to handle these on a case-by-case
>>   # basis for now.
>> -(cd roms/edk2 && \
>> -    git submodule update --init -- \
>> +cd roms/edk2
>> +git submodule update --init --depth 1 -- \
>>           ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
>>           BaseTools/Source/C/BrotliCompress/brotli \
>> -        CryptoPkg/Library/OpensslLib/openssl \
>> -        MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
>> +        CryptoPkg/Library/OpensslLib/openssl
>> +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \
>> +   License-softfloat.txt
>> +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt
>> +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt
>> +edk2rev=$(git rev-parse --short=12 HEAD)
>> +rm -r $(ls | grep -v License) .[a-z]*
>> +cat > sources.txt <<EOF
>> +The edk2 sources can be downloaded from:
>> +
>> +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
> 
> Please use QEMU's edk2 mirror:
> https://gitlab.com/qemu-project/edk2
> 
> QEMU mirrors all dependencies so that even if upstream projects go
> offline we can still rebuild all of QEMU from source.

Sure, that makes sense, indeed. I'll wait a little bit longer for other 
comments, then I'll respin the series with the URLs updated (also in the 
second patch).

  Thomas

Re: [PATCH 2/3] scripts/make-release: Do not include the skiboot sources in the tarball anymore

[Cédric Le Goater]

On 7/4/22 08:42, Thomas Huth wrote:
> The skiboot sources are licensed under the Apache license, so we don't
> have to include them in our tarball as long as we continue to distribute
> the skiboot license information in our release tarball.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>

Reviewed-by: Cédric Le Goater <clg@kaod.org>

Thanks,

C.

> ---
>   scripts/make-release | 15 ++++++++++++++-
>   1 file changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/make-release b/scripts/make-release
> index f79ed7a2ae..176304f30b 100755
> --- a/scripts/make-release
> +++ b/scripts/make-release
> @@ -19,7 +19,6 @@ pushd ${destination}
>   git checkout "v${version}"
>   git submodule update --init
>   (cd roms/seabios && git describe --tags --long --dirty > .version)
> -(cd roms/skiboot && ./make_version.sh > .version)
>   
>   # Fetch edk2 submodule's submodules to get their license information.
>   # Edk2 is distributed under permissive licenses, so we don't have to
> @@ -48,6 +47,20 @@ https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
>   EOF
>   cd ../..
>   
> +# Same for skiboot - it has a permissive Apache license, so we don't
> +# need to ship the sources for this.
> +cd roms/skiboot
> +skibootrev=$(git rev-parse --short=12 HEAD)
> +head -n 38 libstb/tss2/ibmtpm20tss/utils/tss.c > LICENSE-tss.txt
> +head -n 50 libfdt/fdt.c > LICENSE-libfdt.txt
> +rm -r $(ls | grep -v LICEN) .[a-z]*
> +cat > sources.txt <<EOF
> +The skiboot sources can be downloaded from:
> +
> +https://github.com/open-power/skiboot/archive/${skibootrev}.tar.gz
> +EOF
> +cd ../..
> +
>   popd
>   tar --exclude=.git -cjf ${destination}.tar.bz2 ${destination}
>   rm -rf ${destination}

Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Daniel P. Berrangé]

On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote:
> On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote:
> >
> > The edk2 sources are bigger than the sources of QEMU - so they double the
> > size of our release tarballs if we include them. Fortunately, edk2 has a
> > permissive license, so there is no need for us to do this as long as we
> > continue to distribute the edk2 license information in our release tarball.
> >
> > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > ---
> >  scripts/make-release | 27 +++++++++++++++++++++------
> >  1 file changed, 21 insertions(+), 6 deletions(-)
> >
> > diff --git a/scripts/make-release b/scripts/make-release
> > index 05b14ecc95..f79ed7a2ae 100755
> > --- a/scripts/make-release
> > +++ b/scripts/make-release
> > @@ -20,19 +20,34 @@ git checkout "v${version}"
> >  git submodule update --init
> >  (cd roms/seabios && git describe --tags --long --dirty > .version)
> >  (cd roms/skiboot && ./make_version.sh > .version)
> > -# Fetch edk2 submodule's submodules, since it won't have access to them via
> > -# the tarball later.
> > +
> > +# Fetch edk2 submodule's submodules to get their license information.
> > +# Edk2 is distributed under permissive licenses, so we don't have to
> > +# include the full source tree here (which is bigger than QEMU's sources)
> > +# as long as we provide the license information in the tarball.
> >  #
> >  # A more uniform way to handle this sort of situation would be nice, but we
> >  # don't necessarily have much control over how a submodule handles its
> >  # submodule dependencies, so we continue to handle these on a case-by-case
> >  # basis for now.
> > -(cd roms/edk2 && \
> > -    git submodule update --init -- \
> > +cd roms/edk2
> > +git submodule update --init --depth 1 -- \
> >          ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3 \
> >          BaseTools/Source/C/BrotliCompress/brotli \
> > -        CryptoPkg/Library/OpensslLib/openssl \
> > -        MdeModulePkg/Library/BrotliCustomDecompressLib/brotli)
> > +        CryptoPkg/Library/OpensslLib/openssl
> > +mv ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3/COPYING.txt \
> > +   License-softfloat.txt
> > +mv BaseTools/Source/C/BrotliCompress/brotli/LICENSE License-brotli.txt
> > +mv CryptoPkg/Library/OpensslLib/openssl/LICENSE License-openssl.txt
> > +edk2rev=$(git rev-parse --short=12 HEAD)
> > +rm -r $(ls | grep -v License) .[a-z]*
> > +cat > sources.txt <<EOF
> > +The edk2 sources can be downloaded from:
> > +
> > +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
> 
> Please use QEMU's edk2 mirror:
> https://gitlab.com/qemu-project/edk2
> 
> QEMU mirrors all dependencies so that even if upstream projects go
> offline we can still rebuild all of QEMU from source.

Note that the github/lab generated tarballs are not signed, while
QEMU's release tarballs are gpg signed, so from that POV this would
be a regression no matter which site we point to.

Also it would need more guidance on what to actually do with the
tarball, as if you merely unpack it into this dir, it won't work
as it will be one level of dirs nesting too deep for QEMU's
build scripts to work.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/3] scripts/make-release: Decrease the size of the release tarballs

[Daniel P. Berrangé]

On Mon, Jul 04, 2022 at 08:42:51AM +0200, Thomas Huth wrote:
> Our release tarballs are huge - qemu-7.0.0.tar.xz has a size of 119 MiB.
> If you look at the contents, more than half of the size is used for the
> edk2 sources that we ship along to provide the sources for the firmware
> binaries, too. This feels very wrong, why do we urge users to download
> such huge tarballs while 99.9% of them never will rebuilt the firmware
> sources? We were also struggeling a bit in the past already with server
> load and costs, so we should really try to decrease the size of our
> release tarballs to a saner level.
> 
> Fortunately, edk2 has a permissive BSD license, so we are not forced
> to distribute the sources for this. Thus instead of packaging the whole
> edk2 source tree in our tarballs, let's just do the bare minimum and
> provide the license information and a pointer to where the users can
> download the edk2 sources instead. This decreases the size of our tarballs
> already to the half of the original size.

Regardless of license, we are not required to bundle the source code
and binaries in the same tarball. We've merely done that because it
was a convenient & easy way to approach the problem historically.

It would be valid to not ship *any* of the source for the pre-built
roms in the main qemu-x.y.z.tar.xz file /provided/ we ensure that
we *always* have a qemu-firmware-src-x.y.z.tar.xz file alongside it.


> Some few additional MiBs can be saved by omitting the sources of the
> skiboot firmware, which has a permissive license, too (see second patch).
> The final patch is rather cosmetics only - it drops some additional
> .yml and .git files from the tarball that are of no use for the normal
> user without the corresponding git repository.

I look at a few more scenarios

  * Current tarball:            119 MB
  * minus edk/skiboot source:    54 MB
  * also minus edk2 binaries:    45 MB
  * also minus pc-bios/ + roms/: 19 MB
  * minus roms/ only:            31 MB

IOW, cutting the tarball in half is great, but if we split off firmware
binaries and source into completely separated tarballs we would win big.
If we fully split off only the firmware source we still win quite alot.

IOW, rather than special casing edk/skiboot, I would prefer to see us
have make a consistent approach to firmware.

Either

 * qemu-x.y.z.tar.gz              (only qemu maintained src, 19 MB)
 * qemu-firmware-x.y.z.tar.gz     (pre-built blobs aka pc-bios/, 13 MB)
 * qemu-firmware-src-x.y.z.tar.gz (source for pre-built blobs aka roms/, 92 MB)

Or 

 * qemu-x.y.z.tar.gz              (qemu maintained src and pre-built blobs, 31 MB)
 * qemu-firmware-src-x.y.z.tar.gz (source for pre-built blobs aka roms/, 92 MB)


The second option is probably the least disruptive option for end users
building QEMU directly, while still giving distros most of the benefits
they desire. And probably easiest to put into practice for us.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Thomas Huth]

On 04/07/2022 11.14, Daniel P. Berrangé wrote:
> On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote:
>> On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote:
...
>>> +cat > sources.txt <<EOF
>>> +The edk2 sources can be downloaded from:
>>> +
>>> +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
>>
>> Please use QEMU's edk2 mirror:
>> https://gitlab.com/qemu-project/edk2
>>
>> QEMU mirrors all dependencies so that even if upstream projects go
>> offline we can still rebuild all of QEMU from source.
> 
> Note that the github/lab generated tarballs are not signed, while
> QEMU's release tarballs are gpg signed, so from that POV this would
> be a regression no matter which site we point to.

Maybe we should point to the edk2 release page instead? I.e.:

  https://github.com/tianocore/edk2/releases

Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here.

> Also it would need more guidance on what to actually do with the
> tarball, as if you merely unpack it into this dir, it won't work
> as it will be one level of dirs nesting too deep for QEMU's
> build scripts to work.

I could add some wording how to use the edk2-build.sh script with a 
downloaded edk2 tarball if that helps ... not sure whether it is really 
required, though, we also don't provide instruction for building any other 
firmware yet, as far as I know.

  Thomas

Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Gerd Hoffmann]

> > Also it would need more guidance on what to actually do with the
> > tarball, as if you merely unpack it into this dir, it won't work
> > as it will be one level of dirs nesting too deep for QEMU's
> > build scripts to work.
> 
> I could add some wording how to use the edk2-build.sh script with a
> downloaded edk2 tarball if that helps ... not sure whether it is really
> required, though, we also don't provide instruction for building any other
> firmware yet, as far as I know.

The instructions are printed by "make -C roms help".

We could teach roms/Makefile to either automatically fetch the sources
(be that the upstream edk2 tarball or a qemu-firmware-src tarball should
we go with the approach suggested by Daniel elsewhere in this thread),
or throw an error with instructions when they are not present.

take care,
  Gerd

Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Daniel P. Berrangé]

On Mon, Jul 04, 2022 at 11:37:11AM +0200, Thomas Huth wrote:
> On 04/07/2022 11.14, Daniel P. Berrangé wrote:
> > On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote:
> > > On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote:
> ...
> > > > +cat > sources.txt <<EOF
> > > > +The edk2 sources can be downloaded from:
> > > > +
> > > > +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
> > > 
> > > Please use QEMU's edk2 mirror:
> > > https://gitlab.com/qemu-project/edk2
> > > 
> > > QEMU mirrors all dependencies so that even if upstream projects go
> > > offline we can still rebuild all of QEMU from source.
> > 
> > Note that the github/lab generated tarballs are not signed, while
> > QEMU's release tarballs are gpg signed, so from that POV this would
> > be a regression no matter which site we point to.
> 
> Maybe we should point to the edk2 release page instead? I.e.:
> 
>  https://github.com/tianocore/edk2/releases
> 
> Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here.
> 
> > Also it would need more guidance on what to actually do with the
> > tarball, as if you merely unpack it into this dir, it won't work
> > as it will be one level of dirs nesting too deep for QEMU's
> > build scripts to work.
> 
> I could add some wording how to use the edk2-build.sh script with a
> downloaded edk2 tarball if that helps ... not sure whether it is really
> required, though, we also don't provide instruction for building any other
> firmware yet, as far as I know.

Well my thought is that currently you can do  'make -C roms efi' from
the unpacked release tarball, and with this change, that no longer
works. So we ought to explain what to do to get that working again.

This doesn't neccessarily means docs. It could just be 'make' prints
out an error telling people what to do, or figure it out itself.

If we split off the entire of the 'roms' directory into a self-contained
firmware-src tarball, the 'roms/Makefile' would still "just work".

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/3] scripts/make-release: Do not include the edk2 sources in the tarball anymore

[Thomas Huth]

On 04/07/2022 12.45, Daniel P. Berrangé wrote:
> On Mon, Jul 04, 2022 at 11:37:11AM +0200, Thomas Huth wrote:
>> On 04/07/2022 11.14, Daniel P. Berrangé wrote:
>>> On Mon, Jul 04, 2022 at 08:26:34AM +0100, Stefan Hajnoczi wrote:
>>>> On Mon, 4 Jul 2022 at 07:45, Thomas Huth <thuth@redhat.com> wrote:
>> ...
>>>>> +cat > sources.txt <<EOF
>>>>> +The edk2 sources can be downloaded from:
>>>>> +
>>>>> +https://github.com/tianocore/edk2/archive/${edk2rev}.tar.gz
>>>>
>>>> Please use QEMU's edk2 mirror:
>>>> https://gitlab.com/qemu-project/edk2
>>>>
>>>> QEMU mirrors all dependencies so that even if upstream projects go
>>>> offline we can still rebuild all of QEMU from source.
>>>
>>> Note that the github/lab generated tarballs are not signed, while
>>> QEMU's release tarballs are gpg signed, so from that POV this would
>>> be a regression no matter which site we point to.
>>
>> Maybe we should point to the edk2 release page instead? I.e.:
>>
>>   https://github.com/tianocore/edk2/releases
>>
>> Anyway, it's IMHO certainly not *our* job to provide signed edk2 sources here.
>>
>>> Also it would need more guidance on what to actually do with the
>>> tarball, as if you merely unpack it into this dir, it won't work
>>> as it will be one level of dirs nesting too deep for QEMU's
>>> build scripts to work.
>>
>> I could add some wording how to use the edk2-build.sh script with a
>> downloaded edk2 tarball if that helps ... not sure whether it is really
>> required, though, we also don't provide instruction for building any other
>> firmware yet, as far as I know.
> 
> Well my thought is that currently you can do  'make -C roms efi' from
> the unpacked release tarball, and with this change, that no longer
> works. So we ought to explain what to do to get that working again.
> 
> This doesn't neccessarily means docs. It could just be 'make' prints
> out an error telling people what to do, or figure it out itself.
> 
> If we split off the entire of the 'roms' directory into a self-contained
> firmware-src tarball, the 'roms/Makefile' would still "just work".

Sorry for the late reply, I've been pondering about this for a while and was 
busy with other stuff...

So I think I tend to agree - if we still want to ship the edk2 (and skiboot) 
sources, it's likely best if we put all roms/* sources into a separate 
tarball. Maybe even split the roms folder into a separate git repository.

However, there's at least one minor obstacle: The sources of the s390-ccw 
bios (in pc-bios/s390-ccw) currently partly depend on the roms/SLOF sources 
to be available for the full build. So if we slice out the roms/SLOF stuff, 
we should likely move the s390-ccw bios also to same location...

I guess it's now too late for 7.1, so I'll postpone my further work on this 
to the 7.2 release cycle.

  Thomas