[PATCH v2 0/2] automation: fix building in the openSUSE Tumbleweed

[PATCH v2 0/2] automation: fix building in the openSUSE Tumbleweed

[Dario Faggioli]

Fix the build in the openSUSE Tumbleweed container within our CI. There
was a missing dependency (libzstd-devel), which needs to be added to the
dockerfile.

OTOH, python3-devel was in the dockerfile already, and hence it should
be there in the image. Yet, build was failing due to that... Maybe we
forgot to build and then push a new image after adding it?

Well, whatever. If this change is accepted, I'm happy to push a new,
updated, image to our registry (ISTR that I used to have the right to
do that).

While there, extend the generalization of the container runtime to use
(we have that in containerize already, through the DOCKER_CMD variable)
to the local building of the containers as well.

Thanks and Regards
---
Dario Faggioli (2):
      automation: use DOCKER_CMD for building containers too
      automation: fix dependencies on openSUSE Tumbleweed containers

 automation/build/suse/opensuse-tumbleweed.dockerfile | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
--
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[PATCH v2 1/2] automation: use DOCKER_CMD for building containers too

[Dario Faggioli]

Use DOCKER_CMD from the environment (if defined) in the containers'
makefile too, so that, e.g., when doing `export DOCKED_CMD=podman`
podman is used for building the containers too.

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Doug Goldstein <cardoe@cardoe.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes from v1:
- fix my email address in From:
---
 automation/build/Makefile |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/automation/build/Makefile b/automation/build/Makefile
index 7c7612b1d9..a4b2b85178 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -2,6 +2,7 @@
 # the base of where these containers will appear
 REGISTRY := registry.gitlab.com/xen-project/xen
 CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
+DOCKER_CMD ?= docker
 
 help:
 	@echo "Builds containers for building Xen based on different distros"
@@ -10,9 +11,9 @@ help:
 	@echo "To push container builds, set the env var PUSH"
 
 %: %.dockerfile ## Builds containers
-	docker build -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
+	$(DOCKER_CMD) build -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
 	@if [ ! -z $${PUSH+x} ]; then \
-		docker push $(REGISTRY)/$(@D):$(@F); \
+		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
 	fi
 
 .PHONY: all

[PATCH v2 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Dario Faggioli]

Fix the build inside our openSUSE Tumbleweed container by using
adding libzstd headers. While there, remove the explicit dependency
for python and python3 as the respective -devel packages will pull
them in anyway.

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
---
Cc: Doug Goldstein <cardoe@cardoe.com>
Cc: Roger Pau Monne <roger.pau@citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes from v1:
- fix my email address in From:
- don't request python38-devel explicitly, python3-devel
  is just fine and is more generic (and hence better!)
---
 .../build/suse/opensuse-tumbleweed.dockerfile      |    3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
index 8ff7b9b5ce..a33ab0d870 100644
--- a/automation/build/suse/opensuse-tumbleweed.dockerfile
+++ b/automation/build/suse/opensuse-tumbleweed.dockerfile
@@ -45,6 +45,7 @@ RUN zypper install -y --no-recommends \
         libtasn1-devel \
         libuuid-devel \
         libyajl-devel \
+        libzstd-devel \
         lzo-devel \
         make \
         nasm \
@@ -56,9 +57,7 @@ RUN zypper install -y --no-recommends \
         pandoc \
         patch \
         pkg-config \
-        python \
         python-devel \
-        python3 \
         python3-devel \
         systemd-devel \
         tar \

Re: [PATCH v2 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Roger Pau Monné]

On Tue, May 18, 2021 at 06:42:45PM +0200, Dario Faggioli wrote:
> Fix the build inside our openSUSE Tumbleweed container by using
> adding libzstd headers. While there, remove the explicit dependency
> for python and python3 as the respective -devel packages will pull
> them in anyway.
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>

Acked-by: Roger Pau Monné <roger.pau@citrix.com>

Can you try to push an updated container to the registry?

Thanks!

Re: [PATCH v2 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

On Wed, 2021-05-19 at 08:28 +0200, Roger Pau Monné wrote:
> On Tue, May 18, 2021 at 06:42:45PM +0200, Dario Faggioli wrote:
> > Fix the build inside our openSUSE Tumbleweed container by using
> > adding libzstd headers. While there, remove the explicit dependency
> > for python and python3 as the respective -devel packages will pull
> > them in anyway.
> > 
> > Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> 
> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
> 
Thanks!

> Can you try to push an updated container to the registry?
> 
Yeah, I tried, but I'm getting this:

STEP 8: COMMIT registry.gitlab.com/xen-project/xen/suse:opensuse-tumbleweed
--> 940c6edbff9
940c6edbff965135a25bc20f0e2a59cf6062b9e8bc3516858828cbb7bba92d8f
Getting image source signatures
Copying blob acc28ee93e9b [--------------------------------------] 8.0b / 3.5KiB
Copying blob 89c6eef91991 [--------------------------------------] 8.0b / 57.0MiB
Copying blob 20dabc80d591 [--------------------------------------] 8.0b / 90.6MiB
Copying blob 5ea007576ed8 [--------------------------------------] 8.0b / 2.0GiB
Error: error copying image to the remote destination: Error writing blob: Error initiating layer upload to /v2/xen-project/xen/suse/blobs/uploads/ in registry.gitlab.com: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

make: *** [Makefile:15: suse/opensuse-tumbleweed] Error 125

So, either I'm doing something wrong, or I was just misremembering and
I don't have the permission to do that... Can we check if I do?

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Andrew Cooper]

On 19/05/2021 18:52, Dario Faggioli wrote:
> On Wed, 2021-05-19 at 08:28 +0200, Roger Pau Monné wrote:
>> On Tue, May 18, 2021 at 06:42:45PM +0200, Dario Faggioli wrote:
>>> Fix the build inside our openSUSE Tumbleweed container by using
>>> adding libzstd headers. While there, remove the explicit dependency
>>> for python and python3 as the respective -devel packages will pull
>>> them in anyway.
>>>
>>> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
>> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
>>
> Thanks!
>
>> Can you try to push an updated container to the registry?
>>
> Yeah, I tried, but I'm getting this:
>
> STEP 8: COMMIT registry.gitlab.com/xen-project/xen/suse:opensuse-tumbleweed
> --> 940c6edbff9
> 940c6edbff965135a25bc20f0e2a59cf6062b9e8bc3516858828cbb7bba92d8f
> Getting image source signatures
> Copying blob acc28ee93e9b [--------------------------------------] 8.0b / 3.5KiB
> Copying blob 89c6eef91991 [--------------------------------------] 8.0b / 57.0MiB
> Copying blob 20dabc80d591 [--------------------------------------] 8.0b / 90.6MiB
> Copying blob 5ea007576ed8 [--------------------------------------] 8.0b / 2.0GiB
> Error: error copying image to the remote destination: Error writing blob: Error initiating layer upload to /v2/xen-project/xen/suse/blobs/uploads/ in registry.gitlab.com: errors:
> denied: requested access to the resource is denied
> unauthorized: authentication required
>
> make: *** [Makefile:15: suse/opensuse-tumbleweed] Error 125
>
> So, either I'm doing something wrong, or I was just misremembering and
> I don't have the permission to do that... Can we check if I do?

Hmm.

I built the container locally, which reused some of the layers you
pushed, and it all pushed successfully for me.

I've committed this series so xen.git matches reality.  Lets see how the
updated container fairs...

~Andrew

Re: [PATCH v2 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 1876 bytes --]

On Wed, 2021-05-19 at 20:25 +0100, Andrew Cooper wrote:
> 
> I built the container locally, which reused some of the layers you
> pushed, and it all pushed successfully for me.
> 
> I've committed this series so xen.git matches reality.  Lets see how
> the
> updated container fairs...
> 
Well, something still looks off, although I don't know what.

In fact, we still see: 
https://gitlab.com/xen-project/xen/-/jobs/1277448985

checking for deflateCopy in -lz... no
2608configure: error: Could not find zlib
2609configure: error: ./configure failed for tools

Which means we're missing libz.

In fact, if I pull the container that's currently in the registry, I
can see that:

dario@b17aaa86cacf:~> rpm -qa|grep zlib                               
zlib-devel-1.2.11-18.3.x86_64 

However, if I build it locally, with:

dario@Solace:~/src/xen/xen.git/automation/build> make suse/opensuse-tumbleweed

And then enter and use it for building, the package is there and
configure works.

dario@51e463d1d47e:~> rpm -qa|grep libz
libzstd1-1.4.9-1.2.x86_64
libz1-1.2.11-18.3.x86_64
libzck1-1.1.11-1.1.x86_64
libzypp-17.25.10-1.1.x86_64
libzio1-1.06-4.26.x86_64
libzzip-0-13-0.13.72-1.2.x86_64
libzstd-devel-1.4.9-1.2.x86_64

dario@51e463d1d47e:~> ./configure
checking build system type... x86_64-pc-linux-gnu                     
checking host system type... x86_64-pc-linux-gnu
... ... ...
checking for pandoc... /usr/bin/pandoc
checking for perl... /usr/bin/perl
configure: creating ./config.status

So, what am I missing or doing wrong?

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]