[PATCH 0/2] automation: fix building in the openSUSE Tumbleweed

[PATCH 0/2] automation: fix building in the openSUSE Tumbleweed

[Dario Faggioli]

Building Xen in openSUSE Tumbleweed in the GitLab CI was broken due to
python and libzstd development dependencies, so let's update the docker
file to fix that.

If this change is accepted, I'm happy to push a new, updated, image to
our registry (ISTR that I used to have the right to do that).

While there, extend the generalization of the container runtime to use
(we have that in containerize already, through the DOCKER_CMD variable)
to the local building of the containers as well.

Regards
---
Dario Faggioli (2):
      automation: use DOCKER_CMD for building containers too
      automation: fix dependencies on openSUSE Tumbleweed containers

 automation/build/suse/opensuse-tumbleweed.dockerfile | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
--
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[PATCH 1/2] automation: use DOCKER_CMD for building containers too

[Dario Faggioli]

From: Dario Faggioli <dario@Solace.fritz.box>

Use DOCKER_CMD from the environment (if defined) in the containers'
makefile too, so that, e.g., when doing `export DOCKED_CMD=podman`
podman is used for building the containers too.

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
---
Cc: Doug Goldstein <cardoe@cardoe.com>
Cc: Roger Pau Monne <roger.pau@citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
 automation/build/Makefile |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/automation/build/Makefile b/automation/build/Makefile
index 7c7612b1d9..a4b2b85178 100644
--- a/automation/build/Makefile
+++ b/automation/build/Makefile
@@ -2,6 +2,7 @@
 # the base of where these containers will appear
 REGISTRY := registry.gitlab.com/xen-project/xen
 CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile))
+DOCKER_CMD ?= docker
 
 help:
 	@echo "Builds containers for building Xen based on different distros"
@@ -10,9 +11,9 @@ help:
 	@echo "To push container builds, set the env var PUSH"
 
 %: %.dockerfile ## Builds containers
-	docker build -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
+	$(DOCKER_CMD) build -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
 	@if [ ! -z $${PUSH+x} ]; then \
-		docker push $(REGISTRY)/$(@D):$(@F); \
+		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
 	fi
 
 .PHONY: all

Re: [PATCH 1/2] automation: use DOCKER_CMD for building containers too

[Roger Pau Monné]

On Tue, May 18, 2021 at 02:04:07PM +0200, Dario Faggioli wrote:
> From: Dario Faggioli <dario@Solace.fritz.box>
> 
> Use DOCKER_CMD from the environment (if defined) in the containers'
> makefile too, so that, e.g., when doing `export DOCKED_CMD=podman`
> podman is used for building the containers too.
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>

Acked-by: Roger Pau Monné <roger.pau@citrix.com>

Thanks!

Re: [PATCH 1/2] automation: use DOCKER_CMD for building containers too

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

On Tue, 2021-05-18 at 16:26 +0200, Roger Pau Monné wrote:
> On Tue, May 18, 2021 at 02:04:07PM +0200, Dario Faggioli wrote:
> > From: Dario Faggioli <dario@Solace.fritz.box>
> > 
> > Use DOCKER_CMD from the environment (if defined) in the containers'
> > makefile too, so that, e.g., when doing `export DOCKED_CMD=podman`
> > podman is used for building the containers too.
> > 
> > Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> 
> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
>
Thanks! If it has not been committed yet, can I resend with a From:
that makes sense?

And, either way, sorry for the noise... :-(

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Dario Faggioli]

From: Dario Faggioli <dario@Solace.fritz.box>

Fix the build inside our openSUSE Tumbleweed container by using
the proper python development packages (and adding zstd headers).

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
---
Cc: Doug Goldstein <cardoe@cardoe.com>
Cc: Roger Pau Monne <roger.pau@citrix.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
 .../build/suse/opensuse-tumbleweed.dockerfile      |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
index 8ff7b9b5ce..5b99cb8a53 100644
--- a/automation/build/suse/opensuse-tumbleweed.dockerfile
+++ b/automation/build/suse/opensuse-tumbleweed.dockerfile
@@ -45,6 +45,7 @@ RUN zypper install -y --no-recommends \
         libtasn1-devel \
         libuuid-devel \
         libyajl-devel \
+        libzstd-devel \
         lzo-devel \
         make \
         nasm \
@@ -56,10 +57,8 @@ RUN zypper install -y --no-recommends \
         pandoc \
         patch \
         pkg-config \
-        python \
         python-devel \
-        python3 \
-        python3-devel \
+        python38-devel \
         systemd-devel \
         tar \
         transfig \

Re: [PATCH 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Roger Pau Monné]

On Tue, May 18, 2021 at 02:04:13PM +0200, Dario Faggioli wrote:
> From: Dario Faggioli <dario@Solace.fritz.box>
> 
> Fix the build inside our openSUSE Tumbleweed container by using
> the proper python development packages (and adding zstd headers).
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> ---
> Cc: Doug Goldstein <cardoe@cardoe.com>
> Cc: Roger Pau Monne <roger.pau@citrix.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
>  .../build/suse/opensuse-tumbleweed.dockerfile      |    5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
> index 8ff7b9b5ce..5b99cb8a53 100644
> --- a/automation/build/suse/opensuse-tumbleweed.dockerfile
> +++ b/automation/build/suse/opensuse-tumbleweed.dockerfile
> @@ -45,6 +45,7 @@ RUN zypper install -y --no-recommends \
>          libtasn1-devel \
>          libuuid-devel \
>          libyajl-devel \
> +        libzstd-devel \
>          lzo-devel \
>          make \
>          nasm \
> @@ -56,10 +57,8 @@ RUN zypper install -y --no-recommends \
>          pandoc \
>          patch \
>          pkg-config \
> -        python \
>          python-devel \
> -        python3 \
> -        python3-devel \
> +        python38-devel \

When I tested python3-devel was translated into python38-devel, which
I think is better as we don't need to modify the docker file for every
new Python version?

Thanks, Roger.

Re: [PATCH 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 2372 bytes --]

On Tue, 2021-05-18 at 15:20 +0200, Roger Pau Monné wrote:
> On Tue, May 18, 2021 at 02:04:13PM +0200, Dario Faggioli wrote:
> > From: Dario Faggioli <dario@Solace.fritz.box>
> > 
Mmm... this email address does not really exist, and it's a mistake
that it ended up here. :-/

> > Fix the build inside our openSUSE Tumbleweed container by using
> > the proper python development packages (and adding zstd headers).
> > 
> > Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> > ---
> > Cc: Doug Goldstein <cardoe@cardoe.com>
> > Cc: Roger Pau Monne <roger.pau@citrix.com>
> > Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> > ---
> >  .../build/suse/opensuse-tumbleweed.dockerfile      |    5 ++---
> >  1 file changed, 2 insertions(+), 3 deletions(-)
> > 
> > diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile
> > b/automation/build/suse/opensuse-tumbleweed.dockerfile
> > index 8ff7b9b5ce..5b99cb8a53 100644
> > --- a/automation/build/suse/opensuse-tumbleweed.dockerfile
> > +++ b/automation/build/suse/opensuse-tumbleweed.dockerfile
> > 
> > @@ -56,10 +57,8 @@ RUN zypper install -y --no-recommends \
> >          pandoc \
> >          patch \
> >          pkg-config \
> > -        python \
> >          python-devel \
> > -        python3 \
> > -        python3-devel \
> > +        python38-devel \
> 
> When I tested python3-devel was translated into python38-devel, 
>
Oh, really? And when was it that you tested it, if I can ask?

> which
> I think is better as we don't need to modify the docker file for
> every
> new Python version?
> 
That would definitely be better. Right now, I don't see any
python3-devel package. If python3-devel can still be used (and it
somehow translates to the proper -devel package), then sure we should
use it. I'm not sure how that would happen, but maybe it's just me
being unaware of some packaging magic.

Let me put "python3-devel" there and test locally again, so we know if
it actually works.

Thanks and Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Roger Pau Monné]

On Tue, May 18, 2021 at 04:33:43PM +0200, Dario Faggioli wrote:
> On Tue, 2021-05-18 at 15:20 +0200, Roger Pau Monné wrote:
> > On Tue, May 18, 2021 at 02:04:13PM +0200, Dario Faggioli wrote:
> > > From: Dario Faggioli <dario@Solace.fritz.box>
> > > 
> Mmm... this email address does not really exist, and it's a mistake
> that it ended up here. :-/
> 
> > > Fix the build inside our openSUSE Tumbleweed container by using
> > > the proper python development packages (and adding zstd headers).
> > > 
> > > Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> > > ---
> > > Cc: Doug Goldstein <cardoe@cardoe.com>
> > > Cc: Roger Pau Monne <roger.pau@citrix.com>
> > > Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> > > ---
> > >  .../build/suse/opensuse-tumbleweed.dockerfile      |    5 ++---
> > >  1 file changed, 2 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > b/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > index 8ff7b9b5ce..5b99cb8a53 100644
> > > --- a/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > +++ b/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > 
> > > @@ -56,10 +57,8 @@ RUN zypper install -y --no-recommends \
> > >          pandoc \
> > >          patch \
> > >          pkg-config \
> > > -        python \
> > >          python-devel \
> > > -        python3 \
> > > -        python3-devel \
> > > +        python38-devel \
> > 
> > When I tested python3-devel was translated into python38-devel, 
> >
> Oh, really? And when was it that you tested it, if I can ask?

I've tried just now with the current docker file, and this is the
(trimmed) output:

Step 7/7 : RUN zypper install -y --no-recommends         acpica         bc         bin86         bison         bzip2         checkpolicy         clang         cmake         dev86         discount         flex         gcc         gcc-c++         gettext-tools         git         glib2-devel         glibc-devel         glibc-devel-32bit         gzip         hostname         libSDL2-devel         libaio-devel         libbz2-devel         libext2fs-devel         libgnutls-devel         libjpeg62-devel         libnl3-devel         libnuma-devel         libpixman-1-0-devel         libpng16-devel         libssh2-devel         libtasn1-devel         libuuid-devel         libyajl-devel         lzo-devel         make         nasm         ncurses-devel         ocaml         ocaml-findlib-devel         ocaml-ocamlbuild         ocaml-ocamldoc         pandoc         patch         pkg-config         python         python-devel         python3         python3-devel         systemd-devel         tar         transfig         valgrind-devel         wget         which         xz-devel         zlib-devel         &&         zypper clean -a
 ---> Running in af6a184e482b
Retrieving repository 'openSUSE-Tumbleweed-Non-Oss' metadata [..done]
Building repository 'openSUSE-Tumbleweed-Non-Oss' cache [....done]
Retrieving repository 'openSUSE-Tumbleweed-Oss' metadata [.......done]
Building repository 'openSUSE-Tumbleweed-Oss' cache [....done]
Retrieving repository 'openSUSE-Tumbleweed-Update' metadata [.done]
Building repository 'openSUSE-Tumbleweed-Update' cache [....done]
Loading repository data...
Reading installed packages...
'python3' not found in package names. Trying capabilities.
'python3-devel' not found in package names. Trying capabilities.

There's this message here ...

'pkg-config' not found in package names. Trying capabilities.
Resolving package dependencies...

The following 509 NEW packages are going to be installed:
[...] python38 python38-base python38-devel [...]

... but it seems python3-devel gets translated into python38-devel, or
maybe something else selects python38-devel?

Not familiar with the system, so maybe it's indeed a dependency of
some other package.

> > which
> > I think is better as we don't need to modify the docker file for
> > every
> > new Python version?
> > 
> That would definitely be better. Right now, I don't see any
> python3-devel package. If python3-devel can still be used (and it
> somehow translates to the proper -devel package), then sure we should
> use it. I'm not sure how that would happen, but maybe it's just me
> being unaware of some packaging magic.
> 
> Let me put "python3-devel" there and test locally again, so we know if
> it actually works.

It does seem to be picked up, whether that's because python3-devel
gets translated into python38-devel, or something else pulls it in
I don't certainly know.

Thanks, Roger.

Re: [PATCH 2/2] automation: fix dependencies on openSUSE Tumbleweed containers

[Dario Faggioli]

[-- Attachment #1: Type: text/plain, Size: 4683 bytes --]

On Tue, 2021-05-18 at 16:33 +0200, Dario Faggioli wrote:
> On Tue, 2021-05-18 at 15:20 +0200, Roger Pau Monné wrote:
> > On Tue, May 18, 2021 at 02:04:13PM +0200, Dario Faggioli wrote:
> > > From: Dario Faggioli <dario@Solace.fritz.box>
> > > 
> Mmm... this email address does not really exist, and it's a mistake
> that it ended up here. :-/
> 
> > > Fix the build inside our openSUSE Tumbleweed container by using
> > > the proper python development packages (and adding zstd headers).
> > > 
> > > Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> > > ---
> > > Cc: Doug Goldstein <cardoe@cardoe.com>
> > > Cc: Roger Pau Monne <roger.pau@citrix.com>
> > > Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> > > ---
> > >  .../build/suse/opensuse-tumbleweed.dockerfile      |    5 ++---
> > >  1 file changed, 2 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > b/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > index 8ff7b9b5ce..5b99cb8a53 100644
> > > --- a/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > +++ b/automation/build/suse/opensuse-tumbleweed.dockerfile
> > > 
> > > @@ -56,10 +57,8 @@ RUN zypper install -y --no-recommends \
> > >          pandoc \
> > >          patch \
> > >          pkg-config \
> > > -        python \
> > >          python-devel \
> > > -        python3 \
> > > -        python3-devel \
> > > +        python38-devel \
> > 
> > When I tested python3-devel was translated into python38-devel, 
> > 
> Oh, really? And when was it that you tested it, if I can ask?
> 
> > which
> > I think is better as we don't need to modify the docker file for
> > every
> > new Python version?
> > 
> That would definitely be better. Right now, I don't see any
> python3-devel package. If python3-devel can still be used (and it
> somehow translates to the proper -devel package), then sure we should
> use it. I'm not sure how that would happen, but maybe it's just me
> being unaware of some packaging magic.
> 
> Let me put "python3-devel" there and test locally again, so we know
> if
> it actually works.
> 
Ok, indeed it works. And, on a second though, it's not obscure at all
that it does.

It's just that python38-devel _provides_ python3-devel, which makes a
lot of sense, and it was silly of me to not think it does that and use
just python3-devel in the first place:

dario@4b10a592ca98:~> zypper if --provides python38-devel
Information for package python38-devel:                                                                                                                                                                                                       
---------------------------------------                                                                                                                                                                                                       
Repository     : @System                                                                                                                                                                                                                      
Name           : python38-devel                                                                                                                                                                                                               
Version        : 3.8.10-1.2
Arch           : x86_64
Vendor         : openSUSE
Installed Size : 882.7 KiB
Installed      : Yes
Status         : up-to-date
Source package : python38-core-3.8.10-1.2.src
Summary        : Include Files and Libraries Mandatory for Building Python Modules
[...]
Provides       : [8]
    libpython3.so()(64bit)
    pkgconfig(python-3.8) = 3.8
    pkgconfig(python-3.8-embed) = 3.8
    pkgconfig(python3) = 3.8
    pkgconfig(python3-embed) = 3.8
    python3-devel = 3.8.10
    python38-devel = 3.8.10-1.2
    python38-devel(x86-64) = 3.8.10-1.2

What now puzzles me a little, though, is why the build was failing, as
python3-devel was already there in the docker file. Maybe we "just"
forgot to push the image?

Well, there's a different issue (missing libzstd-devel), so I'll send a
v2 of this series anyway.

Regards
-- 
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]