[PATCH v3 0/4] tests/vm/freebsd: Get up-to-date package list from lcitool

[PATCH v3 0/4] tests/vm/freebsd: Get up-to-date package list from lcitool

[Philippe Mathieu-Daudé]

Inspired by this patch from Thomas:
https://lore.kernel.org/qemu-devel/20230531090415.40421-1-thuth@redhat.com/

Instead of updating the package list manually, use lcitool vars file.

Since v2:
- Commit generated json (Daniel)

Since v1:
- Addressed Erik & Daniel comments (generate in JSON)

Philippe Mathieu-Daudé (4):
  tests/lcitool: Generate distribution packages list in JSON format
  tests/lcitool: Refresh generated files
  tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
  tests/vm/freebsd: Get up-to-date package list from lcitool vars file

 tests/docker/dockerfiles/debian-amd64.docker |  2 -
 tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
 tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
 tests/lcitool/refresh                        | 11 +++
 tests/vm/basevm.py                           | 11 +++
 tests/vm/freebsd                             | 42 +----------
 tests/vm/generated/README                    |  5 ++
 tests/vm/generated/freebsd.json              | 77 ++++++++++++++++++++
 8 files changed, 106 insertions(+), 46 deletions(-)
 create mode 100644 tests/vm/generated/README
 create mode 100644 tests/vm/generated/freebsd.json

-- 
2.38.1

[PATCH v3 1/4] tests/lcitool: Generate distribution packages list in JSON format

[Philippe Mathieu-Daudé]

Add the generate_pkglist() helper to generate a list of packages
required by a distribution to build QEMU.

Since we can not add a "THIS FILE WAS AUTO-GENERATED" comment in
JSON, create the files under tests/vm/generated/ sub-directory;
add a README mentioning the files are generated.

Suggested-by: Erik Skultety <eskultet@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/lcitool/refresh     | 11 +++++++++++
 tests/vm/generated/README |  5 +++++
 2 files changed, 16 insertions(+)
 create mode 100644 tests/vm/generated/README

diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
index b54566edcc..4584870ea1 100755
--- a/tests/lcitool/refresh
+++ b/tests/lcitool/refresh
@@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None):
     generate(filename, cmd, trailer)
 
 
+def generate_pkglist(vm, target):
+    filename = Path(src_dir, "tests", "vm", "generated", vm + ".json")
+    cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"]
+    generate(filename, cmd, None)
+
+
 # Netmap still needs to be manually built as it is yet to be packaged
 # into a distro. We also add cscope and gtags which are used in the CI
 # test
@@ -191,6 +197,11 @@ try:
     generate_cirrus("freebsd-13")
     generate_cirrus("macos-12")
 
+    #
+    # VM packages lists
+    #
+    generate_pkglist("freebsd", "freebsd-13")
+
     sys.exit(0)
 except Exception as ex:
     print(str(ex), file=sys.stderr)
diff --git a/tests/vm/generated/README b/tests/vm/generated/README
new file mode 100644
index 0000000000..7ccc6ffd3d
--- /dev/null
+++ b/tests/vm/generated/README
@@ -0,0 +1,5 @@
+# FILES IN THIS FOLDER WERE AUTO-GENERATED
+#
+#  $ make lcitool-refresh
+#
+# https://gitlab.com/libvirt/libvirt-ci
-- 
2.38.1

[PATCH v3 2/4] tests/lcitool: Refresh generated files

[Philippe Mathieu-Daudé]

Refresh the generated files by running:

  $ make lcitool-refresh

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/docker/dockerfiles/debian-amd64.docker |  2 -
 tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
 tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
 tests/vm/generated/freebsd.json              | 77 ++++++++++++++++++++
 4 files changed, 77 insertions(+), 6 deletions(-)
 create mode 100644 tests/vm/generated/freebsd.json

diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
index e39871c7bb..8f7521fdc2 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -70,7 +70,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libpam0g-dev \
                       libpcre2-dev \
                       libpixman-1-dev \
-                      libpmem-dev \
                       libpng-dev \
                       libpulse-dev \
                       librbd-dev \
@@ -96,7 +95,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libvdeplug-dev \
                       libvirglrenderer-dev \
                       libvte-2.91-dev \
-                      libxen-dev \
                       libzstd-dev \
                       llvm \
                       locales \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
index 8f864d19e6..7f60143cbb 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -69,7 +69,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libpam0g-dev \
                       libpcre2-dev \
                       libpixman-1-dev \
-                      libpmem-dev \
                       libpng-dev \
                       libpulse-dev \
                       librbd-dev \
@@ -94,7 +93,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libvdeplug-dev \
                       libvirglrenderer-dev \
                       libvte-2.91-dev \
-                      libxen-dev \
                       libzstd-dev \
                       llvm \
                       locales \
diff --git a/tests/docker/dockerfiles/ubuntu2204.docker b/tests/docker/dockerfiles/ubuntu2204.docker
index 1d442cdfe6..5162927016 100644
--- a/tests/docker/dockerfiles/ubuntu2204.docker
+++ b/tests/docker/dockerfiles/ubuntu2204.docker
@@ -70,7 +70,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libpam0g-dev \
                       libpcre2-dev \
                       libpixman-1-dev \
-                      libpmem-dev \
                       libpng-dev \
                       libpulse-dev \
                       librbd-dev \
@@ -96,7 +95,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
                       libvdeplug-dev \
                       libvirglrenderer-dev \
                       libvte-2.91-dev \
-                      libxen-dev \
                       libzstd-dev \
                       llvm \
                       locales \
diff --git a/tests/vm/generated/freebsd.json b/tests/vm/generated/freebsd.json
new file mode 100644
index 0000000000..7c435cf23e
--- /dev/null
+++ b/tests/vm/generated/freebsd.json
@@ -0,0 +1,77 @@
+{
+  "ccache": "/usr/local/bin/ccache",
+  "cpan_pkgs": [],
+  "cross_pkgs": [],
+  "make": "/usr/local/bin/gmake",
+  "ninja": "/usr/local/bin/ninja",
+  "packaging_command": "pkg",
+  "pip3": "/usr/local/bin/pip-3.8",
+  "pkgs": [
+    "alsa-lib",
+    "bash",
+    "bison",
+    "bzip2",
+    "ca_root_nss",
+    "capstone4",
+    "ccache",
+    "cmocka",
+    "ctags",
+    "curl",
+    "cyrus-sasl",
+    "dbus",
+    "diffutils",
+    "dtc",
+    "flex",
+    "fusefs-libs3",
+    "gettext",
+    "git",
+    "glib",
+    "gmake",
+    "gnutls",
+    "gsed",
+    "gtk3",
+    "json-c",
+    "libepoxy",
+    "libffi",
+    "libgcrypt",
+    "libjpeg-turbo",
+    "libnfs",
+    "libslirp",
+    "libspice-server",
+    "libssh",
+    "libtasn1",
+    "llvm",
+    "lzo2",
+    "meson",
+    "mtools",
+    "ncurses",
+    "nettle",
+    "ninja",
+    "opencv",
+    "pixman",
+    "pkgconf",
+    "png",
+    "py39-numpy",
+    "py39-pillow",
+    "py39-pip",
+    "py39-sphinx",
+    "py39-sphinx_rtd_theme",
+    "py39-yaml",
+    "python3",
+    "rpm2cpio",
+    "sdl2",
+    "sdl2_image",
+    "snappy",
+    "sndio",
+    "socat",
+    "spice-protocol",
+    "tesseract",
+    "usbredir",
+    "virglrenderer",
+    "vte3",
+    "xorriso",
+    "zstd"
+  ],
+  "pypi_pkgs": [],
+  "python": "/usr/local/bin/python3"
+}
-- 
2.38.1

[PATCH v3 3/4] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

[Philippe Mathieu-Daudé]

Add the get_qemu_packages_from_lcitool_json() helper which return
such package list from a lcitool env var file in JSON format.

Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/vm/basevm.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/tests/vm/basevm.py b/tests/vm/basevm.py
index 23229e23d1..a97e23b0ce 100644
--- a/tests/vm/basevm.py
+++ b/tests/vm/basevm.py
@@ -27,6 +27,7 @@
 import multiprocessing
 import traceback
 import shlex
+import json
 
 from qemu.machine import QEMUMachine
 from qemu.utils import get_info_usernet_hostfwd_port, kvm_available
@@ -501,6 +502,16 @@ def gen_cloud_init_iso(self):
                               stderr=self._stdout)
         return os.path.join(cidir, "cloud-init.iso")
 
+    def get_qemu_packages_from_lcitool_json(self, json_path=None):
+        """Parse a lcitool variables json file and return the PKGS list."""
+        if json_path is None:
+            json_path = os.path.join(
+                os.path.dirname(__file__), "generated", self.name + ".json"
+            )
+        with open(json_path, "r") as fh:
+            return json.load(fh)["pkgs"]
+
+
 def get_qemu_path(arch, build_path=None):
     """Fetch the path to the qemu binary."""
     # If QEMU environment variable set, it takes precedence
-- 
2.38.1

[PATCH v3 4/4] tests/vm/freebsd: Get up-to-date package list from lcitool vars file

[Philippe Mathieu-Daudé]

Get an up-to-date package list from lcitool, that way we
don't need to manually keep this array in sync.

Inspired-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
---
 tests/vm/freebsd | 42 ++----------------------------------------
 1 file changed, 2 insertions(+), 40 deletions(-)

diff --git a/tests/vm/freebsd b/tests/vm/freebsd
index 11de6473f4..ac51376c82 100755
--- a/tests/vm/freebsd
+++ b/tests/vm/freebsd
@@ -31,45 +31,6 @@ class FreeBSDVM(basevm.BaseVM):
     link = "https://download.freebsd.org/releases/CI-IMAGES/13.2-RELEASE/amd64/Latest/FreeBSD-13.2-RELEASE-amd64-BASIC-CI.raw.xz"
     csum = "a4fb3b6c7b75dd4d58fb0d75e4caf72844bffe0ca00e66459c028b198ffb3c0e"
     size = "20G"
-    pkgs = [
-        # build tools
-        "git",
-        "pkgconf",
-        "bzip2",
-        "python39",
-        "ninja",
-
-        # gnu tools
-        "bash",
-        "gmake",
-        "gsed",
-        "gettext",
-
-        # libs: crypto
-        "gnutls",
-
-        # libs: images
-        "jpeg-turbo",
-        "png",
-
-        # libs: ui
-        "sdl2",
-        "gtk3",
-        "libxkbcommon",
-
-        # libs: opengl
-        "libepoxy",
-        "mesa-libs",
-
-        # libs: migration
-        "zstd",
-
-        # libs: networking
-        "libslirp",
-
-        # libs: sndio
-        "sndio",
-    ]
 
     BUILD_SCRIPT = """
         set -e;
@@ -151,8 +112,9 @@ class FreeBSDVM(basevm.BaseVM):
         self.console_wait(prompt)
         self.console_send("echo 'chmod 666 /dev/vtbd1' >> /etc/rc.local\n")
 
+        pkgs = self.get_qemu_packages_from_lcitool_json()
         self.print_step("Installing packages")
-        self.ssh_root_check("pkg install -y %s\n" % " ".join(self.pkgs))
+        self.ssh_root_check("pkg install -y %s\n" % " ".join(pkgs))
 
         # shutdown
         self.ssh_root(self.poweroff)
-- 
2.38.1

Re: [PATCH v3 1/4] tests/lcitool: Generate distribution packages list in JSON format

[Daniel P. Berrangé]

On Tue, Jul 11, 2023 at 04:49:19PM +0200, Philippe Mathieu-Daudé wrote:
> Add the generate_pkglist() helper to generate a list of packages
> required by a distribution to build QEMU.
> 
> Since we can not add a "THIS FILE WAS AUTO-GENERATED" comment in
> JSON, create the files under tests/vm/generated/ sub-directory;
> add a README mentioning the files are generated.
> 
> Suggested-by: Erik Skultety <eskultet@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/lcitool/refresh     | 11 +++++++++++
>  tests/vm/generated/README |  5 +++++

Or 's/generated/packages/' perhaps ?

>  2 files changed, 16 insertions(+)
>  create mode 100644 tests/vm/generated/README

Either way, at your discretion

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Daniel P. Berrangé]

On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
> Refresh the generated files by running:
> 
>   $ make lcitool-refresh
> 
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/docker/dockerfiles/debian-amd64.docker |  2 -
>  tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
>  tests/docker/dockerfiles/ubuntu2204.docker   |  2 -

I don't know why this would be removing xen/pmem from these files. If
I run 'lcitool-refresh' on current git master that doesn't happen

Do you have any other local changes on top ?

>  tests/vm/generated/freebsd.json              | 77 ++++++++++++++++++++
>  4 files changed, 77 insertions(+), 6 deletions(-)
>  create mode 100644 tests/vm/generated/freebsd.json

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


> 
> diff --git a/tests/docker/dockerfiles/debian-amd64.docker b/tests/docker/dockerfiles/debian-amd64.docker
> index e39871c7bb..8f7521fdc2 100644
> --- a/tests/docker/dockerfiles/debian-amd64.docker
> +++ b/tests/docker/dockerfiles/debian-amd64.docker
> @@ -70,7 +70,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libpam0g-dev \
>                        libpcre2-dev \
>                        libpixman-1-dev \
> -                      libpmem-dev \
>                        libpng-dev \
>                        libpulse-dev \
>                        librbd-dev \
> @@ -96,7 +95,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libvdeplug-dev \
>                        libvirglrenderer-dev \
>                        libvte-2.91-dev \
> -                      libxen-dev \
>                        libzstd-dev \
>                        llvm \
>                        locales \
> diff --git a/tests/docker/dockerfiles/ubuntu2004.docker b/tests/docker/dockerfiles/ubuntu2004.docker
> index 8f864d19e6..7f60143cbb 100644
> --- a/tests/docker/dockerfiles/ubuntu2004.docker
> +++ b/tests/docker/dockerfiles/ubuntu2004.docker
> @@ -69,7 +69,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libpam0g-dev \
>                        libpcre2-dev \
>                        libpixman-1-dev \
> -                      libpmem-dev \
>                        libpng-dev \
>                        libpulse-dev \
>                        librbd-dev \
> @@ -94,7 +93,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libvdeplug-dev \
>                        libvirglrenderer-dev \
>                        libvte-2.91-dev \
> -                      libxen-dev \
>                        libzstd-dev \
>                        llvm \
>                        locales \
> diff --git a/tests/docker/dockerfiles/ubuntu2204.docker b/tests/docker/dockerfiles/ubuntu2204.docker
> index 1d442cdfe6..5162927016 100644
> --- a/tests/docker/dockerfiles/ubuntu2204.docker
> +++ b/tests/docker/dockerfiles/ubuntu2204.docker
> @@ -70,7 +70,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libpam0g-dev \
>                        libpcre2-dev \
>                        libpixman-1-dev \
> -                      libpmem-dev \
>                        libpng-dev \
>                        libpulse-dev \
>                        librbd-dev \
> @@ -96,7 +95,6 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>                        libvdeplug-dev \
>                        libvirglrenderer-dev \
>                        libvte-2.91-dev \
> -                      libxen-dev \
>                        libzstd-dev \
>                        llvm \
>                        locales \
> diff --git a/tests/vm/generated/freebsd.json b/tests/vm/generated/freebsd.json
> new file mode 100644
> index 0000000000..7c435cf23e
> --- /dev/null
> +++ b/tests/vm/generated/freebsd.json
> @@ -0,0 +1,77 @@
> +{
> +  "ccache": "/usr/local/bin/ccache",
> +  "cpan_pkgs": [],
> +  "cross_pkgs": [],
> +  "make": "/usr/local/bin/gmake",
> +  "ninja": "/usr/local/bin/ninja",
> +  "packaging_command": "pkg",
> +  "pip3": "/usr/local/bin/pip-3.8",
> +  "pkgs": [
> +    "alsa-lib",
> +    "bash",
> +    "bison",
> +    "bzip2",
> +    "ca_root_nss",
> +    "capstone4",
> +    "ccache",
> +    "cmocka",
> +    "ctags",
> +    "curl",
> +    "cyrus-sasl",
> +    "dbus",
> +    "diffutils",
> +    "dtc",
> +    "flex",
> +    "fusefs-libs3",
> +    "gettext",
> +    "git",
> +    "glib",
> +    "gmake",
> +    "gnutls",
> +    "gsed",
> +    "gtk3",
> +    "json-c",
> +    "libepoxy",
> +    "libffi",
> +    "libgcrypt",
> +    "libjpeg-turbo",
> +    "libnfs",
> +    "libslirp",
> +    "libspice-server",
> +    "libssh",
> +    "libtasn1",
> +    "llvm",
> +    "lzo2",
> +    "meson",
> +    "mtools",
> +    "ncurses",
> +    "nettle",
> +    "ninja",
> +    "opencv",
> +    "pixman",
> +    "pkgconf",
> +    "png",
> +    "py39-numpy",
> +    "py39-pillow",
> +    "py39-pip",
> +    "py39-sphinx",
> +    "py39-sphinx_rtd_theme",
> +    "py39-yaml",
> +    "python3",
> +    "rpm2cpio",
> +    "sdl2",
> +    "sdl2_image",
> +    "snappy",
> +    "sndio",
> +    "socat",
> +    "spice-protocol",
> +    "tesseract",
> +    "usbredir",
> +    "virglrenderer",
> +    "vte3",
> +    "xorriso",
> +    "zstd"
> +  ],
> +  "pypi_pkgs": [],
> +  "python": "/usr/local/bin/python3"
> +}
> -- 
> 2.38.1
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 3/4] tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper

[Daniel P. Berrangé]

On Tue, Jul 11, 2023 at 04:49:21PM +0200, Philippe Mathieu-Daudé wrote:
> Add the get_qemu_packages_from_lcitool_json() helper which return
> such package list from a lcitool env var file in JSON format.
> 
> Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/vm/basevm.py | 11 +++++++++++
>  1 file changed, 11 insertions(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 4/4] tests/vm/freebsd: Get up-to-date package list from lcitool vars file

[Daniel P. Berrangé]

On Tue, Jul 11, 2023 at 04:49:22PM +0200, Philippe Mathieu-Daudé wrote:
> Get an up-to-date package list from lcitool, that way we
> don't need to manually keep this array in sync.
> 
> Inspired-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> ---
>  tests/vm/freebsd | 42 ++----------------------------------------
>  1 file changed, 2 insertions(+), 40 deletions(-)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 1/4] tests/lcitool: Generate distribution packages list in JSON format

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2193 bytes --]

On Tue, Jul 11, 2023 at 8:49 AM Philippe Mathieu-Daudé <philmd@linaro.org>
wrote:

> Add the generate_pkglist() helper to generate a list of packages
> required by a distribution to build QEMU.
>
> Since we can not add a "THIS FILE WAS AUTO-GENERATED" comment in
> JSON, create the files under tests/vm/generated/ sub-directory;
> add a README mentioning the files are generated.
>
> Suggested-by: Erik Skultety <eskultet@redhat.com>
> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>

Reviewed-by: Warner Losh <imp@bsdimp.com>

also, FreeBSD 14 branches next month... do I just grep for FreeBSD-13 to
find all the places to update for 14.0?

Warner

---
>  tests/lcitool/refresh     | 11 +++++++++++
>  tests/vm/generated/README |  5 +++++
>  2 files changed, 16 insertions(+)
>  create mode 100644 tests/vm/generated/README
>
> diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
> index b54566edcc..4584870ea1 100755
> --- a/tests/lcitool/refresh
> +++ b/tests/lcitool/refresh
> @@ -84,6 +84,12 @@ def generate_cirrus(target, trailer=None):
>      generate(filename, cmd, trailer)
>
>
> +def generate_pkglist(vm, target):
> +    filename = Path(src_dir, "tests", "vm", "generated", vm + ".json")
> +    cmd = lcitool_cmd + ["variables", "--format", "json", target, "qemu"]
> +    generate(filename, cmd, None)
> +
> +
>  # Netmap still needs to be manually built as it is yet to be packaged
>  # into a distro. We also add cscope and gtags which are used in the CI
>  # test
> @@ -191,6 +197,11 @@ try:
>      generate_cirrus("freebsd-13")
>      generate_cirrus("macos-12")
>
> +    #
> +    # VM packages lists
> +    #
> +    generate_pkglist("freebsd", "freebsd-13")
> +
>      sys.exit(0)
>  except Exception as ex:
>      print(str(ex), file=sys.stderr)
> diff --git a/tests/vm/generated/README b/tests/vm/generated/README
> new file mode 100644
> index 0000000000..7ccc6ffd3d
> --- /dev/null
> +++ b/tests/vm/generated/README
> @@ -0,0 +1,5 @@
> +# FILES IN THIS FOLDER WERE AUTO-GENERATED
> +#
> +#  $ make lcitool-refresh
> +#
> +# https://gitlab.com/libvirt/libvirt-ci
> --
> 2.38.1
>
>

[-- Attachment #2: Type: text/html, Size: 3249 bytes --]

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Philippe Mathieu-Daudé]

On 11/7/23 19:58, Daniel P. Berrangé wrote:
> On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
>> Refresh the generated files by running:
>>
>>    $ make lcitool-refresh
>>
>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> ---
>>   tests/docker/dockerfiles/debian-amd64.docker |  2 -
>>   tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
>>   tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
> 
> I don't know why this would be removing xen/pmem from these files. If
> I run 'lcitool-refresh' on current git master that doesn't happen
> 
> Do you have any other local changes on top ?

No...

$ git checkout origin/master
((887cba855b...))$ git submodule sync
Synchronizing submodule url for 'roms/QemuMacDrivers'
Synchronizing submodule url for 'roms/SLOF'
Synchronizing submodule url for 'roms/edk2'
Synchronizing submodule url for 'roms/ipxe'
Synchronizing submodule url for 'roms/openbios'
Synchronizing submodule url for 'roms/opensbi'
Synchronizing submodule url for 'roms/qboot'
Synchronizing submodule url for 'roms/qemu-palcode'
Synchronizing submodule url for 'roms/seabios'
Synchronizing submodule url for 'roms/seabios-hppa'
Synchronizing submodule url for 'roms/skiboot'
Synchronizing submodule url for 'roms/u-boot'
Synchronizing submodule url for 'roms/u-boot-sam460ex'
Synchronizing submodule url for 'roms/vbootrom'
Synchronizing submodule url for 'tests/lcitool/libvirt-ci'
((887cba855b...))$ git submodule update
((887cba855b...))$ git submodule status
  90c488d5f4a407342247b9ea869df1c2d9c8e266 roms/QemuMacDrivers 
(heads/master)
  6b6c16b4b40763507cf1f518096f3c3883c5cf2d roms/SLOF 
(qemu-slof-20190703-109-g6b6c16b)
  f80f052277c88a67c55e107b550f504eeea947d3 roms/edk2 
(edk2-stable201903-5245-gf80f052277)
  4bd064de239dab2426b31c9789a1f4d78087dc63 roms/ipxe (v1.20.1-70-g4bd064de)
  af97fd7af5e7c18f591a7b987291d3db4ffb28b5 roms/openbios 
(remotes/origin/HEAD)
  2552799a1df30a3dcd2321a8b75d61d06f5fb9fc roms/opensbi (v1.3)
  8ca302e86d685fa05b16e2b208888243da319941 roms/qboot (remotes/origin/HEAD)
  99d9b4dcf27d7fbcbadab71bdc88ef6531baf6bf roms/qemu-palcode (heads/master)
  ea1b7a0733906b8425d948ae94fba63c32b1d425 roms/seabios (rel-1.16.2)
  673d2595d4f773cc266cbf8dbaf2f475a6adb949 roms/seabios-hppa 
(seabios-hppa-v8)
  24a7eb35966d93455520bc2debdd7954314b638b roms/skiboot (v7.0)
  840658b093976390e9537724f802281c9c8439f5 roms/u-boot (v2021.07)
  60b3916f33e617a815973c5a6df77055b2e3a588 roms/u-boot-sam460ex 
(heads/master)
  0c37a43527f0ee2b9584e7fb2fdc805e902635ac roms/vbootrom (heads/master)
  b0f44f929a81c0a604fb7fbf8afc34d37ab0eae9 tests/lcitool/libvirt-ci 
(heads/master-555-gb0f44f92)
((887cba855b...))$ make -C build lcitool-refresh
Generate tests/docker/dockerfiles/alpine.docker
Generate tests/docker/dockerfiles/centos8.docker
Generate tests/docker/dockerfiles/debian-amd64.docker
Generate tests/docker/dockerfiles/fedora.docker
Generate tests/docker/dockerfiles/opensuse-leap.docker
Generate tests/docker/dockerfiles/ubuntu2004.docker
Generate tests/docker/dockerfiles/ubuntu2204.docker
Generate tests/docker/dockerfiles/debian-amd64-cross.docker
Generate tests/docker/dockerfiles/debian-arm64-cross.docker
Generate tests/docker/dockerfiles/debian-armel-cross.docker
Generate tests/docker/dockerfiles/debian-armhf-cross.docker
Generate tests/docker/dockerfiles/debian-mips64el-cross.docker
Generate tests/docker/dockerfiles/debian-mipsel-cross.docker
Generate tests/docker/dockerfiles/debian-ppc64el-cross.docker
Generate tests/docker/dockerfiles/debian-riscv64-cross.docker
Generate tests/docker/dockerfiles/debian-s390x-cross.docker
Generate tests/docker/dockerfiles/fedora-win32-cross.docker
Generate tests/docker/dockerfiles/fedora-win64-cross.docker
Generate .gitlab-ci.d/cirrus/freebsd-13.vars
Generate .gitlab-ci.d/cirrus/macos-12.vars
((887cba855b...))$ git diff -U0
diff --git a/tests/docker/dockerfiles/debian-amd64.docker 
b/tests/docker/dockerfiles/debian-amd64.docker
index e39871c7bb..8f7521fdc2 100644
--- a/tests/docker/dockerfiles/debian-amd64.docker
+++ b/tests/docker/dockerfiles/debian-amd64.docker
@@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
-                      libpmem-dev \
@@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
-                      libxen-dev \
diff --git a/tests/docker/dockerfiles/ubuntu2004.docker 
b/tests/docker/dockerfiles/ubuntu2004.docker
index 8f864d19e6..7f60143cbb 100644
--- a/tests/docker/dockerfiles/ubuntu2004.docker
+++ b/tests/docker/dockerfiles/ubuntu2004.docker
@@ -72 +71,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
-                      libpmem-dev \
@@ -97 +95,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
-                      libxen-dev \
diff --git a/tests/docker/dockerfiles/ubuntu2204.docker 
b/tests/docker/dockerfiles/ubuntu2204.docker
index 1d442cdfe6..5162927016 100644
--- a/tests/docker/dockerfiles/ubuntu2204.docker
+++ b/tests/docker/dockerfiles/ubuntu2204.docker
@@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
-                      libpmem-dev \
@@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
-                      libxen-dev \
((887cba855b...))$

🤷

Re: [PATCH v3 1/4] tests/lcitool: Generate distribution packages list in JSON format

[Philippe Mathieu-Daudé]

On 11/7/23 21:39, Warner Losh wrote:
> On Tue, Jul 11, 2023 at 8:49 AM Philippe Mathieu-Daudé 
> <philmd@linaro.org <mailto:philmd@linaro.org>> wrote:
> 
>     Add the generate_pkglist() helper to generate a list of packages
>     required by a distribution to build QEMU.
> 
>     Since we can not add a "THIS FILE WAS AUTO-GENERATED" comment in
>     JSON, create the files under tests/vm/generated/ sub-directory;
>     add a README mentioning the files are generated.
> 
>     Suggested-by: Erik Skultety <eskultet@redhat.com
>     <mailto:eskultet@redhat.com>>
>     Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org
>     <mailto:philmd@linaro.org>>
> 
> 
> Reviewed-by: Warner Losh <imp@bsdimp.com <mailto:imp@bsdimp.com>>
> 
> also, FreeBSD 14 branches next month... do I just grep for FreeBSD-13 to 
> find all the places to update for 14.0?

Per docs/about/build-platforms.rst:

   Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
   -----------------------------------------

   The project aims to support the most recent major version at all
   times for up to five years after its initial release. Support
   for the previous major version will be dropped 2 years after the
   new major version is released or when the vendor itself drops
   support, whichever comes first. [...]

We want to be able to test the oldest/newest releases.

Maybe we can add a pair of definitions, so we'd have to only
change 2 lines in a single place when releases occur?


>     @@ -191,6 +197,11 @@ try:
>           generate_cirrus("freebsd-13")
>           generate_cirrus("macos-12")
> 
>     +    #
>     +    # VM packages lists
>     +    #
>     +    generate_pkglist("freebsd", "freebsd-13")

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Daniel P. Berrangé]

On Wed, Jul 12, 2023 at 01:00:38PM +0200, Philippe Mathieu-Daudé wrote:
> On 11/7/23 19:58, Daniel P. Berrangé wrote:
> > On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
> > > Refresh the generated files by running:
> > > 
> > >    $ make lcitool-refresh
> > > 
> > > Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> > > ---
> > >   tests/docker/dockerfiles/debian-amd64.docker |  2 -
> > >   tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
> > >   tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
> > 
> > I don't know why this would be removing xen/pmem from these files. If
> > I run 'lcitool-refresh' on current git master that doesn't happen
> > 
> > Do you have any other local changes on top ?
> 


> ((887cba855b...))$ make -C build lcitool-refresh
> Generate tests/docker/dockerfiles/alpine.docker
> Generate tests/docker/dockerfiles/centos8.docker
> Generate tests/docker/dockerfiles/debian-amd64.docker
> Generate tests/docker/dockerfiles/fedora.docker
> Generate tests/docker/dockerfiles/opensuse-leap.docker
> Generate tests/docker/dockerfiles/ubuntu2004.docker
> Generate tests/docker/dockerfiles/ubuntu2204.docker
> Generate tests/docker/dockerfiles/debian-amd64-cross.docker
> Generate tests/docker/dockerfiles/debian-arm64-cross.docker
> Generate tests/docker/dockerfiles/debian-armel-cross.docker
> Generate tests/docker/dockerfiles/debian-armhf-cross.docker
> Generate tests/docker/dockerfiles/debian-mips64el-cross.docker
> Generate tests/docker/dockerfiles/debian-mipsel-cross.docker
> Generate tests/docker/dockerfiles/debian-ppc64el-cross.docker
> Generate tests/docker/dockerfiles/debian-riscv64-cross.docker
> Generate tests/docker/dockerfiles/debian-s390x-cross.docker
> Generate tests/docker/dockerfiles/fedora-win32-cross.docker
> Generate tests/docker/dockerfiles/fedora-win64-cross.docker
> Generate .gitlab-ci.d/cirrus/freebsd-13.vars
> Generate .gitlab-ci.d/cirrus/macos-12.vars
> ((887cba855b...))$ git diff -U0
> diff --git a/tests/docker/dockerfiles/debian-amd64.docker
> b/tests/docker/dockerfiles/debian-amd64.docker
> index e39871c7bb..8f7521fdc2 100644
> --- a/tests/docker/dockerfiles/debian-amd64.docker
> +++ b/tests/docker/dockerfiles/debian-amd64.docker
> @@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> -                      libpmem-dev \
> @@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> -                      libxen-dev \
> diff --git a/tests/docker/dockerfiles/ubuntu2004.docker
> b/tests/docker/dockerfiles/ubuntu2004.docker
> index 8f864d19e6..7f60143cbb 100644
> --- a/tests/docker/dockerfiles/ubuntu2004.docker
> +++ b/tests/docker/dockerfiles/ubuntu2004.docker
> @@ -72 +71,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> -                      libpmem-dev \
> @@ -97 +95,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> -                      libxen-dev \
> diff --git a/tests/docker/dockerfiles/ubuntu2204.docker
> b/tests/docker/dockerfiles/ubuntu2204.docker
> index 1d442cdfe6..5162927016 100644
> --- a/tests/docker/dockerfiles/ubuntu2204.docker
> +++ b/tests/docker/dockerfiles/ubuntu2204.docker
> @@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> -                      libpmem-dev \
> @@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> -                      libxen-dev \

What architecture are you running from ? I'm suspecting it is a non
x86_64 architecture as these pacakges are both arch conditionalized.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Philippe Mathieu-Daudé]

On 12/7/23 13:12, Daniel P. Berrangé wrote:
> On Wed, Jul 12, 2023 at 01:00:38PM +0200, Philippe Mathieu-Daudé wrote:
>> On 11/7/23 19:58, Daniel P. Berrangé wrote:
>>> On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
>>>> Refresh the generated files by running:
>>>>
>>>>     $ make lcitool-refresh
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>>>> ---
>>>>    tests/docker/dockerfiles/debian-amd64.docker |  2 -
>>>>    tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
>>>>    tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
>>>
>>> I don't know why this would be removing xen/pmem from these files. If
>>> I run 'lcitool-refresh' on current git master that doesn't happen
>>>
>>> Do you have any other local changes on top ?

(I just noticed manually updating the libvirt-ci submodule is
  pointless because the 'lcitool-refresh' rule does that)

>> diff --git a/tests/docker/dockerfiles/ubuntu2204.docker
>> b/tests/docker/dockerfiles/ubuntu2204.docker
>> index 1d442cdfe6..5162927016 100644
>> --- a/tests/docker/dockerfiles/ubuntu2204.docker
>> +++ b/tests/docker/dockerfiles/ubuntu2204.docker
>> @@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>> -                      libpmem-dev \
>> @@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>> -                      libxen-dev \
> 
> What architecture are you running from ? I'm suspecting it is a non
> x86_64 architecture as these pacakges are both arch conditionalized.

My host is Darwin aarch64, but how is this relevant to what we
generate for guests? Are we missing specifying the target arch
somewhere? (This is not the '--cross-arch' argument, because we
don't want to cross-build). These dockerfiles always targeted x86_64,
in particular the debian-amd64.docker one...

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Daniel P. Berrangé]

On Wed, Jul 12, 2023 at 02:55:10PM +0200, Philippe Mathieu-Daudé wrote:
> On 12/7/23 13:12, Daniel P. Berrangé wrote:
> > On Wed, Jul 12, 2023 at 01:00:38PM +0200, Philippe Mathieu-Daudé wrote:
> > > On 11/7/23 19:58, Daniel P. Berrangé wrote:
> > > > On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
> > > > > Refresh the generated files by running:
> > > > > 
> > > > >     $ make lcitool-refresh
> > > > > 
> > > > > Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> > > > > ---
> > > > >    tests/docker/dockerfiles/debian-amd64.docker |  2 -
> > > > >    tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
> > > > >    tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
> > > > 
> > > > I don't know why this would be removing xen/pmem from these files. If
> > > > I run 'lcitool-refresh' on current git master that doesn't happen
> > > > 
> > > > Do you have any other local changes on top ?
> 
> (I just noticed manually updating the libvirt-ci submodule is
>  pointless because the 'lcitool-refresh' rule does that)
> 
> > > diff --git a/tests/docker/dockerfiles/ubuntu2204.docker
> > > b/tests/docker/dockerfiles/ubuntu2204.docker
> > > index 1d442cdfe6..5162927016 100644
> > > --- a/tests/docker/dockerfiles/ubuntu2204.docker
> > > +++ b/tests/docker/dockerfiles/ubuntu2204.docker
> > > @@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> > > -                      libpmem-dev \
> > > @@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
> > > -                      libxen-dev \
> > 
> > What architecture are you running from ? I'm suspecting it is a non
> > x86_64 architecture as these pacakges are both arch conditionalized.
> 
> My host is Darwin aarch64, but how is this relevant to what we
> generate for guests? Are we missing specifying the target arch
> somewhere? (This is not the '--cross-arch' argument, because we
> don't want to cross-build). These dockerfiles always targeted x86_64,
> in particular the debian-amd64.docker one...

This is an unfortunate design choice of lcitool.

If you give '--cross-arch' it will always spit out the dockerfile
suitable for cross-compiling to that arch.

If you omit '--cross-arch' it will always spit out the dockerfile
suitable for compiling on the *current* host arch.

When we're committing the dockerfiles to qemu (or any libvirt project
using lcitool), we've got an implicit assumption that the non-cross
dockerfiles were for x86_64, and hence an implicit assumption that
the person who ran 'lcitool' was also on x86_64.

This is clearly a bad design choice mistake in retrospect as people
are increasingly likely to be using aarch64 for day-to-day dev work.

So I guess we need to come up with a more explicit way to say give
me a native container for x86_64.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 1/4] tests/lcitool: Generate distribution packages list in JSON format

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

On Wed, Jul 12, 2023 at 5:07 AM Philippe Mathieu-Daudé <philmd@linaro.org>
wrote:

> On 11/7/23 21:39, Warner Losh wrote:
> > On Tue, Jul 11, 2023 at 8:49 AM Philippe Mathieu-Daudé
> > <philmd@linaro.org <mailto:philmd@linaro.org>> wrote:
> >
> >     Add the generate_pkglist() helper to generate a list of packages
> >     required by a distribution to build QEMU.
> >
> >     Since we can not add a "THIS FILE WAS AUTO-GENERATED" comment in
> >     JSON, create the files under tests/vm/generated/ sub-directory;
> >     add a README mentioning the files are generated.
> >
> >     Suggested-by: Erik Skultety <eskultet@redhat.com
> >     <mailto:eskultet@redhat.com>>
> >     Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org
> >     <mailto:philmd@linaro.org>>
> >
> >
> > Reviewed-by: Warner Losh <imp@bsdimp.com <mailto:imp@bsdimp.com>>
> >
> > also, FreeBSD 14 branches next month... do I just grep for FreeBSD-13 to
> > find all the places to update for 14.0?
>
> Per docs/about/build-platforms.rst:
>
>    Linux OS, macOS, FreeBSD, NetBSD, OpenBSD
>    -----------------------------------------
>
>    The project aims to support the most recent major version at all
>    times for up to five years after its initial release. Support
>    for the previous major version will be dropped 2 years after the
>    new major version is released or when the vendor itself drops
>    support, whichever comes first. [...]
>
> We want to be able to test the oldest/newest releases.
>
> Maybe we can add a pair of definitions, so we'd have to only
> change 2 lines in a single place when releases occur?
>

I'd love that. There's two bits of information per release: name and
checksum
of release artifacts.

Warner


>
> >     @@ -191,6 +197,11 @@ try:
> >           generate_cirrus("freebsd-13")
> >           generate_cirrus("macos-12")
> >
> >     +    #
> >     +    # VM packages lists
> >     +    #
> >     +    generate_pkglist("freebsd", "freebsd-13")
>
>

[-- Attachment #2: Type: text/html, Size: 3346 bytes --]

Re: [PATCH v3 0/4] tests/vm/freebsd: Get up-to-date package list from lcitool

[Thomas Huth]

On 11/07/2023 16.49, Philippe Mathieu-Daudé wrote:
> Inspired by this patch from Thomas:
> https://lore.kernel.org/qemu-devel/20230531090415.40421-1-thuth@redhat.com/
> 
> Instead of updating the package list manually, use lcitool vars file.
> 
> Since v2:
> - Commit generated json (Daniel)
> 
> Since v1:
> - Addressed Erik & Daniel comments (generate in JSON)
> 
> Philippe Mathieu-Daudé (4):
>    tests/lcitool: Generate distribution packages list in JSON format
>    tests/lcitool: Refresh generated files
>    tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
>    tests/vm/freebsd: Get up-to-date package list from lcitool vars file
> 
>   tests/docker/dockerfiles/debian-amd64.docker |  2 -
>   tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
>   tests/docker/dockerfiles/ubuntu2204.docker   |  2 -

Thanks, I'm queuing your patches for my next pull request - but I'm dropping 
the hunks that change libpmem-dev and libxen-dev in the docker files - 
otherwise this would just get reverted once somebody using a x86 host runs 
the refresh step again. I hope this issue can be tackled by a future lcitool 
update instead.

  Thomas

Re: [PATCH v3 0/4] tests/vm/freebsd: Get up-to-date package list from lcitool

[Erik Skultety]

On Mon, Jul 17, 2023 at 02:18:34PM +0200, Thomas Huth wrote:
> On 11/07/2023 16.49, Philippe Mathieu-Daudé wrote:
> > Inspired by this patch from Thomas:
> > https://lore.kernel.org/qemu-devel/20230531090415.40421-1-thuth@redhat.com/
> > 
> > Instead of updating the package list manually, use lcitool vars file.
> > 
> > Since v2:
> > - Commit generated json (Daniel)
> > 
> > Since v1:
> > - Addressed Erik & Daniel comments (generate in JSON)
> > 
> > Philippe Mathieu-Daudé (4):
> >    tests/lcitool: Generate distribution packages list in JSON format
> >    tests/lcitool: Refresh generated files
> >    tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
> >    tests/vm/freebsd: Get up-to-date package list from lcitool vars file
> > 
> >   tests/docker/dockerfiles/debian-amd64.docker |  2 -
> >   tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
> >   tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
> 
> Thanks, I'm queuing your patches for my next pull request - but I'm dropping
> the hunks that change libpmem-dev and libxen-dev in the docker files -
> otherwise this would just get reverted once somebody using a x86 host runs
> the refresh step again. I hope this issue can be tackled by a future lcitool
> update instead.
> 
>  Thomas
> 

Can you guys please create an issue at
https://gitlab.com/libvirt/libvirt-ci.git to have this officially tracked?

Thanks,
Erik

Re: [PATCH v3 0/4] tests/vm/freebsd: Get up-to-date package list from lcitool

[Daniel P. Berrangé]

On Mon, Jul 17, 2023 at 02:30:03PM +0200, Erik Skultety wrote:
> On Mon, Jul 17, 2023 at 02:18:34PM +0200, Thomas Huth wrote:
> > On 11/07/2023 16.49, Philippe Mathieu-Daudé wrote:
> > > Inspired by this patch from Thomas:
> > > https://lore.kernel.org/qemu-devel/20230531090415.40421-1-thuth@redhat.com/
> > > 
> > > Instead of updating the package list manually, use lcitool vars file.
> > > 
> > > Since v2:
> > > - Commit generated json (Daniel)
> > > 
> > > Since v1:
> > > - Addressed Erik & Daniel comments (generate in JSON)
> > > 
> > > Philippe Mathieu-Daudé (4):
> > >    tests/lcitool: Generate distribution packages list in JSON format
> > >    tests/lcitool: Refresh generated files
> > >    tests/vm: Introduce get_qemu_packages_from_lcitool_json() helper
> > >    tests/vm/freebsd: Get up-to-date package list from lcitool vars file
> > > 
> > >   tests/docker/dockerfiles/debian-amd64.docker |  2 -
> > >   tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
> > >   tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
> > 
> > Thanks, I'm queuing your patches for my next pull request - but I'm dropping
> > the hunks that change libpmem-dev and libxen-dev in the docker files -
> > otherwise this would just get reverted once somebody using a x86 host runs
> > the refresh step again. I hope this issue can be tackled by a future lcitool
> > update instead.
> > 
> >  Thomas
> > 
> 
> Can you guys please create an issue at
> https://gitlab.com/libvirt/libvirt-ci.git to have this officially tracked?

Done:

https://gitlab.com/libvirt/libvirt-ci/-/issues/26


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Philippe Mathieu-Daudé]

On 12/7/23 15:07, Daniel P. Berrangé wrote:
> On Wed, Jul 12, 2023 at 02:55:10PM +0200, Philippe Mathieu-Daudé wrote:
>> On 12/7/23 13:12, Daniel P. Berrangé wrote:
>>> On Wed, Jul 12, 2023 at 01:00:38PM +0200, Philippe Mathieu-Daudé wrote:
>>>> On 11/7/23 19:58, Daniel P. Berrangé wrote:
>>>>> On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
>>>>>> Refresh the generated files by running:
>>>>>>
>>>>>>      $ make lcitool-refresh
>>>>>>
>>>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>>>>>> ---
>>>>>>     tests/docker/dockerfiles/debian-amd64.docker |  2 -
>>>>>>     tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
>>>>>>     tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
>>>>>
>>>>> I don't know why this would be removing xen/pmem from these files. If
>>>>> I run 'lcitool-refresh' on current git master that doesn't happen

FTR since commit cb039ef3d9 libxdp-devel is also being changed on my
host, similarly to libpmem-devel, so I suppose it also has some host
specific restriction.

>>>>> Do you have any other local changes on top ?
>>
>> (I just noticed manually updating the libvirt-ci submodule is
>>   pointless because the 'lcitool-refresh' rule does that)
>>
>>>> diff --git a/tests/docker/dockerfiles/ubuntu2204.docker
>>>> b/tests/docker/dockerfiles/ubuntu2204.docker
>>>> index 1d442cdfe6..5162927016 100644
>>>> --- a/tests/docker/dockerfiles/ubuntu2204.docker
>>>> +++ b/tests/docker/dockerfiles/ubuntu2204.docker
>>>> @@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>>>> -                      libpmem-dev \
>>>> @@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>>>> -                      libxen-dev \
>>>
>>> What architecture are you running from ? I'm suspecting it is a non
>>> x86_64 architecture as these pacakges are both arch conditionalized.
>>
>> My host is Darwin aarch64, but how is this relevant to what we
>> generate for guests? Are we missing specifying the target arch
>> somewhere? (This is not the '--cross-arch' argument, because we
>> don't want to cross-build). These dockerfiles always targeted x86_64,
>> in particular the debian-amd64.docker one...
> 
> This is an unfortunate design choice of lcitool.
> 
> If you give '--cross-arch' it will always spit out the dockerfile
> suitable for cross-compiling to that arch.
> 
> If you omit '--cross-arch' it will always spit out the dockerfile
> suitable for compiling on the *current* host arch.
> 
> When we're committing the dockerfiles to qemu (or any libvirt project
> using lcitool), we've got an implicit assumption that the non-cross
> dockerfiles were for x86_64, and hence an implicit assumption that
> the person who ran 'lcitool' was also on x86_64.
> 
> This is clearly a bad design choice mistake in retrospect as people
> are increasingly likely to be using aarch64 for day-to-day dev work.
> 
> So I guess we need to come up with a more explicit way to say give
> me a native container for x86_64.
> 
> With regards,
> Daniel

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Ilya Maximets]

On 1/2/24 19:19, Philippe Mathieu-Daudé wrote:
> On 12/7/23 15:07, Daniel P. Berrangé wrote:
>> On Wed, Jul 12, 2023 at 02:55:10PM +0200, Philippe Mathieu-Daudé wrote:
>>> On 12/7/23 13:12, Daniel P. Berrangé wrote:
>>>> On Wed, Jul 12, 2023 at 01:00:38PM +0200, Philippe Mathieu-Daudé wrote:
>>>>> On 11/7/23 19:58, Daniel P. Berrangé wrote:
>>>>>> On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
>>>>>>> Refresh the generated files by running:
>>>>>>>
>>>>>>>      $ make lcitool-refresh
>>>>>>>
>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>>>>>>> ---
>>>>>>>     tests/docker/dockerfiles/debian-amd64.docker |  2 -
>>>>>>>     tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
>>>>>>>     tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
>>>>>>
>>>>>> I don't know why this would be removing xen/pmem from these files. If
>>>>>> I run 'lcitool-refresh' on current git master that doesn't happen
> 
> FTR since commit cb039ef3d9 libxdp-devel is also being changed on my
> host, similarly to libpmem-devel, so I suppose it also has some host
> specific restriction.

Yeah, many distributions are not building libxdp for non-x86
architectures today.  There are no real reasons not to, but
they just do not, because the package is relatively new.  It
will likely become available in the future.

I see this thread is a bit old.  Was a solution found for the
pmem/xen?  i.e. do I need to do something specific for libxdp
at this point?

Best regards, Ilya Maximets.

> 
>>>>>> Do you have any other local changes on top ?
>>>
>>> (I just noticed manually updating the libvirt-ci submodule is
>>>   pointless because the 'lcitool-refresh' rule does that)
>>>
>>>>> diff --git a/tests/docker/dockerfiles/ubuntu2204.docker
>>>>> b/tests/docker/dockerfiles/ubuntu2204.docker
>>>>> index 1d442cdfe6..5162927016 100644
>>>>> --- a/tests/docker/dockerfiles/ubuntu2204.docker
>>>>> +++ b/tests/docker/dockerfiles/ubuntu2204.docker
>>>>> @@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>>>>> -                      libpmem-dev \
>>>>> @@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>>>>> -                      libxen-dev \
>>>>
>>>> What architecture are you running from ? I'm suspecting it is a non
>>>> x86_64 architecture as these pacakges are both arch conditionalized.
>>>
>>> My host is Darwin aarch64, but how is this relevant to what we
>>> generate for guests? Are we missing specifying the target arch
>>> somewhere? (This is not the '--cross-arch' argument, because we
>>> don't want to cross-build). These dockerfiles always targeted x86_64,
>>> in particular the debian-amd64.docker one...
>>
>> This is an unfortunate design choice of lcitool.
>>
>> If you give '--cross-arch' it will always spit out the dockerfile
>> suitable for cross-compiling to that arch.
>>
>> If you omit '--cross-arch' it will always spit out the dockerfile
>> suitable for compiling on the *current* host arch.
>>
>> When we're committing the dockerfiles to qemu (or any libvirt project
>> using lcitool), we've got an implicit assumption that the non-cross
>> dockerfiles were for x86_64, and hence an implicit assumption that
>> the person who ran 'lcitool' was also on x86_64.
>>
>> This is clearly a bad design choice mistake in retrospect as people
>> are increasingly likely to be using aarch64 for day-to-day dev work.
>>
>> So I guess we need to come up with a more explicit way to say give
>> me a native container for x86_64.
>>
>> With regards,
>> Daniel
> 

Re: [PATCH v3 2/4] tests/lcitool: Refresh generated files

[Philippe Mathieu-Daudé]

On 2/1/24 19:36, Ilya Maximets wrote:
> On 1/2/24 19:19, Philippe Mathieu-Daudé wrote:
>> On 12/7/23 15:07, Daniel P. Berrangé wrote:
>>> On Wed, Jul 12, 2023 at 02:55:10PM +0200, Philippe Mathieu-Daudé wrote:
>>>> On 12/7/23 13:12, Daniel P. Berrangé wrote:
>>>>> On Wed, Jul 12, 2023 at 01:00:38PM +0200, Philippe Mathieu-Daudé wrote:
>>>>>> On 11/7/23 19:58, Daniel P. Berrangé wrote:
>>>>>>> On Tue, Jul 11, 2023 at 04:49:20PM +0200, Philippe Mathieu-Daudé wrote:
>>>>>>>> Refresh the generated files by running:
>>>>>>>>
>>>>>>>>       $ make lcitool-refresh
>>>>>>>>
>>>>>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>>>>>>>> ---
>>>>>>>>      tests/docker/dockerfiles/debian-amd64.docker |  2 -
>>>>>>>>      tests/docker/dockerfiles/ubuntu2004.docker   |  2 -
>>>>>>>>      tests/docker/dockerfiles/ubuntu2204.docker   |  2 -
>>>>>>>
>>>>>>> I don't know why this would be removing xen/pmem from these files. If
>>>>>>> I run 'lcitool-refresh' on current git master that doesn't happen
>>
>> FTR since commit cb039ef3d9 libxdp-devel is also being changed on my
>> host, similarly to libpmem-devel, so I suppose it also has some host
>> specific restriction.
> 
> Yeah, many distributions are not building libxdp for non-x86
> architectures today.  There are no real reasons not to, but
> they just do not, because the package is relatively new.  It
> will likely become available in the future.
> 
> I see this thread is a bit old.  Was a solution found for the
> pmem/xen?  i.e. do I need to do something specific for libxdp
> at this point?

Maybe this is tracked in this issue?
https://gitlab.com/libvirt/libvirt-ci/-/issues/30

Otherwise libvirt/lcitool maintainers are Cc'ed :)

> Best regards, Ilya Maximets.
> 
>>
>>>>>>> Do you have any other local changes on top ?
>>>>
>>>> (I just noticed manually updating the libvirt-ci submodule is
>>>>    pointless because the 'lcitool-refresh' rule does that)
>>>>
>>>>>> diff --git a/tests/docker/dockerfiles/ubuntu2204.docker
>>>>>> b/tests/docker/dockerfiles/ubuntu2204.docker
>>>>>> index 1d442cdfe6..5162927016 100644
>>>>>> --- a/tests/docker/dockerfiles/ubuntu2204.docker
>>>>>> +++ b/tests/docker/dockerfiles/ubuntu2204.docker
>>>>>> @@ -73 +72,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>>>>>> -                      libpmem-dev \
>>>>>> @@ -99 +97,0 @@ RUN export DEBIAN_FRONTEND=noninteractive && \
>>>>>> -                      libxen-dev \
>>>>>
>>>>> What architecture are you running from ? I'm suspecting it is a non
>>>>> x86_64 architecture as these pacakges are both arch conditionalized.
>>>>
>>>> My host is Darwin aarch64, but how is this relevant to what we
>>>> generate for guests? Are we missing specifying the target arch
>>>> somewhere? (This is not the '--cross-arch' argument, because we
>>>> don't want to cross-build). These dockerfiles always targeted x86_64,
>>>> in particular the debian-amd64.docker one...
>>>
>>> This is an unfortunate design choice of lcitool.
>>>
>>> If you give '--cross-arch' it will always spit out the dockerfile
>>> suitable for cross-compiling to that arch.
>>>
>>> If you omit '--cross-arch' it will always spit out the dockerfile
>>> suitable for compiling on the *current* host arch.
>>>
>>> When we're committing the dockerfiles to qemu (or any libvirt project
>>> using lcitool), we've got an implicit assumption that the non-cross
>>> dockerfiles were for x86_64, and hence an implicit assumption that
>>> the person who ran 'lcitool' was also on x86_64.
>>>
>>> This is clearly a bad design choice mistake in retrospect as people
>>> are increasingly likely to be using aarch64 for day-to-day dev work.
>>>
>>> So I guess we need to come up with a more explicit way to say give
>>> me a native container for x86_64.
>>>
>>> With regards,
>>> Daniel
>>
>