debian cloud collaboration for kdevops

debian cloud collaboration for kdevops

[Luis Chamberlain]

Ross,

kdevops now uses guestfs [0] and it has its own respective set of base
images for each distro [1], it occurs to me that as debian puts out
debian testing vagrant images, wouldn't it make sense to put out also
guestfs debian testintg base images, and perhaps more reguarly up to
date as with vagrant?

Would this be possible? If that is, would a respectgive -dev postfix
image be possible which already comes with some things which we expect
in a kernel development environment (like grub dev console setup, we
have a list off things we do in kdevops for it)?

[0] https://libguestfs.org/
[1] https://builder.libguestfs.org/

  Luis

Re: debian cloud collaboration for kdevops

[Ross Vandegrift]

Hi Luis,

On Thu, Mar 28, 2024 at 10:41:59AM -0700, Luis Chamberlain wrote:
> kdevops now uses guestfs [0] and it has its own respective set of base
> images for each distro [1], it occurs to me that as debian puts out
> debian testing vagrant images, wouldn't it make sense to put out also
> guestfs debian testintg base images, and perhaps more reguarly up to
> date as with vagrant?

I'm not sure I understand your ask - when I've used libguestfs, it's
able to access lots of different image formats.  Does kdevops require a
specific format?  Based on the builder.libguestfs.org images, here are
the latest trixie builds that I'd expect to work:

raw: https://cloud.debian.org/images/cloud/trixie/daily/20240329-1701/debian-13-nocloud-arm64-daily-20240329-1701.tar.xz
qcow2: https://cloud.debian.org/images/cloud/trixie/daily/20240329-1701/debian-13-nocloud-arm64-daily-20240329-1701.qcow2

If you browse around, there's also amd64 & ppc64el images, and a nocloud
variant that should boot on bare metal.  I'll take a shot, after I catch
up on the newer kdevops work.


> Would this be possible? If that is, would a respectgive -dev postfix
> image be possible which already comes with some things which we expect
> in a kernel development environment (like grub dev console setup, we
> have a list off things we do in kdevops for it)?

Debian won't publish modified images for this purpose - but time
permitting, I may be able to help with unofficial images with those
changes.  My time's been a bit slim in recent months, though I expect
this will be easier than vagrant.

Ross

Re: debian cloud collaboration for kdevops

[Luis Chamberlain]

Given I have issues posting to the guestfs mailing list I'm just CC'ing
rjones foro now (Sorry Richard!)

Richard, do we have a way for virt-builder to deal with daily images
changes? Seems distros may just use a symlink to the latest directory
but what if the index file upstream changes daily?

On Fri, Mar 29, 2024 at 04:52:30PM -0700, Ross Vandegrift wrote:
> Hi Luis,
> 
> On Thu, Mar 28, 2024 at 10:41:59AM -0700, Luis Chamberlain wrote:
> > kdevops now uses guestfs [0] and it has its own respective set of base
> > images for each distro [1], it occurs to me that as debian puts out
> > debian testing vagrant images, wouldn't it make sense to put out also
> > guestfs debian testintg base images, and perhaps more reguarly up to
> > date as with vagrant?
> 
> I'm not sure I understand your ask - when I've used libguestfs, it's
> able to access lots of different image formats.  Does kdevops require a
> specific format?  Based on the builder.libguestfs.org images, here are
> the latest trixie builds that I'd expect to work:

> raw: https://cloud.debian.org/images/cloud/trixie/daily/20240329-1701/debian-13-nocloud-arm64-daily-20240329-1701.tar.xz
> qcow2: https://cloud.debian.org/images/cloud/trixie/daily/20240329-1701/debian-13-nocloud-arm64-daily-20240329-1701.qcow2

Either of these work nicely!

I see the symlink with "latest" on the daily directory works better for
a static directory:

Debian 12:
https://cloud.debian.org/images/cloud/bookworm/
Debian 13, debian testing:
https://cloud.debian.org/images/cloud/trixie/daily/latest/
Unstable:
https://cloud.debian.org/images/cloud/sid/daily/latest/

Now we just need a published index file for these, let me demonstrate
what the opensuse folks did with the upstream package, they just added
the opensuse.conf and a opensuse.gpg used to sign the index file file with:

cat /etc/virt-builder/repos.d/opensuse.conf 
[opensuse.org]
uri=http://download.opensuse.org/repositories/Virtualization:/virt-builder-images/images/index
gpgkey=file:///etc/virt-builder/repos.d/opensuse.gpg

Let's look at the index file:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

[opensuse-tumbleweed]
name=openSUSE Tumbleweed
osinfo=opensuse-factory
arch=x86_64
file=openSUSE-Tumbleweed.x86_64-2022.11.22-Build8.411.qcow2.xz
checksum=8e0a60d5df90a0ae4127ec0a006a0e5d30f36a7a4a23a66ed63ce8df934ab25568f663e73b3b5defecf5ae5c44bfb0ebc831c903b212bca2c68346049b18401c
revision=202211228411
format=qcow2
size=6442450944
compressed_size=187566376
expand=/dev/sda1
notes=This openSUSE Tumbleweed image only contains the base pattern.
 It is designed to run as either KVM or Xen HVM guest.

 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.0.7 (GNU/Linux)

 iQEVAwUBZgiPNqGT+7VyF0/CAQgwiwgAlb8wKbPLzB0QwLOM7QV+TN2Qp4eQsOHj
 Sjq2GrND/trf1IXQi07B4d7XvL0XmkJY8HoCHYR2+bN3Yef/0HQeDH8Nb9ly1QOA
 YZFGtNyjx8tkrpCiePmRAyPZw9T7F6Pz3He6O0broOqp+71Cxv4DtfscdNj8ady/
 pQE4KLe2xjpjoqGt+4kwA684Jlp88k5wHHyDGZpGJogdU00CtfCSX7VcbA72Cd4+
 lQUw8XI0GhQSl75MvDlAXifHsRR4uHlmerN3l1ZeamEBrzHT5l8ZAvAMhvmayvH3
 wnxaaVt18eM3+hWW5fitIaN4EBsoFPlMaMP3ulGIVx8+Mw2+PvZGqw==
 =9Sz5
 -----END PGP SIGNATURE-----

The checksum is the sha512sum on the xz file. This is already computed
in the file https://cloud.debian.org/images/cloud/sid/daily/latest/SHA512SUMS

So I take it we'd need two files created on these directories:

  * index
  * index.gpg

I checked the bookworm tarball:

https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-nocloud-amd64.tar.xz

This just has *one* file: disk.raw

> If you browse around, there's also amd64 & ppc64el images, and a nocloud
> variant that should boot on bare metal.

Neat!

> I'll take a shot, after I catch
> up on the newer kdevops work.

We can add the support for adding a custom kdevops-debian.conf until we
get a respective upstream package for debian with its own conf file, and
we can also add the Kconfig options on kdevops to allow you to pick the
distro. But let's start out small. Just getting debian trixie up with
an index I think would be my first preferred target on debian. If debian
can set the index file and index.gpg funcitonal we can work by testing it.

> > Would this be possible? If that is, would a respectgive -dev postfix
> > image be possible which already comes with some things which we expect
> > in a kernel development environment (like grub dev console setup, we
> > have a list off things we do in kdevops for it)?
> 
> Debian won't publish modified images for this purpose - but time
> permitting, I may be able to help with unofficial images with those
> changes.  My time's been a bit slim in recent months, though I expect
> this will be easier than vagrant.

No worries, given virt-builder's flexibility to do a lot of
customizations, and if we have debian-testing images available I am
confident now we can take care of the rest easily.

  Luis

Re: debian cloud collaboration for kdevops

[Richard W.M. Jones]

On Tue, Apr 02, 2024 at 05:56:08PM -0700, Luis Chamberlain wrote:
> Given I have issues posting to the guestfs mailing list I'm just CC'ing
> rjones foro now (Sorry Richard!)

I'm seeing them, just been dealing with the whole xz thing ...
I'll get around to it, likely later today.

Rich.

> Richard, do we have a way for virt-builder to deal with daily images
> changes? Seems distros may just use a symlink to the latest directory
> but what if the index file upstream changes daily?
> 
> On Fri, Mar 29, 2024 at 04:52:30PM -0700, Ross Vandegrift wrote:
> > Hi Luis,
> > 
> > On Thu, Mar 28, 2024 at 10:41:59AM -0700, Luis Chamberlain wrote:
> > > kdevops now uses guestfs [0] and it has its own respective set of base
> > > images for each distro [1], it occurs to me that as debian puts out
> > > debian testing vagrant images, wouldn't it make sense to put out also
> > > guestfs debian testintg base images, and perhaps more reguarly up to
> > > date as with vagrant?
> > 
> > I'm not sure I understand your ask - when I've used libguestfs, it's
> > able to access lots of different image formats.  Does kdevops require a
> > specific format?  Based on the builder.libguestfs.org images, here are
> > the latest trixie builds that I'd expect to work:
> 
> > raw: https://cloud.debian.org/images/cloud/trixie/daily/20240329-1701/debian-13-nocloud-arm64-daily-20240329-1701.tar.xz
> > qcow2: https://cloud.debian.org/images/cloud/trixie/daily/20240329-1701/debian-13-nocloud-arm64-daily-20240329-1701.qcow2
> 
> Either of these work nicely!
> 
> I see the symlink with "latest" on the daily directory works better for
> a static directory:
> 
> Debian 12:
> https://cloud.debian.org/images/cloud/bookworm/
> Debian 13, debian testing:
> https://cloud.debian.org/images/cloud/trixie/daily/latest/
> Unstable:
> https://cloud.debian.org/images/cloud/sid/daily/latest/
> 
> Now we just need a published index file for these, let me demonstrate
> what the opensuse folks did with the upstream package, they just added
> the opensuse.conf and a opensuse.gpg used to sign the index file file with:
> 
> cat /etc/virt-builder/repos.d/opensuse.conf 
> [opensuse.org]
> uri=http://download.opensuse.org/repositories/Virtualization:/virt-builder-images/images/index
> gpgkey=file:///etc/virt-builder/repos.d/opensuse.gpg
> 
> Let's look at the index file:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> [opensuse-tumbleweed]
> name=openSUSE Tumbleweed
> osinfo=opensuse-factory
> arch=x86_64
> file=openSUSE-Tumbleweed.x86_64-2022.11.22-Build8.411.qcow2.xz
> checksum=8e0a60d5df90a0ae4127ec0a006a0e5d30f36a7a4a23a66ed63ce8df934ab25568f663e73b3b5defecf5ae5c44bfb0ebc831c903b212bca2c68346049b18401c
> revision=202211228411
> format=qcow2
> size=6442450944
> compressed_size=187566376
> expand=/dev/sda1
> notes=This openSUSE Tumbleweed image only contains the base pattern.
>  It is designed to run as either KVM or Xen HVM guest.
> 
>  -----BEGIN PGP SIGNATURE-----
>  Version: GnuPG v1.0.7 (GNU/Linux)
> 
>  iQEVAwUBZgiPNqGT+7VyF0/CAQgwiwgAlb8wKbPLzB0QwLOM7QV+TN2Qp4eQsOHj
>  Sjq2GrND/trf1IXQi07B4d7XvL0XmkJY8HoCHYR2+bN3Yef/0HQeDH8Nb9ly1QOA
>  YZFGtNyjx8tkrpCiePmRAyPZw9T7F6Pz3He6O0broOqp+71Cxv4DtfscdNj8ady/
>  pQE4KLe2xjpjoqGt+4kwA684Jlp88k5wHHyDGZpGJogdU00CtfCSX7VcbA72Cd4+
>  lQUw8XI0GhQSl75MvDlAXifHsRR4uHlmerN3l1ZeamEBrzHT5l8ZAvAMhvmayvH3
>  wnxaaVt18eM3+hWW5fitIaN4EBsoFPlMaMP3ulGIVx8+Mw2+PvZGqw==
>  =9Sz5
>  -----END PGP SIGNATURE-----
> 
> The checksum is the sha512sum on the xz file. This is already computed
> in the file https://cloud.debian.org/images/cloud/sid/daily/latest/SHA512SUMS
> 
> So I take it we'd need two files created on these directories:
> 
>   * index
>   * index.gpg
> 
> I checked the bookworm tarball:
> 
> https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-nocloud-amd64.tar.xz
> 
> This just has *one* file: disk.raw
> 
> > If you browse around, there's also amd64 & ppc64el images, and a nocloud
> > variant that should boot on bare metal.
> 
> Neat!
> 
> > I'll take a shot, after I catch
> > up on the newer kdevops work.
> 
> We can add the support for adding a custom kdevops-debian.conf until we
> get a respective upstream package for debian with its own conf file, and
> we can also add the Kconfig options on kdevops to allow you to pick the
> distro. But let's start out small. Just getting debian trixie up with
> an index I think would be my first preferred target on debian. If debian
> can set the index file and index.gpg funcitonal we can work by testing it.
> 
> > > Would this be possible? If that is, would a respectgive -dev postfix
> > > image be possible which already comes with some things which we expect
> > > in a kernel development environment (like grub dev console setup, we
> > > have a list off things we do in kdevops for it)?
> > 
> > Debian won't publish modified images for this purpose - but time
> > permitting, I may be able to help with unofficial images with those
> > changes.  My time's been a bit slim in recent months, though I expect
> > this will be easier than vagrant.
> 
> No worries, given virt-builder's flexibility to do a lot of
> customizations, and if we have debian-testing images available I am
> confident now we can take care of the rest easily.
> 
>   Luis

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW