a

* Re: [PATCH 05/10] guestfs: add initial debian trixie support with custom URLs
  @ 2024-05-11 23:46  1%     ` Luis Chamberlain
  0 siblings, 0 replies; 200+ results
From: Luis Chamberlain @ 2024-05-11 23:46 UTC (permalink / raw)
  To: Scott Mayhew, Richard W.M. Jones; +Cc: kdevops

On Wed, May 08, 2024 at 01:30:42PM -0400, Scott Mayhew wrote:
> On Tue, 07 May 2024, Luis Chamberlain wrote:
> 
> > debian does not yet provide an index file for virt-builder, but we
> > have URLS with qcow2 and raw files. Using them in guestfs is actually
> > not quite trivial. So we do the handy work to enable others to also
> > use custom URLs and build a virt-builder local source and index file
> > for you. All we need really, is to check the checksums.
> > 
> > Sadly this does not yet work, as it seems we get stuck on the grub
> > prompt for some reason. This needs some more investigation.
> 
> Yeah when I run 'make bringup' the consoles on all the guests are stuck
> on "Booting `Debian GNU/Linux'".

Is that with Trixie or also with y9our own custom ISO?

> I tried manually running virt-builder + virt-install using the same
> image and I get the same result.  Granted, I've only ever used the
> images in the libguestfs repos or images that I've built myself, so I
> could be doing something wrong.

I'm in hopes we can figure this out, there is likely something special
done on the images hosted by libguestfs.org which regular nocloud images
like debian's trixie is not doing:

https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily.raw

Once we figure that out, we should be able to easily do that as a post
up with virt-builder.

Richard, any hints on what likely should help fix boot?

> > diff --git a/kconfigs/Kconfig.guestfs b/kconfigs/Kconfig.guestfs
> > index 5838522908e8..5839fbedfd08 100644
> > --- a/kconfigs/Kconfig.guestfs
> > +++ b/kconfigs/Kconfig.guestfs
> > @@ -1,5 +1,30 @@
> >  if GUESTFS
> >  
> > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > +	bool
> 
> It might be useful for these variables to have prompts so they show up
> in 'make menuconfig'. 

That's a one line change, it just needs a description, for now its auto
because there is no custom raw image URL option, what you descrie seems
for us to want such a new entry and option. So sure, but I think that
can be done in subsequent patch?

> I have a whole bunch of workflows running for
> RHEL8 and RHEL9 nightly builds... they start with 8.9 or 9.3
> images 

I see, yes, we wouldn't need to let you enable GUESTFS_HAS_CUSTOM_RAW_IMAGE
but rather waht we want is just a drop down menu for distro to let you
select "custom", that way you could use a set of defconfigs which would
use the custom image URL, when that is enabled it selects
GUESTFS_HAS_CUSTOM_RAW_IMAGE.

> and use the CONFIG_KDEVOPS_CUSTOM_YUM_REPOFILE to update them to
> the latest bits.

My patch after this "guestfs: add support to infer host distro
mirrororing optimizations" helps to infer when we want to copy a
custom sources list, based on a host's hop count. What you describe
seems to beg for another auto-infererence which could be based on
some target directory and based on on the custom URL. That could be
added and then you wouldn't even need to make any custom .config other
than having a few defconfigs for each distro point release.

> If I could build my own images and dump them in a
> directory without having to worry about updating the index file that
> would probably speed things up a bit.

I think this is possible, we could just look for directory we add to
.gitignore, and if the user has it as a symlink, we leverage that as
a key indicator for "hunting season open for custom images, please
populate my Kconfig with these other custom options".

> > +
> > +config GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > +	bool
> > +
> > +config GUESTFS_CUSTOM_RAW_IMAGE_URL
> > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE
> > +	depends on GUESTFS_HAS_CUSTOM_RAW_IMAGE_URL
> > +	string
> > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_AMD64
> > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-genericcloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_GENERIC_CLOUD_AMD64
> > +	default "https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-nocloud-amd64-daily" if GUESTFS_DEBIAN_TRIXIE_NOCLOUD_AMD64
> 
> You need '.raw' at the end of the URL, right?

Oops yes sorry.

> > diff --git a/scripts/bringup_guestfs.sh b/scripts/bringup_guestfs.sh
> > index f90ed499051b..514a26a60436 100755
> > --- a/scripts/bringup_guestfs.sh
> > +++ b/scripts/bringup_guestfs.sh
> > @@ -30,8 +30,135 @@ GUESTFSDIR="${TOPDIR}/guestfs"
> >  OS_VERSION=${CONFIG_VIRT_BUILDER_OS_VERSION}
> >  BASE_IMAGE_DIR="${STORAGEDIR}/base_images"
> >  BASE_IMAGE="${BASE_IMAGE_DIR}/${OS_VERSION}.raw"
> > +
> > +build_custom_source()
> > +{
> > +	SOURCE_TMP=$(mktemp)
> > +	cat <<_EOT >$SOURCE_TMP
> > +[local]
> ^^^
> You probably want this to be unique based on the flavor.  If you switch
> flavors, then you're going to wind up with multiple repo configs with the
> same repo id and virt-builder will only pick up the last one it reads... 
> the end result being that it probably won't find the template you want
> to install.

So this is for the source, not the index, the source is local, can we
not have multiple arbitrary sources with the same name? The indexes for
each file would be different. Here I just wanted to emphasize that the
source is local.

> 
> > +uri=file:///${CUSTOM_INDEX}
> > +proxy=off
> > +_EOT

See the source, points to a custom index, and so we'd have multiple
local sources, the index files do change per flavor.

But certainly, it would be good to get this right to make this scale.
For now to start off with trixie, and later to scale to whatever custom
setup we need for random builds.

Thoughts / advice?

Anyway, since this still has a big fat warning as "this doesn't work
yet", I've made the changes required so far and pushed these patches in,
we can make changes to scale this, now that at laest its easier to test
these things.

  Luis

^ permalink raw reply	[relevance 1%]