Re: [OE-core] [PATCH v6 0/3] pkg-database and systemd-sysext image

[SCHNEIDER Johannes <johannes.schneider@leica-geosystems.com>]

Hoi Alexandre,


and thanks again for pointing out the build-failure.

but since the recent test restructuring didn't help either... i could use help to reproduce what the autobuilder is *exactly* doing :-S
because when running the tests directly, they pass (see below)

my guess is that in the whole/complete test-run the autobuilder pulls sth out of the (sstate) cache instead of actually building the image with the different feature-set?
or could it be some other side-effect when the whole test-set is run?

i had based the testcases for this patch-set on the others already present - so the same logical flow of building images is used (or did i miss sth?)

any hints/pointers/thoughts would be greatly appreciated :-)


gruß
Johannes

$> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
2024-04-15 07:50:39,668 - oe-selftest - INFO - Adding layer libraries:
2024-04-15 07:50:39,668 - oe-selftest - INFO -  /bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/meta/lib
2024-04-15 07:50:39,668 - oe-selftest - INFO -  /bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/meta-selftest/lib
2024-04-15 07:50:39,669 - oe-selftest - INFO - Checking base configuration is valid/parsable
NOTE: Starting bitbake server...
2024-04-15 07:50:40,549 - oe-selftest - INFO - Adding: "include selftest.inc" in /bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/build-st/conf/local.conf
2024-04-15 07:50:40,549 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2024-04-15 07:51:03,338 - oe-selftest - INFO - test_image_gen_pkgdbfs (imagefeatures.ImageFeatures)
2024-04-15 07:51:03,344 - oe-selftest - DEBUG - Writing to: /bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/build-st/conf/selftest.inc
IMAGE_GEN_PKGDBFS = "1"


2024-04-15 07:51:56,862 - oe-selftest - DEBUG - running cmd: /bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/build-st/tmp-glibc/sysroots/x86_64/usr/bin/opkg --volatile-cache --offline-root=/bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/build-st/tmp-glibc/deploy/images/qemux86-64 --add-arch qemux86-64:11 --add-arch qemux86_64:12 list-installed
2024-04-15 07:51:56,875 - oe-selftest - DEBUG - list-installed:
base-files - 3.0.14-r0
init-ifupdown - 1.0-r0
kernel-6.6.23-yocto-standard - 6.6.23+git0+f7f00b22ef_2d01bc1d4e-r0
kernel-image-6.6.23-yocto-standard - 6.6.23+git0+f7f00b22ef_2d01bc1d4e-r0
kernel-image-bzimage-6.6.23-yocto-standard - 6.6.23+git0+f7f00b22ef_2d01bc1d4e-r0
kernel-module-uvesafb-6.6.23-yocto-standard - 6.6.23+git0+f7f00b22ef_2d01bc1d4e-r0
packagegroup-core-boot - 1.0-r0
sysvinit-inittab - 2.88dsf-r0
v86d - 0.1.10-r0
2024-04-15 07:51:56,875 - oe-selftest - DEBUG - running cmd: /bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/build-st/tmp-glibc/sysroots/x86_64/usr/bin/opkg --volatile-cache --offline-root=/bsp-user-workspaces/scjoh/upstream-patches/openembedded-core/build-st/tmp-glibc/deploy/images/qemux86-64 --add-arch core2-64:11 --add-arch core2_64:12 list-installed
2024-04-15 07:51:56,888 - oe-selftest - DEBUG - list-installed:
base-passwd - 3.6.3-r0
busybox - 1.36.1-r0
busybox-hwclock - 1.36.1-r0
busybox-syslog - 1.36.1-r0
busybox-udhcpc - 1.36.1-r0
eudev - 3.2.14-r0
init-system-helpers-service - 1.66-r0
initscripts - 1.0-r0
initscripts-functions - 1.0-r0
kmod - 31-r0
ldconfig - 2.39+git0+1b9c1a0047-r0
libblkid1 - 2.39.3-r0
libc6 - 2.39+git0+1b9c1a0047-r0
libcrypto3 - 3.2.1-r0
libkmod2 - 31-r0
liblzma5 - 5.4.6-r0
libz1 - 1.3.1-r0
modutils-initscripts - 1.0-r0
openssl-conf - 3.2.1-r0
openssl-ossl-module-legacy - 3.2.1-r0
sysvinit - 3.04-r0
sysvinit-pidof - 3.04-r0
ttyrun - 2.31.0-r0
update-alternatives-opkg - 0.6.3-r0
util-linux-fcntl-lock - 2.39.3-r0
2024-04-15 07:51:56,888 - oe-selftest - INFO -  ... ok
2024-04-15 07:51:58,138 - oe-selftest - INFO - ----------------------------------------------------------------------
2024-04-15 07:51:58,138 - oe-selftest - INFO - Ran 1 test in 77.808s
2024-04-15 07:51:58,139 - oe-selftest - INFO - OK
2024-04-15 07:52:01,880 - oe-selftest - INFO - RESULTS:
2024-04-15 07:52:01,880 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_image_gen_pkgdbfs: PASSED (53.55s)
2024-04-15 07:52:01,881 - oe-selftest - INFO - SUMMARY:
2024-04-15 07:52:01,882 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 77.809s
2024-04-15 07:52:01,882 - oe-selftest - INFO - oe-selftest - OK - All required tests passed (successes=1, skipped=0, failures=0, errors=0)


________________________________________
From: Alexandre Belloni <alexandre.belloni@bootlin.com>
Sent: Sunday, April 21, 2024 22:36
To: SCHNEIDER Johannes
Cc: openembedded-core@lists.openembedded.org; richard.purdie@linuxfoundation.org; alex.kanavin@gmail.com
Subject: Re: [OE-core] [PATCH v6 0/3] pkg-database and systemd-sysext image

[You don't often get email from alexandre.belloni@bootlin.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

This email is not from Hexagon’s Office 365 instance. Please be careful while clicking links, opening attachments, or replying to this email.


On 15/04/2024 08:02:00+0200, Johannes Schneider via lists.openembedded.org wrote:
> systemd-sysext allows to overlay another image (or multiple) ontop of
> a "base-image" = the current rootfs, via the use of overlayfs; to add
> tools and features meant for development purposes.
>
> To quote the documentation on systemd-sysext:
> " ...addition in order to make debugging/development easier). System
> extension images should not be misunderstood as a generic software
> packaging framework, ..."
>
> To build a lean image, that only holds packages that are not already
> part of the base-image, a snapshot of the package-database is taken
> after the installation of the base-rootfs is done, and picked up again
> when collecting the rootfs of such a extension image.
>
> with all this in place an example usage could look like this:
> some-core-image.bb
>   inherit core-image
>   IMAGE_GEN_PKGDBFS = "1"
>
> extending-image.bb
>   inherit image-sysext
>   IMAGE_FSTYPES = "squashfs"
>   IMAGE_BASE_PKGDB = "some-core-image"
>   # the above pointing at a package-db similar to:
>   # build/deploy/images/$MACHINE/some-core-image-$MACHINE-20240210172305-pkgdb.rootfs.tar.gz
>
> then on the device, running some-core-image, with the extension image placed at FN:
> $> ln -s "$FN" /run/extensions/$(basename $FN).raw
> $> systemd-sysext list
> $> SYSTEMD_LOG_LEVEL=debug systemd-sysext merge
>
> As long as the VERSION_ID of the extension image matches the os-release
> in the base image, the above commands return sucessfully;
> for details on the compativility check see the docs for systemd-sysext.
>
> =========
>
> changes with v2:
>         rebase from 'kirkstone' onto 'master'
>
> changes with v3:
>       incorporate review suggestions for simplification
>       add task dependency handling
>       add oe-selftest for the pkgdb handling
>       add variable documentation and
>       some more comments, and examples in the commit-msg
>
> changes with v4:
>       rebase onto 'master' => no functional changes
>       fixed patchtest findings
>
> changes with v5:
>         rebase onto 'master'
>       add '.sysext' to the deployed symlink name
>         sidenote on the tests and autobuilder failure: run locally they succeed, e.g.:
>           #> oe-selftest --verbose -r imagefeatures.ImageFeatures.test_image_gen_pkgdbfs
>
> changes with v6:
>       restructure test to call 'bitbake' only once in the testcase itself
>         (in hopes of solving the autobuilder problem; local test runs succeed)


This still fails on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6607/steps/14/logs/stdio

Reached login banner in 3.26 seconds (04/18/24 18:54:41)
Couldn't login into serial console as root using blank password
The output:
root
Password:
Login timed out after 60 seconds.
Poky (Yocto Project Reference Distro) 5.0+snapshot-0ee872ff1baf43e5436ccbd4c9ae4d733289e05e qemux86-64 /dev/ttyS1
qemux86-64 login: <<< run_serial(): command timed out after 120 seconds without output >>>
2024-04-18 19:17:59,061 - oe-selftest - INFO - 6: 20/46 379/588 (131.80s) (0 failed) (imagefeatures.ImageFeatures.test_image_gen_pkgdbfs)
2024-04-18 19:17:59,062 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/imagefeatures.py", line 342, in test_image_gen_pkgdbfs
    self.assertTrue(os.path.exists(os.path.join(img_vars['DEPLOY_DIR_IMAGE'], 'var/lib/opkg/status')), 'opkg\'s status file was not present in: %s' % tar_file)
  File "/usr/lib/python3.9/unittest/case.py", line 682, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : opkg's status file was not present in: /home/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3638382/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-pkgdb.tar.gz


>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#198219): https://lists.openembedded.org/g/openembedded-core/message/198219
> Mute This Topic: https://lists.openembedded.org/mt/105530718/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com/