* [PATCH v2] license_image: Use sstate to populate license directory
@ 2021-07-01 21:44 Douglas
2021-07-02 12:35 ` [OE-core] " Alexandre Belloni
0 siblings, 1 reply; 4+ messages in thread
From: Douglas @ 2021-07-01 21:44 UTC (permalink / raw)
To: openembedded-core; +Cc: Douglas Royds
We were getting an endlessly-growing list of directories in
build/tmp/deploy/licenses/ of the form myimage-mymachine-20210629010203.
Use the normal sstate behaviour to clean them. Brings license_image in
line with the do_populate_lic behaviour in license.bbclass.
We must only clean the (new) WORKDIR/license-destdir at do_rootfs time,
as the write_package_manifest and license_create_manifest rootfs
postprocess commands write the package.manifest and license.manifest
files at that time. They get stored in the sstate cache at
do_populate_lic_deploy time, alongside the image_license.manifest.
Looks like this:
license-destdir/
├── myimage-mymachine -> myimage-mymachine-20210629025723/
└── myimage-mymachine-20210629025723/
├── image_license.manifest
├── license.manifest
└── package.manifest
Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
meta/classes/license_image.bbclass | 38 +++++++++++++++++-------------
1 file changed, 22 insertions(+), 16 deletions(-)
diff --git a/meta/classes/license_image.bbclass b/meta/classes/license_image.bbclass
index 73cebb4d55e..30204b47b1c 100644
--- a/meta/classes/license_image.bbclass
+++ b/meta/classes/license_image.bbclass
@@ -1,8 +1,9 @@
ROOTFS_LICENSE_DIR = "${IMAGE_ROOTFS}/usr/share/common-licenses"
+LICSSTATEDIR = "${WORKDIR}/license-destdir/"
python write_package_manifest() {
# Get list of installed packages
- license_image_dir = d.expand('${LICENSE_DIRECTORY}/${IMAGE_NAME}')
+ license_image_dir = d.expand('${LICSSTATEDIR}/${IMAGE_NAME}')
bb.utils.mkdirhier(license_image_dir)
from oe.rootfs import image_list_installed_packages
from oe.utils import format_pkg_list
@@ -32,7 +33,7 @@ python license_create_manifest() {
pkg_lic_name = "LICENSE_" + pkg_name
pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name]
- rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY'),
+ rootfs_license_manifest = os.path.join(d.getVar('LICSSTATEDIR'),
d.getVar('IMAGE_NAME'), 'license.manifest')
write_license_files(d, rootfs_license_manifest, pkg_dic, rootfs=True)
}
@@ -196,23 +197,18 @@ def license_deployed_manifest(d):
key,val = line.split(": ", 1)
man_dic[dep][key] = val[:-1]
- lic_manifest_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'),
- d.getVar('IMAGE_NAME'))
- bb.utils.mkdirhier(lic_manifest_dir)
- image_license_manifest = os.path.join(lic_manifest_dir, 'image_license.manifest')
+ licsstatedir = d.getVar('LICSSTATEDIR')
+ image_name = d.getVar('IMAGE_NAME')
+ image_license_manifest = os.path.join(licsstatedir, image_name, 'image_license.manifest')
write_license_files(d, image_license_manifest, man_dic, rootfs=False)
link_name = d.getVar('IMAGE_LINK_NAME')
if link_name:
- lic_manifest_symlink_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'),
- link_name)
- # remove old symlink
- if os.path.islink(lic_manifest_symlink_dir):
- os.unlink(lic_manifest_symlink_dir)
-
- # create the image dir symlink
- if lic_manifest_dir != lic_manifest_symlink_dir:
- os.symlink(lic_manifest_dir, lic_manifest_symlink_dir)
+ if image_name != link_name:
+ link_path = os.path.join(licsstatedir, link_name)
+ if os.path.lexists(link_path):
+ os.remove(link_path)
+ os.symlink(image_name, link_path)
def get_deployed_dependencies(d):
"""
@@ -260,14 +256,24 @@ def get_deployed_files(man_file):
return dep_files
ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; "
+do_rootfs[cleandirs] += "${LICSSTATEDIR}"
do_rootfs[recrdeptask] += "do_populate_lic"
python do_populate_lic_deploy() {
license_deployed_manifest(d)
}
-
addtask populate_lic_deploy before do_build after do_image_complete
+
+SSTATETASKS += "do_populate_lic_deploy"
+do_populate_lic_deploy[dirs] = "${LICSSTATEDIR}/${IMAGE_NAME}"
do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
+do_populate_lic_deploy[sstate-inputdirs] = "${LICSSTATEDIR}"
+do_populate_lic_deploy[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
+
+python do_populate_lic_deploy_setscene () {
+ sstate_setscene(d)
+}
+addtask do_populate_lic_deploy_setscene
python license_qa_dead_symlink() {
import os
--
2.25.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [OE-core] [PATCH v2] license_image: Use sstate to populate license directory
2021-07-01 21:44 [PATCH v2] license_image: Use sstate to populate license directory Douglas
@ 2021-07-02 12:35 ` Alexandre Belloni
2021-07-27 21:05 ` Douglas
[not found] ` <3de8742f-498c-7835-ebd7-0ed9fd388bf2@taitradio.com>
0 siblings, 2 replies; 4+ messages in thread
From: Alexandre Belloni @ 2021-07-02 12:35 UTC (permalink / raw)
To: douglas.royds; +Cc: openembedded-core
Hello Douglas,
Thank you for this patch,
On 02/07/2021 09:44:37+1200, Douglas via lists.openembedded.org wrote:
> We were getting an endlessly-growing list of directories in
> build/tmp/deploy/licenses/ of the form myimage-mymachine-20210629010203.
> Use the normal sstate behaviour to clean them. Brings license_image in
> line with the do_populate_lic behaviour in license.bbclass.
>
> We must only clean the (new) WORKDIR/license-destdir at do_rootfs time,
> as the write_package_manifest and license_create_manifest rootfs
> postprocess commands write the package.manifest and license.manifest
> files at that time. They get stored in the sstate cache at
> do_populate_lic_deploy time, alongside the image_license.manifest.
>
> Looks like this:
>
> license-destdir/
> ├── myimage-mymachine -> myimage-mymachine-20210629025723/
> └── myimage-mymachine-20210629025723/
> ├── image_license.manifest
> ├── license.manifest
> └── package.manifest
>
This ran on the autobuilders and I do believe it is the cause of the
following failures:
https://autobuilder.yoctoproject.org/typhoon/#builders/79/builds/2269/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#builders/86/builds/2247/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#builders/87/builds/2282/steps/14/logs/stdio
Those seem to be pseudo aborts because the link is changing.
> Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
> ---
> meta/classes/license_image.bbclass | 38 +++++++++++++++++-------------
> 1 file changed, 22 insertions(+), 16 deletions(-)
>
> diff --git a/meta/classes/license_image.bbclass b/meta/classes/license_image.bbclass
> index 73cebb4d55e..30204b47b1c 100644
> --- a/meta/classes/license_image.bbclass
> +++ b/meta/classes/license_image.bbclass
> @@ -1,8 +1,9 @@
> ROOTFS_LICENSE_DIR = "${IMAGE_ROOTFS}/usr/share/common-licenses"
> +LICSSTATEDIR = "${WORKDIR}/license-destdir/"
>
> python write_package_manifest() {
> # Get list of installed packages
> - license_image_dir = d.expand('${LICENSE_DIRECTORY}/${IMAGE_NAME}')
> + license_image_dir = d.expand('${LICSSTATEDIR}/${IMAGE_NAME}')
> bb.utils.mkdirhier(license_image_dir)
> from oe.rootfs import image_list_installed_packages
> from oe.utils import format_pkg_list
> @@ -32,7 +33,7 @@ python license_create_manifest() {
> pkg_lic_name = "LICENSE_" + pkg_name
> pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name]
>
> - rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY'),
> + rootfs_license_manifest = os.path.join(d.getVar('LICSSTATEDIR'),
> d.getVar('IMAGE_NAME'), 'license.manifest')
> write_license_files(d, rootfs_license_manifest, pkg_dic, rootfs=True)
> }
> @@ -196,23 +197,18 @@ def license_deployed_manifest(d):
> key,val = line.split(": ", 1)
> man_dic[dep][key] = val[:-1]
>
> - lic_manifest_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'),
> - d.getVar('IMAGE_NAME'))
> - bb.utils.mkdirhier(lic_manifest_dir)
> - image_license_manifest = os.path.join(lic_manifest_dir, 'image_license.manifest')
> + licsstatedir = d.getVar('LICSSTATEDIR')
> + image_name = d.getVar('IMAGE_NAME')
> + image_license_manifest = os.path.join(licsstatedir, image_name, 'image_license.manifest')
> write_license_files(d, image_license_manifest, man_dic, rootfs=False)
>
> link_name = d.getVar('IMAGE_LINK_NAME')
> if link_name:
> - lic_manifest_symlink_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'),
> - link_name)
> - # remove old symlink
> - if os.path.islink(lic_manifest_symlink_dir):
> - os.unlink(lic_manifest_symlink_dir)
> -
> - # create the image dir symlink
> - if lic_manifest_dir != lic_manifest_symlink_dir:
> - os.symlink(lic_manifest_dir, lic_manifest_symlink_dir)
> + if image_name != link_name:
> + link_path = os.path.join(licsstatedir, link_name)
> + if os.path.lexists(link_path):
> + os.remove(link_path)
> + os.symlink(image_name, link_path)
>
> def get_deployed_dependencies(d):
> """
> @@ -260,14 +256,24 @@ def get_deployed_files(man_file):
> return dep_files
>
> ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; "
> +do_rootfs[cleandirs] += "${LICSSTATEDIR}"
> do_rootfs[recrdeptask] += "do_populate_lic"
>
> python do_populate_lic_deploy() {
> license_deployed_manifest(d)
> }
> -
> addtask populate_lic_deploy before do_build after do_image_complete
> +
> +SSTATETASKS += "do_populate_lic_deploy"
> +do_populate_lic_deploy[dirs] = "${LICSSTATEDIR}/${IMAGE_NAME}"
> do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
> +do_populate_lic_deploy[sstate-inputdirs] = "${LICSSTATEDIR}"
> +do_populate_lic_deploy[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
> +
> +python do_populate_lic_deploy_setscene () {
> + sstate_setscene(d)
> +}
> +addtask do_populate_lic_deploy_setscene
>
> python license_qa_dead_symlink() {
> import os
> --
> 2.25.1
>
>
>
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [OE-core] [PATCH v2] license_image: Use sstate to populate license directory
2021-07-02 12:35 ` [OE-core] " Alexandre Belloni
@ 2021-07-27 21:05 ` Douglas
[not found] ` <3de8742f-498c-7835-ebd7-0ed9fd388bf2@taitradio.com>
1 sibling, 0 replies; 4+ messages in thread
From: Douglas @ 2021-07-27 21:05 UTC (permalink / raw)
To: Alexandre Belloni; +Cc: OE Core mailing list
On Sat, 3 Jul 2021 at 00:35, Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
> On 02/07/2021 09:44:37+1200, Douglas via lists.openembedded.org wrote:
> > We were getting an endlessly-growing list of directories in
> > build/tmp/deploy/licenses/ of the form myimage-mymachine-20210629010203.
> > Use the normal sstate behaviour to clean them. Brings license_image in
> > line with the do_populate_lic behaviour in license.bbclass.
> >
> > We must only clean the (new) WORKDIR/license-destdir at do_rootfs time,
> > as the write_package_manifest and license_create_manifest rootfs
> > postprocess commands write the package.manifest and license.manifest
> > files at that time. They get stored in the sstate cache at
> > do_populate_lic_deploy time, alongside the image_license.manifest.
> >
> > Looks like this:
> >
> > license-destdir/
> > ├── myimage-mymachine -> myimage-mymachine-20210629025723/
> > └── myimage-mymachine-20210629025723/
> > ├── image_license.manifest
> > ├── license.manifest
> > └── package.manifest
> >
>
> This ran on the autobuilders and I do believe it is the cause of the
> following failures:
>
> https://autobuilder.yoctoproject.org/typhoon/#builders/79/builds/2269/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#builders/86/builds/2247/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#builders/87/builds/2282/steps/14/logs/stdio
>
> Those seem to be pseudo aborts because the link is changing.
Why would pseudo abort when the link changes? Replacing a symlink
seems harmless enough ... any ideas?
> > Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
> > ---
> > meta/classes/license_image.bbclass | 38 +++++++++++++++++-------------
> > 1 file changed, 22 insertions(+), 16 deletions(-)
> >
> > diff --git a/meta/classes/license_image.bbclass b/meta/classes/license_image.bbclass
> > index 73cebb4d55e..30204b47b1c 100644
> > --- a/meta/classes/license_image.bbclass
> > +++ b/meta/classes/license_image.bbclass
> > @@ -1,8 +1,9 @@
> > ROOTFS_LICENSE_DIR = "${IMAGE_ROOTFS}/usr/share/common-licenses"
> > +LICSSTATEDIR = "${WORKDIR}/license-destdir/"
> >
> > python write_package_manifest() {
> > # Get list of installed packages
> > - license_image_dir = d.expand('${LICENSE_DIRECTORY}/${IMAGE_NAME}')
> > + license_image_dir = d.expand('${LICSSTATEDIR}/${IMAGE_NAME}')
> > bb.utils.mkdirhier(license_image_dir)
> > from oe.rootfs import image_list_installed_packages
> > from oe.utils import format_pkg_list
> > @@ -32,7 +33,7 @@ python license_create_manifest() {
> > pkg_lic_name = "LICENSE_" + pkg_name
> > pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name]
> >
> > - rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY'),
> > + rootfs_license_manifest = os.path.join(d.getVar('LICSSTATEDIR'),
> > d.getVar('IMAGE_NAME'), 'license.manifest')
> > write_license_files(d, rootfs_license_manifest, pkg_dic, rootfs=True)
> > }
> > @@ -196,23 +197,18 @@ def license_deployed_manifest(d):
> > key,val = line.split(": ", 1)
> > man_dic[dep][key] = val[:-1]
> >
> > - lic_manifest_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'),
> > - d.getVar('IMAGE_NAME'))
> > - bb.utils.mkdirhier(lic_manifest_dir)
> > - image_license_manifest = os.path.join(lic_manifest_dir, 'image_license.manifest')
> > + licsstatedir = d.getVar('LICSSTATEDIR')
> > + image_name = d.getVar('IMAGE_NAME')
> > + image_license_manifest = os.path.join(licsstatedir, image_name, 'image_license.manifest')
> > write_license_files(d, image_license_manifest, man_dic, rootfs=False)
> >
> > link_name = d.getVar('IMAGE_LINK_NAME')
> > if link_name:
> > - lic_manifest_symlink_dir = os.path.join(d.getVar('LICENSE_DIRECTORY'),
> > - link_name)
> > - # remove old symlink
> > - if os.path.islink(lic_manifest_symlink_dir):
> > - os.unlink(lic_manifest_symlink_dir)
> > -
> > - # create the image dir symlink
> > - if lic_manifest_dir != lic_manifest_symlink_dir:
> > - os.symlink(lic_manifest_dir, lic_manifest_symlink_dir)
> > + if image_name != link_name:
> > + link_path = os.path.join(licsstatedir, link_name)
> > + if os.path.lexists(link_path):
> > + os.remove(link_path)
> > + os.symlink(image_name, link_path)
> >
> > def get_deployed_dependencies(d):
> > """
> > @@ -260,14 +256,24 @@ def get_deployed_files(man_file):
> > return dep_files
> >
> > ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; "
> > +do_rootfs[cleandirs] += "${LICSSTATEDIR}"
> > do_rootfs[recrdeptask] += "do_populate_lic"
> >
> > python do_populate_lic_deploy() {
> > license_deployed_manifest(d)
> > }
> > -
> > addtask populate_lic_deploy before do_build after do_image_complete
> > +
> > +SSTATETASKS += "do_populate_lic_deploy"
> > +do_populate_lic_deploy[dirs] = "${LICSSTATEDIR}/${IMAGE_NAME}"
> > do_populate_lic_deploy[recrdeptask] += "do_populate_lic do_deploy"
> > +do_populate_lic_deploy[sstate-inputdirs] = "${LICSSTATEDIR}"
> > +do_populate_lic_deploy[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
> > +
> > +python do_populate_lic_deploy_setscene () {
> > + sstate_setscene(d)
> > +}
> > +addtask do_populate_lic_deploy_setscene
> >
> > python license_qa_dead_symlink() {
> > import os
> > --
> > 2.25.1
> >
>
> >
> >
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
--
This Communication is Confidential. We only send and receive email on the
basis of the terms set out at www.taitradio.com/email_disclaimer
<http://www.taitradio.com/email_disclaimer>
^ permalink raw reply [flat|nested] 4+ messages in thread
[parent not found: <3de8742f-498c-7835-ebd7-0ed9fd388bf2@taitradio.com>]
end of thread, other threads:[~2021-08-02 21:47 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-01 21:44 [PATCH v2] license_image: Use sstate to populate license directory Douglas
2021-07-02 12:35 ` [OE-core] " Alexandre Belloni
2021-07-27 21:05 ` Douglas
[not found] ` <3de8742f-498c-7835-ebd7-0ed9fd388bf2@taitradio.com>
2021-08-02 21:47 ` Alexandre Belloni
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.